From: "Jeffrey Jasper" <jjasper@(email surpressed)>
Subject: Maya 2011.5 rendering differently on farm vs locally
   Date: Mon, 28 Mar 2011 17:19:40 -0700

Only mental ray seems to be effected.

Thanks,

--
Using Opera's revolutionary email client: http://www.opera.com/mail/

Jeffrey Jasper wrote:
> [posted to rush.general]
> 
> This one is stumping me. We have been having a lot of weird Maya 2011  
> mental ray rendering issues. But rendering on our rush farm seemed more  
> reliable till we upgraded to 2011. I even patched to 2011.5 and am still  
> having issues.
> 
> Local render render fine. If I submit them to the Rush queue all frames  
> will render with no errors. At first we got some renders where it looked  
> like textures had shifted around on the object. Then we got some where the  
> frame only had some white horizontal lines going through the image files.  
> If I set a scene to render on one of the nodes from the Maya interface it  
> all renders fine.

	Can you email me directly (do not post to the group) your .common.pl
	and submit-maya script you're using?

	Some other info:

	1) Does the problem happen only on particular machines,
	   or is it random? (ie. if you re-render that same frame on
	   the machine it rendered badly, it works fine the second time?)

	2) Can you replicate the problem easily,  or does it happen infrequently?

	3) Does more than one instance of maya/mray run on a machine
	   at a time? (depends on the combo of your rush/etc/hosts file
	   settings and/or your Cpus: specification)

	4) Try telling rush to render the job on one render node
	   (ie. Cpus: somehost=1)

	   Does that render OK?

	   If so, then try adding more processors to the job until
	   the problem starts happening. (Try limiting the job to only
	   use one instance per machine on many machines, vs multiple
	   instances on one machine)

	   See if you can find a pattern.

	5) You mentioned some white horizontal lines through the image.
	   Are these 'scanline errors' that seem to run through the entire
	   image, as if the image was corrupt, or does the line follow
	   geometry (ie. cracks in texturemaps or at geometry boundaries)

	   If scanline errors, sounds like a problem writing to the
	   file server; make sure you don't have a situation where two
	   or more jobs are rendering the same frame at the same time
	   (ie. a left over job is still running while the same job
	   is resubmitted again), or check your file server and client
	   system logs to see if the network file system is choking.
	   at around the same time the images in question were being rendered,
	   to see if there's file system corruptions due to network or server load.

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

2. Can't really replicate it consistently, seems to happen at random

3. Yes, 4 instances per machine

Thanks!

On Mon, 28 Mar 2011 17:41:20 -0700, Greg Ercolano <erco@(email surpressed)> wrote:

Jeffrey Jasper wrote:

[posted to rush.general]

This one is stumping me. We have been having a lot of weird Maya 2011
mental ray rendering issues. But rendering on our rush farm seemed more
reliable till we upgraded to 2011. I even patched to 2011.5 and am still
having issues.

Local render render fine. If I submit them to the Rush queue all frames
will render with no errors. At first we got some renders where it looked

like textures had shifted around on the object. Then we got some where the frame only had some white horizontal lines going through the image files. If I set a scene to render on one of the nodes from the Maya interface it

all renders fine.

	Can you email me directly (do not post to the group) your .common.pl
	and submit-maya script you're using?

	Some other info:

	1) Does the problem happen only on particular machines,
	   or is it random? (ie. if you re-render that same frame on
	   the machine it rendered badly, it works fine the second time?)

2) Can you replicate the problem easily, or does it happen infrequently?

	3) Does more than one instance of maya/mray run on a machine
	   at a time? (depends on the combo of your rush/etc/hosts file
	   settings and/or your Cpus: specification)

	4) Try telling rush to render the job on one render node
	   (ie. Cpus: somehost=1)

	   Does that render OK?

	   If so, then try adding more processors to the job until
	   the problem starts happening. (Try limiting the job to only
	   use one instance per machine on many machines, vs multiple
	   instances on one machine)

	   See if you can find a pattern.

	5) You mentioned some white horizontal lines through the image.
	   Are these 'scanline errors' that seem to run through the entire
	   image, as if the image was corrupt, or does the line follow
	   geometry (ie. cracks in texturemaps or at geometry boundaries)

	   If scanline errors, sounds like a problem writing to the
	   file server; make sure you don't have a situation where two
	   or more jobs are rendering the same frame at the same time
	   (ie. a left over job is still running while the same job
	   is resubmitted again), or check your file server and client
	   system logs to see if the network file system is choking.
	   at around the same time the images in question were being rendered,

to see if there's file system corruptions due to network or server load.

--
Using Opera's revolutionary email client: http://www.opera.com/mail/

Jeffrey Jasper wrote:
> 2. Can't really replicate it consistently, seems to happen at random
> 
> 3. Yes, 4 instances per machine
> 
> 4. I will give this a try and let you know, this week all my tests files  
> have been rendering fine on the farm, go figure. I have not made any  
> changes to the nodes or rush settings.
> 
> 5. scanline errors, the image is completely corrupted. All you would see  
> if white irregular horizontal lines. 

	Hmm, send me your .common.pl and submit-maya script via private email.

	If the scripts are really old, it might be an issue with maya's
	temp files overwriting each other. I want to check your scripts
	to see if it has the code that prevents that.


-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Thanks again,
Jeff

On Wed, 06 Apr 2011 23:13:46 -0700, Greg Ercolano <erco@(email surpressed)> wrote:

Jeffrey Jasper wrote:

2. Can't really replicate it consistently, seems to happen at random

3. Yes, 4 instances per machine

4. I will give this a try and let you know, this week all my tests files
have been rendering fine on the farm, go figure. I have not made any
changes to the nodes or rush settings.

5. scanline errors, the image is completely corrupted. All you would see
if white irregular horizontal lines.

	Hmm, send me your .common.pl and submit-maya script via private email.

	If the scripts are really old, it might be an issue with maya's
	temp files overwriting each other. I want to check your scripts
	to see if it has the code that prevents that.

--
Using Opera's revolutionary email client: http://www.opera.com/mail/

Jeffrey Jasper wrote:
> Thanks, just sent them via email. If it helps we are all Mac OS X based.  
> We are running OS X 10.6.7 on Intel Mac Pros and Xserves.

	Yes, it looks like your .common.pl file is from an old release
	(about 5 years old) and did have a problem.

	I suggest this line near the top of the file (on or around line 38):

chdir($G::tmpdir);

	..be changed to instead read:

if ( $G::iswindows && !defined($ENV{RUSH_TMPDIR}) )             # windows: prevents complaints from DOS
    { chdir($G::tmpdir); }                                      # about cwd being a UNC path

	This ensures that during render time, if the renderer decides
	to write to the current working directory, it will do so in
	the RUSH_TMPDIR instead of eg. /var/tmp.

	Not sure if that will help, but it might.

	If it doesn't, see if limiting renders to only one render
	per box solves the problem, as it might be the case maya
	is creating temp files that are overwriting each other when
	more than one render runs on a machine. Every once in a while
	renderers have that problem, then they fix it, then years later
	repeat it again :(

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Thanks again!
Jeff

On Fri, 08 Apr 2011 18:08:51 -0700, Greg Ercolano <erco@(email surpressed)> wrote:

Jeffrey Jasper wrote:

Thanks, just sent them via email. If it helps we are all Mac OS X based.
We are running OS X 10.6.7 on Intel Mac Pros and Xserves.

	Yes, it looks like your .common.pl file is from an old release
	(about 5 years old) and did have a problem.

	I suggest this line near the top of the file (on or around line 38):

chdir($G::tmpdir);

	..be changed to instead read:

if ( $G::iswindows && !defined($ENV{RUSH_TMPDIR}) ) # windows: prevents complaints from DOS { chdir($G::tmpdir); } # about cwd being a UNC path

	This ensures that during render time, if the renderer decides
	to write to the current working directory, it will do so in
	the RUSH_TMPDIR instead of eg. /var/tmp.

	Not sure if that will help, but it might.

	If it doesn't, see if limiting renders to only one render
	per box solves the problem, as it might be the case maya
	is creating temp files that are overwriting each other when
	more than one render runs on a machine. Every once in a while
	renderers have that problem, then they fix it, then years later
	repeat it again :(

--
Using Opera's revolutionary email client: http://www.opera.com/mail/

Jeffrey Jasper wrote:
> Thanks Greg, both were helpfull. Maya Mental Ray rendering is definitely  
> finicky. So setting the nodes to only process one render at a time and  
> fixing the temp directory made all my issues go away.
> 
> Once we migrate to Maya 2012 I will see if I can switch it back to 4  
> renders per node.

	There is also this known issue with Maya 2011 batch rendering that
	corrupts workspace.mel files.. that /might/ be biting you too, not sure.
	For more info on that, see:
	http://mayastation.typepad.com/maya-station/2010/11/workspacemel-.html

	You can try making the file read-only to prevent corruption; haven't
	tested if that works, but one of the folks on that page recommended it
	as a workaround. Autodesk needs to fix that if they haven't already in 2012.

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)