From: Daniel Browne <dbrowne@(email surpressed)>
Subject: Nuke frame log output
   Date: Tue, 26 Oct 2010 21:52:35 -0400

Hi Greg,

I'm sure Dan Rosen's inquired about this before, but has anyone found a resolution to the issue of the nuke frame log not updating until the render is complete? I believe you indicated to him that this was an issue with STDOUT output buffering which was internal to Nuke.

Thanks,

-Dan

Daniel Browne wrote:
> I'm sure Dan Rosen's inquired about this before, but has anyone found a resolution
> to the issue of the nuke frame log not updating until the render is
> complete? I believe you indicated to him that this was an issue with STDOUT
> output buffering which was internal to Nuke.

	It sounds likely.

	Try running the command line of nuke from a  shell prompt,
	and redirect the output to a file, eg:

		nuke -arg1 -arg2 .. > file.out 2>&1

	..and while that is running, open another window, and try to look
	at the contents of the file.out file, eg:

		UNIX: tail -f file.out
		WINDOWS: type file.out

	If you don't see any output until the render is done or close to done,
	yes, that is the problem.

	If you /do/ see the output (eg. if file.out is on a local drive),
	then it might be a problem with the file server.

	If you think it's the file server, then try the same thing,
	but:

		1) do the test on a render node
		2) redirect the output to a file /on the file server/, eg:

		nuke -arg1 -arg2 .. > //yourserver/somedir/file.out 2>&1

	and then, instead of opening another shell on the same render node,
	open a shell on a workstation (or some *other* render node)
	and try to inspect the file, eg:

		UNIX: tail -f /yourserver/somedir/file.out
		WINDOWS: type \\yourserver\somedir\file.out

	If you only see the 'delayed output' problem that way, then it's
	possible this can be corrected, as likely the issue you have is:

		1) Your render nodes windows
		2) Your file server is unix running samba
		3) You have "oplocks" enabled in your samba config

	If that's the case, turning oplocks off in your smb.conf file
	for the volume may solve it. This disables client side caching
	that can cause write buffering to happen at the render node
	when it shouldn't, because samba has some long standing compatibility
	bug with window's client side file caching.

	If so, this issue is covered somewhat here:

	http://seriss.com/cgi-bin/rush/newsgroup-threaded.cgi?-viewthread+801+802+803+805+807+808+810

	I've seen this problem many times over the last 10 years,
	and in this particular case, has nothing to do with the
	renderer in question.

	However, if that's not it, then yes, it might very well be
	stdout buffering in nuke, and you /might/ be able to fix it
	yourself if you can use the correct python or tcl calls to
	disable stdout buffering on nuke start-up. In python, there
	are apparently several ways to disable output buffering:
	http://stackoverflow.com/questions/107705/python-output-buffering

	..which mentions:

   1. Use the python "-u" command line switch
   2. Wrap sys.stdout in an object that flushes after every write
   3. Set PYTHONUNBUFFERED env var
   4. sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

	I'd say in your case option 4 might work.

	You might also ask the Nuke folks about this; assuming you
	can replicate the first case above (eg. redirecting to a /local/
	file and getting delayed buffered output), then ask the nuke dudes
	about disabling output buffering, citing that test case.

	But /don't/ bother the Nuke folks if you have the samba issue
	above (second case), as that's nothing to do with nuke, and
	everything to do with the file server's own config.


-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

tail lets me see the output of the nuke render command itself in real time
from my NFS mounts.

Our current Nuke submit is written in Perl; is there a Python version already,
or are you recommending that we try to write our own from scratch?


On Oct 26, 2010, at 7:20 PM, Greg Ercolano wrote:

[posted to rush.general]

> Daniel Browne wrote:
> > I'm sure Dan Rosen's inquired about this before, but has anyone found a resolution
> > to the issue of the nuke frame log not updating until the render is
> > complete? I believe you indicated to him that this was an issue with STDOUT
> > output buffering which was internal to Nuke.
>
> 	It sounds likely.
>
> 	Try running the command line of nuke from a  shell prompt,
> 	and redirect the output to a file, eg:
>
> 		nuke -arg1 -arg2 .. > file.out 2>&1
>
> 	..and while that is running, open another window, and try to look
> 	at the contents of the file.out file, eg:
>
> 		UNIX: tail -f file.out
> 		WINDOWS: type file.out
>
> 	If you don't see any output until the render is done or close to done,
> 	yes, that is the problem.
>
> 	If you /do/ see the output (eg. if file.out is on a local drive),
> 	then it might be a problem with the file server.
>
> 	If you think it's the file server, then try the same thing,
> 	but:
>
> 		1) do the test on a render node
> 		2) redirect the output to a file /on the file server/, eg:
>
> 		nuke -arg1 -arg2 .. > //yourserver/somedir/file.out 2>&1
>
> 	and then, instead of opening another shell on the same render node,
> 	open a shell on a workstation (or some *other* render node)
> 	and try to inspect the file, eg:
>
> 		UNIX: tail -f /yourserver/somedir/file.out
> 		WINDOWS: type \\yourserver\somedir\file.out
>
> 	If you only see the 'delayed output' problem that way, then it's
> 	possible this can be corrected, as likely the issue you have is:
>
> 		1) Your render nodes windows
> 		2) Your file server is unix running samba
> 		3) You have "oplocks" enabled in your samba config
>
> 	If that's the case, turning oplocks off in your smb.conf file
> 	for the volume may solve it. This disables client side caching
> 	that can cause write buffering to happen at the render node
> 	when it shouldn't, because samba has some long standing compatibility
> 	bug with window's client side file caching.
>
> 	If so, this issue is covered somewhat here:
>
> 	http://seriss.com/cgi-bin/rush/newsgroup-threaded.cgi?-viewthread+801+802+803+805+807+808+810
>
> 	I've seen this problem many times over the last 10 years,
> 	and in this particular case, has nothing to do with the
> 	renderer in question.
>
> 	However, if that's not it, then yes, it might very well be
> 	stdout buffering in nuke, and you /might/ be able to fix it
> 	yourself if you can use the correct python or tcl calls to
> 	disable stdout buffering on nuke start-up. In python, there
> 	are apparently several ways to disable output buffering:
> 	http://stackoverflow.com/questions/107705/python-output-buffering
>
> 	..which mentions:
>
>   1. Use the python "-u" command line switch
>   2. Wrap sys.stdout in an object that flushes after every write
>   3. Set PYTHONUNBUFFERED env var
>   4. sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
>
> 	I'd say in your case option 4 might work.
>
> 	You might also ask the Nuke folks about this; assuming you
> 	can replicate the first case above (eg. redirecting to a /local/
> 	file and getting delayed buffered output), then ask the nuke dudes
> 	about disabling output buffering, citing that test case.
>
> 	But /don't/ bother the Nuke folks if you have the samba issue
> 	above (second case), as that's nothing to do with nuke, and
> 	everything to do with the file server's own config.
>
>
> --
> Greg Ercolano, erco@(email surpressed)
> Seriss Corporation
> Rush Render Queue, http://seriss.com/rush/
> Tel: +1 626-576-0010 ext.23
> Fax: +1 626-576-0020
> Cel: +1 310-266-8906
>

Hi Daniel,

	Thought I'd take this offline for a second to debug.

	Do you have any windows render nodes in the mix,
	or are you entirely unix/osx?

	If so, I don't think the client side caching/file server issue
	is afoot, as I've only seen that issue with windows render nodes.

> tail lets me see the output of the nuke render command itself in real time
> from my NFS mounts.

	When you ran that test, did you run nuke on a different machine
	than the tail? ie. run nuke on a render node, and run the "tail -f"
	on a workstation?

> Our current Nuke submit is written in Perl; is there a Python version already,
> or are you recommending that we try to write our own from scratch?

	No, not what I meant.

	If the problem is nuke buffering its stdout, there's nothing the
	submit script can do to change that; each executable (in this case,
	nuke) has its own internal stdout buffering that's handled by the
	C/C++ library the executable is linked against.

	So nothing short of modifying the actual application can change
	the app's stdout buffering.

	Even though we don't have the source code to e.g. nuke and maya,
	these programs do have their own python front ends which does give
	us access to the application's internals.

	So what I was suggesting, assuming this is the problem,
	is to try adding some code to nuke's startup scripts that attempts
	to disable stdout buffering. This way when nuke starts, stdout
	will be unbuffered.

	But we should first see if we can replicate the problem
	outside rush, so that you can test it easily.

	There's nothing magic about how rush redirects the output
	of the renders, so running the shell test I mentioned
	should replicate the same problem, assuming the test simulates
	the writing being done from a different machine than the read.

	We should maybe interact directly to pin it down.

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Greg Ercolano wrote:
> [posted to rush.general]
> 
> Hi Daniel,
> 
> 	Thought I'd take this offline for a second to debug.
> 

	LOL, so much for being offline ;)

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)