From: Dylan Penhale <dylan@(email surpressed).au>
Subject: Requeing failed frames from a batch
   Date: Thu, 09 Mar 2006 19:04:46 -0500

My thoughts:

_________________________________________

Dylan Penhale
Systems Administrator
Fuel International

I need to find a way to re-submit/re-queue failed frames when submit from batches.

	I might be missing something.

	Let's say after rendering a batch of 20-29, frame 25 fails,
	and you have a way to detect this by grepping the logs.

	So you'll know for sure that 20-24 are OK, and 25-29 need to be
	re-rendered.

	If you just exit the script with exit(2), rush will try to re-render
	the entire batch of 20-29.

	Are you saying that you want it to:

		a) Assume the problem was intermittent (ie. maya crashed due to a bug,
		   and will probably render 25-29 just fine if we re-invoke maya to just
		   render that frame range?
	..or..

		b) Assume the problem was caused by the user (a bad scene file)
		   and we should just append frames 25-29 to a text file somewhere
		   that we can later use to submit a job to fix these frames
		   after the user has intervened to fix the scenefile.

	If a) then yes, just re-invoke maya with the new frame range,
	and the user won't even have to know there was a failure.

	If b) then I would think you could do any of a number of things:

		o Have the render script create 0000.fix frames in the image dir
		  that a later 'fix job' could look for, and just render those frames

		o Append the bad frames to a .txt file

		o Have a 'jobdonecommand' that looks for bad frames, and submits
		  a 'fix job' in the pause state, and emails the user to fix the
		  problem, then unpause the fix job to run the fix frames..

It's easy to re-gueue the whole batch but if only one frame has failed it's re-rendering all the good frames again.

	If you have the logic in the render script to know which frames
	need to be re-rendered, and are confident that just re-rendering those
	is all that's needed to get them to render OK, then just re-invoke maya
	with just the fix range.

	Or, if you just want to tell the user which frames are bad, you can
	use 'rush -notes $ENV{RUSH_FRAME}:"BAD FRAMES: $bad_start - $bad_end"'
	so that the "Frames" report shows a message telling which frames were bad.
	(this is safe to do when errors occur. Using 'rush -notes' isn't recommended
	if run on EVERY frame.. that's too much load to the job server if there are
	100's of render nodes. But logging (uncommon) error conditions is OK..)

--
Greg Ercolano, erco@(email surpressed)
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Cel: (Tel# suppressed)
Fax: (Tel# suppressed)

Cool 

I think option A is what we want. Generally if we have found that if a batch
is going to fail from a scene file error it will either not render at all,
or start failing after a particular point in the frame range and from then
on. So this being the case we would just retry the batch from the failed
frame onwards.

The system we have come up with then is as follows. 

	O Rush submits the batch to mayabatch for rendering. 
	
	O Rush waits for the mayabatch exit and then checks the frames 
	(and frame sizes in our case) according to the log file.
	
	O If it finds that the number of frames written in the log does not
equal 	the number of frames in the batch then reset $opt{sfrm} value to the
last 	good frame +1 and retry. This will effectively retry the batch from
the 	failed frame till the end of the batch ($opt{efrm})

I think this is all we need. We currently have the failures going out as
emails to sysadmins and job owners, so I think we could tweak that slightly
so they only get the error if the 3rd retry (or however many retries are
set) is reached.


|-----Original Message-----
|From: Greg Ercolano [mailto:erco@(email surpressed)] 
|Sent: Friday, 10 March 2006 3:43 PM
|To: void@(email surpressed)
|Subject: Re: Requeing failed frames from a batch
|
|[posted to rush.general]
|
|> I need to find a way to re-submit/re-queue failed frames when submit 
|> from batches.
|
|	I might be missing something.
|
|	Let's say after rendering a batch of 20-29, frame 25 fails,
|	and you have a way to detect this by grepping the logs.
|
|	So you'll know for sure that 20-24 are OK, and 25-29 need to be
|	re-rendered.
|
|	If you just exit the script with exit(2), rush will try 
|to re-render
|	the entire batch of 20-29.
|
|	Are you saying that you want it to:
|
|		a) Assume the problem was intermittent (ie. 
|maya crashed due to a bug,
|		   and will probably render 25-29 just fine if 
|we re-invoke maya to just
|		   render that frame range?
|	..or..
|
|		b) Assume the problem was caused by the user (a 
|bad scene file)
|		   and we should just append frames 25-29 to a 
|text file somewhere
|		   that we can later use to submit a job to fix 
|these frames
|		   after the user has intervened to fix the scenefile.
|
|	If a) then yes, just re-invoke maya with the new frame range,
|	and the user won't even have to know there was a failure.
|
|	If b) then I would think you could do any of a number of things:
|
|		o Have the render script create 0000.fix frames 
|in the image dir
|		  that a later 'fix job' could look for, and 
|just render those frames
|
|		o Append the bad frames to a .txt file
|
|		o Have a 'jobdonecommand' that looks for bad 
|frames, and submits
|		  a 'fix job' in the pause state, and emails 
|the user to fix the
|		  problem, then unpause the fix job to run the 
|fix frames..
|
|> It's easy to re-gueue the whole batch but if only one frame 
|has failed 
|> it's re-rendering all the good frames again.
|
|	If you have the logic in the render script to know which frames
|	need to be re-rendered, and are confident that just 
|re-rendering those
|	is all that's needed to get them to render OK, then 
|just re-invoke maya
|	with just the fix range.
|
|	Or, if you just want to tell the user which frames are 
|bad, you can
|	use 'rush -notes $ENV{RUSH_FRAME}:"BAD FRAMES: 
|$bad_start - $bad_end"'
|	so that the "Frames" report shows a message telling 
|which frames were bad.
|	(this is safe to do when errors occur. Using 'rush 
|-notes' isn't recommended
|	if run on EVERY frame.. that's too much load to the job 
|server if there are
|	100's of render nodes. But logging (uncommon) error 
|conditions is OK..)
|
|--
|Greg Ercolano, erco@(email surpressed)
|Rush Render Queue, http://seriss.com/rush/
|Tel: (Tel# suppressed)
|Cel: (Tel# suppressed)
|Fax: (Tel# suppressed)
|

Dylan Penhale wrote:

The system we have come up with then is as follows. O Rush submits the batch to mayabatch for rendering. O Rush waits for the mayabatch exit and then checks the frames (and frame sizes in our case) according to the log file.

	
	O If it finds that the number of frames written in the log does not
        equal the number of frames in the batch then reset $opt{sfrm} value
        to the last good frame +1 and retry..

	Yes, exactly.

--
Greg Ercolano, erco@(email surpressed)
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Cel: (Tel# suppressed)
Fax: (Tel# suppressed)