From: Craig Allison <craig@(email surpressed)>
Subject: Auto Rendering of QT's on jobcomplete
   Date: Tue, 16 Dec 2008 10:43:55 -0500

On 2008-12-16 07:43:55 -0800, Craig Allison <craig@(email surpressed)> said:

I'm looking at setting up an autorender of QT's when a set of 2k
frames have been rendered in Shake using Rush

Mat X

Craig Allison wrote:
> I'll have a look at AppleScript as a possible interim solution, but I
> really wanted to have it bundled into the Rush submit with a dialogue to
> add text to variables set up in my Shake script for slate notes etc.  I
> could give them a custom slate node to avoid this though, hmmm...
> 
> We're doing 50-100 qts a day here, not sure if that's classed as a lot,
> but it sure would be better than doing them manually!

	It should be possible to have the submit script create the
	.shk script on the fly, and then run it as a jobdonecommand.

	This is more or less what the submit-shake-quicktime.pl script
	does already.

	Maybe just take an existing shake script, and replace all
	the parts you want changeable with environment variables,
	then set these variables before invoking the shake script
	to convert the frames.

	I've been trying to look for a nice cross-platform QT converter
	solution. There are definitely a few commercial apps out there
	(MetaRender looks like one, part of the iridas tools),
	and some freeware apps as well (qt-tools, libquicktime, ffmpeg),
	but many of the freeware apps are not "cross platform".

	Some suggestions for command line quicktime converters I've seen:

		Framecycler (SequencePublisher/MetaRender)
		Nuke
		Shake
		Fusion
		http://avidemux.org/admWiki/index.php?title=Command_line_usage
		djv_convert
		ImageMagick+OpenQuicktime (alternative to ffmpeg)
		MEncoder (http://www.mplayerhq.hu)
		libquicktime + custom code
		qt-tools (qt_export tool)

	Nuke:
n = nuke.createNode('Write')
n.knob('file').setValue('C:/quicktime.mov')
n.knob('codec').setValue(12) # H.264
n.knob('settings').setValue('0000000000000000000000000000019a7365616e0000000100000001000000000000018676696465000000010000000e00000000000000227370746c0000000100000000000000006176633100000000001800000300000000207470726c000000010000000000000000000003000018000000000018000000246472617400000001000000000000000000177000000000530000010000000100000000156d70736f00000001000000000000000000000000186d66726100000001000000000000000000000000000000187073667200000001000000000000000000000000000000156266726100000001000000000000000001000000166d70657300000001000000000000000000000000002868617264000000010000000000000000000000000000000000000000000000000000000000000016656e647300000001000000000000000000000000001663666c67000000010000000000000000004400000018636d66720000000100000000000000006170706c00000014636c75740000000100000000000000000000001c766572730000000100000000000000000003001a00010000')

	Shake has a similar technique, where a magic blob of text
	is needed for the QT codec settings (see submit-shake-quicktime.pl)

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Greg Ercolano schrieb:

	I've been trying to look for a nice cross-platform QT converter
	solution. There are definitely a few commercial apps out there
	(MetaRender looks like one, part of the iridas tools),
	and some freeware apps as well (qt-tools, libquicktime, ffmpeg),
	but many of the freeware apps are not "cross platform".

	Some suggestions for command line quicktime converters I've seen:

		Framecycler (SequencePublisher/MetaRender)
		Nuke
		Shake
		Fusion
		http://avidemux.org/admWiki/index.php?title=Command_line_usage
		djv_convert
		ImageMagick+OpenQuicktime (alternative to ffmpeg)
		MEncoder (http://www.mplayerhq.hu)
		libquicktime + custom code
		qt-tools (qt_export tool)

One you haven't mentioned: Silverstack from Pomfort: www.pomfort.com

Maybe you want to have a look on it.

Abraham

--
Abraham Schneider
Senior VFX Compositor, 2 D Artist
ARRI Film & TV Services GmbH   Tuerkenstr. 89   D-80799 Muenchen / Germany
Phone (Tel# suppressed) EMail aschneider@(email surpressed)
www.arri.de/filmtv
ARRI Film & TV Services GmbH
Sitz: München  -  Registergericht: Amtsgericht München  -  Handelsregisternummer: HRB 69396
Geschäftsführer: Franz Kraus; Thomas Till

> Our pipeline at the moment is: we render downsized, LUT-converted JPEGs 
> in the same setup as the main DPX files as a second output. After the 
> job is done I submit a job to the renderqueue of Silverstack (called 
> Cinemator) on a small Mac mini to generate quicktimes from these JPEGs.
> 
> Silverstack can create templates for burnin, LUT conversion, codec, etc. 
> which you can use while submitting the job to Cinemator on the commandline.
> 
> At the moment, Cinemator is a GUI app which can be fed by a commandline 
> command, but what I heard is that they are working on a more 
> sophisticated solution.
> 
> Maybe you want to have a look on it.

	Sounds interesting, but unless they have a non-gui interface,
	it's not something I think Rush can control properly.

	What I'd like to do is have a way to make it easy to add any
	kind of command line tool to handle QT conversions to the
	submit scripts, and have a pulldown that lets one select which
	converter to use, eg. ffmpeg, MetaRender, SequencePublisher,
	qt-tools, etc.

	Probably what I'd do is add a "QT" tab to all the submit scripts
	that has prompts for the input images and output movie, as well
	as being able to select which codecs to use. All the rush submit
	scripts would then be able to call a single, central script
	to handle the actual QT conversion as a 'jobdonecommand'.

Craig Allison wrote:
> I'm currently working on the Applescript suggestion to find/replace the
> contents of a Shake "setup" script and then submitting to command line
> render.

	Note that shake files are simple ascii, so you can use
	tools like sed or perl to take a template shake script
	and convert it on the fly to what you need.

	IIRC, you can also just edit the .shk file, and replace
	the strings you want to use with environment variables..
	then all you have to do is be sure to set those env vars
	before running shake, and the replacing will be handled by shake.
	I think there may also be a way to pass variables from the shake
	command line.

> Any QT additions to the Rush submit would be helpful for the artists, as
> they could make use of the farm for quick preview QT's for long shots
> instead of Framecycler/DJView/FlipBook.

	Right..

	See also the submit-shake-quicktime submit script,
	which artists can use to do QT conversions.

	Use the one from 102.42a9, which has many codecs,
	and several fixes from previous versions. These tweaks
	are mentioned in the release notes for A9:
	http://www.seriss.com/rush-current/rush/release-notes.html#102.42a9
	..specifically:

	    o submit-shake-quicktime -- numerous fixes to support codecs

> I'll let you know how I get on with the Applescript as I'm just
> unravelling how it all works and am seriously under the cosh with
> deadlines!!!

	Yes, feel free to send pastes.

	I've done a few scripts with AppleScript to descan
	the slitscan sequences from 2001; I used applescript
	to hit the frame advance button in iDVD so that I could
	grab scanlines from the film to unwrap them.
	Fun pictures here:
	http://seriss.com/people/erco/2001/wip/

	..with the scripts at the bottom. See 'shoot.pl' for a perl script
	that sends applescript text strings to osascript on a remote machine. eg:

system("( echo 'tell app \"DVD Player\"'; echo step dvd; echo end tell ) | rsh tow osascript");

	Must admit, applescript is quite neat. But the fact it works
	through GUIs is a problem for automation in contexts where there
	is no window manager, or no one logged in. In the above case,
	I had to make sure the GUI was running before I could run the script.


> Aye, aye, aye bring on Christmas!

	Amen to that..!

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Greg Ercolano schrieb:

	Sounds interesting, but unless they have a non-gui interface,
	it's not something I think Rush can control properly.

Abraham

--
Abraham Schneider
Senior VFX Compositor, 2 D Artist
ARRI Film & TV Services GmbH   Tuerkenstr. 89   D-80799 Muenchen / Germany
Phone (Tel# suppressed) EMail aschneider@(email surpressed)
www.arri.de/filmtv
ARRI Film & TV Services GmbH
Sitz: München  -  Registergericht: Amtsgericht München  -  Handelsregisternummer: HRB 69396
Geschäftsführer: Franz Kraus; Thomas Till

Craig Allison wrote:
> Ok, so I've finally got around to looking into this and have decided to
> change my original way of thinking and give the artists the opportunity
> to render a quicktime on the farm once they are happy with the results
> previewed in Framecycler.
> 
> I have taken the submit-shake-quicktime script and amended it to allow
> users to select which film they are working on, select the first frame
> in the sequence to render and dialogue for slate notes etc - the
> combination of all this information chooses the correct shake setup
> script on the network, replaces the variables, saves a copy and submits
> it to the farm.
> 
> The script renders a 2K slate & greyscale (named and numbered correctly)
> and also a QT with the appropriate settings for the production.  I have
> it working fine but now want to speed it up by splitting the job into
> two, i.e. one script to render the two 2k frames and one script to
> render the QT.

	Sounds cool

	But does it really take that long to generate the 2K slate and
	greyscale to make it worth while to have those happen in separate
	executions?

	I would assume that the quicktime gen can't happen until the
	2K slate/grayscale are done anyway, so even if another box
	were working on the slate, the qt part would still have to
	wait for it to get done.

	But if there's something like 50 frames of slate, then I guess
	it pays to have each frame render on a separate machine,
	and then have the qt gen run as a 'jobdonecommand', so that
	it doesn't run until all the slate frames are finished.

	Thing is, if what you're doing is generating just one slate
	image and one grayscale, but need those to hold for eg. 25x each,
	then just have the render create one image each, then make symbolic
	links for all the others; that'll save on disk space and render time,
	so that the qt gen can refer to the links.

	For instance, the script could render these two images:

slate-myshow-myshot.dpx
grayscale-myshow-myshot.dbx

	and then create symlinks for the frame range, eg:

# GRAYSCALE
ln grayscale-myshow-myshot.dpx myshow-shot-frame-0001.dpx
ln grayscale-myshow-myshot.dpx myshow-shot-frame-0002.dpx
ln grayscale-myshow-myshot.dpx myshow-shot-frame-0003.dpx
:
ln grayscale-myshow-myshot.dpx myshow-shot-frame-0025.dpx
# SLATE
ln slate-myshow-myshot.dpx     myshow-shot-frame-0026.dpx
ln slate-myshow-myshot.dpx     myshow-shot-frame-0027.dpx
ln slate-myshow-myshot.dpx     myshow-shot-frame-0028.dpx
:
ln slate-myshow-myshot.dpx     myshow-shot-frame-0050.dpx

	That way you can 'instantly' make all the frames from just
	the single 2k image, so that the quicktime can insert them
	into the shot as needed.

	HTH.

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

On 2008-12-16 11:16:35 -0800, Craig Allison <craig@(email surpressed)> said:

I'll have a look at AppleScript as a possible interim solution,

:)

Mat X

matx wrote:
> [posted to rush.general]
> 
> On 2008-12-16 11:16:35 -0800, Craig Allison <craig@(email surpressed)> said:
>> I'll have a look at AppleScript as a possible interim solution,
> 
> We use an AppleScript to interface between a template Shake script and 
> the image sequence selected (drag drop folder onto script). After the 
> images are rendered by an artist they make a QT using this tools which 
> basically substitutes new fileins and fileouts, etc.

	If this is something you can call as a command, then
	you can make it the jobdonecommand for the job.

> I'd love to see how you could incorporate a Shake script in a 
> shake-submit for Rush as a followup job using the variables of the 
> previous job. Can RUSH pass variables to another job?

	You can pass anything you want, but you'd need to add
	the operation to do it.

	The manual way is: submit your shake job, and lets say
	its jobid is "tahoe.1". While it's rendering, you can
	fill out the submit-shake-quicktime.pl form, and set
	these fields:

WaitFor: tahoe.1
Wait For State: Done

	..and submit the job. This way when your shake job
	finishes the last frame, the shake quicktime script
	runs to do the QT conversion.

	Or, if you're clever with scripting, you can modify
	the submit-shake script to automatically fire off
	submit-shake-quicktime to do the QT conversion as a "jobdonecommand".
	
	It's currently a two step process use submit-shake-quicktime.pl,
	
	After you submit the shake job, you can use submit-shake-quicktime.pl
	to submit a separate job to handle the conversion.

	It needs to know the input images and output quicktime file.


-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Craig Allison wrote:
> Ok, the retiming of the slate didn't work, Shake still insisted on
> rendering the slate/grey over and over again as it rendered the quicktime.
> 
> Have used to following to submit the two scripts from one submit:
> 
> my $submit = <<"EOF";
> title        $in{JobTitle}
> frames          1
> command         perl $G::self -render $QT_script_to_render
> $in{Verbosity} $in{Retries} $in{RetryBehavior} $in{MaxLogSize}
> $in{PrintEnvironment} $in{Frames} $in{Production} $in{OutputQuicktime}
> $in{ShakeFlags} 
> $submitoptions
> EOF
> 
> my $submit2 = <<"EOF";
> title        $in{JobTitle}
> frames          1
> command         perl $G::self -render $Slate_script_to_render
> $in{Verbosity} $in{Retries} $in{RetryBehavior} $in{MaxLogSize}
> $in{PrintEnvironment} $in{ShakeFlags} 
> $submitoptions
> EOF
> 
>  # ErrorWindow("$submit\n"); # DEBUGGING
>     RushSubmit($submithost, $submit2, $in{JobTitle}, $in{StartIrush});
>     exit(RushSubmit($submithost, $submit, $in{JobTitle}, $in{StartIrush}));
> 
> This works fine, only that when the second job starts, the log for
> $submit2 is overwritten as both jobs share the same directory, not such
> a problem, but may try to iron this out over the coming days.

	This is because both submits are using the same $submitoptions:

my $submit = <<"EOF";
..
$submitoptions
..
EOF

my $submit2 = <<"EOF";
..
$submitoptions
..
EOF

	If you look at the contents of the $submitoptions variable,
	I think you'll understand why you're getting so many settings
	common between the Shake and QT jobs.

	In this case, the $submitoptions contains the 'logdir' and 'frames'
	command from the shake job, and passing that to both jobs is what's
	causing the overlap. ie. thats why the frame range and logdir are
	coming up the same for both.

	What you'll want to do is /not/ include the $submitoptions variable
	in the second job, and just set the frames/logdir commands you want, eg:

### CREATE LOGDIR FOR QT JOB
my $qtlogdir = "${Slate_script_to_render}.log";
if ( ! -d $qtlogdir ) {
    mkdir($qtlogdir, 0777);
    if ( $G::iswindows ) {
        my $dir = $qtlogdir; $dir =~ s%/%\\%g;
	system("cacls $dirname /e /c /g everyone:f");
    }
}

### SUBMIT THE QT JOB
my $submit2 = <<"EOF";
title           $in{JobTitle}_QT
frames          1
command         perl $G::self -render $Slate_script_to_render ..
logdir          $qtlogdir
EOF


	BTW, if you want to see what the first job's $submitoptions is set to,
	you can stick an 'ErrorWindow();' command in there just above the first
	submit, eg:

ErrorWindow("SUBMITOPTIONS IS SET TO:\n$submitoptions\n");

my $submit = <<"EOF";
..

	This way when you submit a test job, you'll see what the variable
	is set to, so you can then determine what you'll need to pass in
	the second job, in case there's more than just 'frames' and 'logdir'.

> As an aside does anyone know a quick way of always resetting to defaults
> for the QT render instead of Rush picking up the last settings every
> time? I assume it's a deletion of a certain local file, but maybe I
> could just reset certain fields which would work much better?

	I'm thinking my answers above should help with that.

	Basically its that same "$submitoptions" variable being
	passed to both jobs.

	Let us know if you have any other questions.

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)