From: Daniel Browne <dbrowne@(email surpressed)>
Subject: Rendering with CS5 AE
   Date: Wed, 02 Jun 2010 21:46:29 -0400

Hi Greg,

I'm working out a setup for rendering with CS5 After Effects and it appears Adobe still expects you to be logged into the desktop as the user executing the job. Is there a way to force only AE scripts to execute as a specific user? I know there is a parameter in the rush config file to execute all jobs as a specific user, but I'd rather it just be exclusive to AE jobs. Is that possible?

Thanks,

-Dan


----------
Dan "Doc" Browne
System Administrator

Attachment: pastedGraphic.tiff
Description: pastedGraphic.tiff

Evil Eye Pictures
dbrowne@(email surpressed)
Office: (415) 777-0666

Daniel Browne wrote:
> Hi Greg,
> 
> I'm working out a setup for rendering with CS5 After Effects and it appears Adobe still
> expects you to be logged into the desktop as the user executing the job. 

	Yes, after effects is still one of those renderers that has not yet
	separated their command line renderer from the GUI. AE's command line
	renderer, "aerender", still fires off the GUI and communicates with it
	to make the render happen. So it is still tied to the window manager,
	and this causes lots of weirdness. AE has always been this way, and
	it seems they have a very hard time changing it.


> Is there a way
> to force only AE scripts to execute as a specific user?

	You can use normal unix techniques to make the 'aerender' executable
	run as any user you want, just by setting the perms on the executable, eg:

chown 0:0  /path/to/aerender
chmod 4755 /path/to/aerender

	..which will make aerender run as root (and all that implies)
	no matter what user invoked the command.

	Root is never denied access to the window manager, so that would
	force it to not have permission issues with the window manager.

	However, if you prefer to have aerender run as a normal user, you can
	use the same commands in a similar way.

	For instance if the username is 'render' and the group is "users",
	then you can use:

chown render:users /path/to/aerender
chmod 4755 /path/to/aerender

	..to make "aerender" run as 'render' instead.

	Usually just making the aerender executable setuid as the user you want
	would do the trick, but in the recent versions of AE (CS3 and CS4 and
	possibly 5), you may need to go an extra step and make the AE GUI binary
	setuid.

	(It used to be the case that 'aerender' started the AE GUI with fork()/exec(),
	such that the GUI was part of the same process hierarchy, but they made a mistake
	in CS3 that causes the GUI to disconnect from the process hierarchy.. meh)

	Someday we really all should hope Adobe will rewrite aerender so that it
	does the rendering itself, and does not need the GUI to do any rendering.
	This way distributed rendering with AE would be solid like the rest of the
	compositors (nuke, shake..)

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Thanks Greg,

I've gotten it to render, with a number of caveats (perhaps you can shed some light on some):

- Our NFS based /Network/Library/Fonts folder is not recognized by AE (I believe this has always been the case), so the fonts needed for the scene had to be copied locally. So far I've had no success trying to make it recognize the central library from environment parameters. This happens even with a render user logged in on the console, so it's not an automount issue.

- In that same vein I cannot centralize plug-ins either. Nor for that matter could I centralize the entire AE package itself (even if I didn't have the mounts set with the nosuid flag)

- The aerender binary does not seem to return its error number consistently on exit; I've had batches register as Done even when they've failed in the frame logs, despite what is in my submit script. Do you know of another way, or do I simply have to capture the stdout and search for an error message?

- When I fail or dump a batch to re-que it, it doesn't kill the aerendercore process and I have to kill it on the individual machine. This is probably the most serious problem as often even killing aerendercore isn't enough. The kill results in a zombie process that makes it impossible to run another render on the machine until you do a "reboot -q" on it.

-Dan



On Jun 2, 2010, at 10:24 PM, Greg Ercolano wrote:

[posted to rush.general]

Daniel Browne wrote:
> Hi Greg,
> 
> I'm working out a setup for rendering with CS5 After Effects and it appears Adobe still
> expects you to be logged into the desktop as the user executing the job. 

	Yes, after effects is still one of those renderers that has not yet
	separated their command line renderer from the GUI. AE's command line
	renderer, "aerender", still fires off the GUI and communicates with it
	to make the render happen. So it is still tied to the window manager,
	and this causes lots of weirdness. AE has always been this way, and
	it seems they have a very hard time changing it.


> Is there a way
> to force only AE scripts to execute as a specific user?

	You can use normal unix techniques to make the 'aerender' executable
	run as any user you want, just by setting the perms on the executable, eg:

chown 0:0  /path/to/aerender
chmod 4755 /path/to/aerender

	..which will make aerender run as root (and all that implies)
	no matter what user invoked the command.

	Root is never denied access to the window manager, so that would
	force it to not have permission issues with the window manager.

	However, if you prefer to have aerender run as a normal user, you can
	use the same commands in a similar way.

	For instance if the username is 'render' and the group is "users",
	then you can use:

chown render:users /path/to/aerender
chmod 4755 /path/to/aerender

	..to make "aerender" run as 'render' instead.

	Usually just making the aerender executable setuid as the user you want
	would do the trick, but in the recent versions of AE (CS3 and CS4 and
	possibly 5), you may need to go an extra step and make the AE GUI binary
	setuid.

	(It used to be the case that 'aerender' started the AE GUI with fork()/exec(),
	such that the GUI was part of the same process hierarchy, but they made a mistake
	in CS3 that causes the GUI to disconnect from the process hierarchy.. meh)

	Someday we really all should hope Adobe will rewrite aerender so that it
	does the rendering itself, and does not need the GUI to do any rendering.
	This way distributed rendering with AE would be solid like the rest of the
	compositors (nuke, shake..)

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Daniel Browne wrote:
> - Our NFS based /Network/Library/Fonts folder is not recognized by AE (I 
>   believe this has always been the case), so the fonts needed for the scene had
>   to be copied locally. So far I've had no success trying to make it recognize
>   the central library from environment parameters. This happens even with
>   a render user logged in on the console, so it's not an automount issue.

	I'd suggest contacting Adobe about that one.

	Assuming you can replicate the problem from the command line
	(ie. ssh into the remote machine as the same user rush is rendering
	the job as when the problem occurs), and then report the problem to
	them in that context. Leave "rush" or "we're using a render queue"
	out of the support call, so that it doesn't confuse them.

> - In that same vein I cannot centralize plug-ins either. Nor for that matter
>   could I centralize the entire AE package itself (even if I didn't have the
>   mounts set with the nosuid flag)

	Can't answer to this, as I'm not sure what limitations are built
	into AE that prevent this.

	I have a feeling that AE might still be internally using OS9 techniques
	for accessing files (eg. the old "some:path:dir" instead of "/some/path/dir"
	pathnames) One can sometimes see those old OS9 style pathnames in AE's output.

	I understand they're trying to get away from that old stuff, but it's
	been taking them a while because internally Adobe shares the same libraries
	among all their products.

	Long ago AE had problems accessing files on more than one network volume..
	not sure if that's still the case, but I believe that was an OS9 limitation
	as well. Might be related.

> - The aerender binary does not seem to return its error number consistently
>   on exit; I've had batches register as Done even when they've failed in the
>   frame logs, despite what is in my submit script. 

	Can you include a copy of the complete log from one of these frames?
	Then I can help show you what commands to add to the submit script
	logic so you can catch those errors and handle them properly.

> Do you know of another way
> or do I simply have to capture the stdout and search for an error message?

	stdout/err capture is the only other way, and there's a function
	built into the scripts called LogCheck() which lets you specify
	one or more regex strings that searches the logs, correctly skipping
	over retry attempts.

> - When I fail or dump a batch to re-que it, it doesn't kill the aerendercore
>   process and I have to kill it on the individual machine. This is probably
>   the most serious problem as often even killing aerendercore isn't enough.

	Yes, this is really annoying, details on that were posted
	here on the group a few months ago here:
	http://seriss.com/cgi-bin/rush/newsgroup-threaded.cgi?-view+1906+1906

	This cropped up because Adobe made a change in CS3 that causes the
	render process to disconnect itself from the process hierarchy,
	effectively orphaning the process.

	Adobe is aware of the problem as of November 2009, and is trying
	to fix it. I worked directly with the AE devs on that issue.

	The AE engineers came up with a javascript mod as a temporary
	workaround to solve the problem; I can try to follow up with
	those engineers through myself on your behalf; contact me off list.

	This can be replicated from the command line as well by just
	running aerender interactively, and then killing it.

	You don't have to reboot to fix the problem, you can do something
	like a 'killall' command as root to kill the aerendercore and
	associated processes.

	Until Adobe fixes this, all render queue systems and distributed scripting
	of AE is affected by this.

>   The kill results in a zombie process that makes it impossible to run another
>   render on the machine until you do a "reboot -q" on it.

	If you look carefully at the process hierarchy, you can determine which
	processes need to be killed.. there's more than one.

	Mac's ps(1) command unfortunately doesn't seem to have an equivalent of
	the linux 'ps -fax' command, which shows the process hierarchy as a 'tree'
	so you can see what's parented to what.

	I do, however, have a script on my homepage that does the equivalent
	on the mac (and in fact, on most all unix platforms):
	http://seriss.com/people/erco/unixtools/piss

	..which if you run it, will show you the process tree on a Mac
	so you can see which processes are parented to what, and will help you
	determine what needs to be killed to free up the box. Example report:
	http://seriss.com/people/erco/unixtools/sample-piss.html

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

On 2010-06-17 21:49:08 -0400, Greg Ercolano said:

Yes, this is really annoying, details on that were posted
	here on the group a few months ago here:
	http://seriss.com/cgi-bin/rush/newsgroup-threaded.cgi?-view+1906+1906

	This cropped up because Adobe made a change in CS3 that causes the
	render process to disconnect itself from the process hierarchy,
	effectively orphaning the process.

	Adobe is aware of the problem as of November 2009, and is trying
	to fix it. I worked directly with the AE devs on that issue.

	The AE engineers came up with a javascript mod as a temporary
	workaround to solve the problem; I can try to follow up with
	those engineers through myself on your behalf; contact me off list.

	This can be replicated from the command line as well by just
	running aerender interactively, and then killing it.

	You don't have to reboot to fix the problem, you can do something
	like a 'killall' command as root to kill the aerendercore and
	associated processes.

	Until Adobe fixes this, all render queue systems and distributed scripting
	of AE is affected by this.

Brett Buddin wrote:
> On 2010-06-17 21:49:08 -0400, Greg Ercolano said:
> 
>>      Yes, this is really annoying, details on that were posted
>> 	here on the group a few months ago here:
>> 	http://seriss.com/cgi-bin/rush/newsgroup-threaded.cgi?-view+1906+1906
>>
>> 	This cropped up because Adobe made a change in CS3 that causes the
>> 	render process to disconnect itself from the process hierarchy,
>> 	effectively orphaning the process.
>>
>> 	Adobe is aware of the problem as of November 2009, and is trying
>> 	to fix it. I worked directly with the AE devs on that issue.
>>
>> 	The AE engineers came up with a javascript mod as a temporary
>> 	workaround to solve the problem; I can try to follow up with
>> 	those engineers through myself on your behalf; contact me off list.

> Have you received any updates as to when, or if, this might be 
> corrected in CS5? It appears that the issue is still around.

	No, I haven't. The Javascript mod they offered is the only one
	I've seen so far.

> If not, what sort of measures have you implemented to determining that 
> renders have actually completed (when you receive false negatives or 
> positives from aerender)?

	Renders finish OK, the problem only occurs when you try to
	stop a render while it's running (eg. requeuing busy frames,
	or dumping jobs that have busy frames).

	To that, there's really no answer except to 'End' these
	jobs instead of 'Dump' them, and not requeue frames that are busy.

	There's no technical solution I'm aware of that rush can do;
	it really has to be fixed on the Adobe side, due to how the
	processes become so completely disconnected that it's fairly
	impossible to tell which process is associated with which.

> So far, I've been watching the status 
> returned by the aerender process and monitoring the log file it creates 
> to determine when the render is complete, but that doesn't seem to be 
> completely reliable.

	Right, there's no good solution.

	The only one I can think of is to somehow use the TCP socket
	connections between aerender and AfterFX to determine which
	process goes with which, but that's very unreliable too.

	The javascript solution Adobe offered sounded like the
	right direction to go, where the main program watches
	aerender to see if it disappears, and if so, stops the main
	application right away.

	They said they weren't completely happy with that, and there's
	probably a cleaner way to do it, and they were investigating this.

> We're running multiple instances of aerender on a single machine so a 
> "killall" against aerendercore, when it hangs, isn't an option.

	Right. No clear solutions externally to AE seem possible.
	It has to be fixed from within.

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Unfortunately I've already overwritten the bad frame logs from today's tests. On the off chance that it will be of use you'll find a successful frame log at the end of this message.

To me it looks as if AfterEffects is bound to the static location of the Mac's /Library as well as the intended install locations of some if not all of the other pieces of the CS5 suite itself, so it may not be possible at all to have a central library.

Using killall on aerendercore or aerender so far has not solved the zombie problem, so this issue may have gotten worse in CS5 from what you've described. One aspect of MacOS X that frustrates people who've used other unix platforms is that there are situations in which even kill -9 as root will not terminate a stuck process. This is particularly true if you have an OpenGL or graphics driver error. You'd think having created OpenCL that Apple would finally want to do something about that...

In the mean time I'll give the LogCheck function a try.

-Dan


###
### neildiamond.333: 0006
###
--------------- Rush 102.42a9d --------------
--       Host: r32
--        Pid: 275
--      Title: render_test5
--      Jobid: neildiamond.339
--      Frame: 0006
--      Tries: 0
--      Owner: dbrowne (1089/20)
--  RunningAs: dbrowne (1089/20)
--   Priority: 1
--       Nice: 10
--     Tmpdir: /var/tmp/.RUSH_TMP.6
--    LogFile: /Casino/Poker/Shows/frd/frdCOMMON/frdDEV/gfx/scripts/dennisw/orDev_typeExploration_037-test.aep.log/0006
--    Command: perl /EEP/Tools/Settings/rush/rushscripts_v102.42a9/submit-afterfx.pl -render CS5 /Casino/Poker/Shows/frd/frdCOMMON/frdDEV/gfx/scripts/dennisw/orDev_typeExploration_037-test.aep origamiMAINAssemblev3EDIT /Casino/Poker/Shows/frd/frdCOMMON/frdDEV/gfx/images/Renders/rush_tests/origami_MAIN_Assemble_v3_EDIT_[####].tif 5 100 3 Fail 0 off
--    Started: Thu Jun 17 18:02:06 2010
--------------------------------------------
Possible unintended interpolation of @10 in string at /EEP/Tools/Settings/rush/rushscripts_v102.42a9/submit-afterfx.pl line 83.

	  SCENEPATH: /Casino/Poker/Shows/frd/frdCOMMON/frdDEV/gfx/scripts/dennisw/orDev_typeExploration_037-test.aep
	       COMP: origami MAIN Assemble v3 EDIT
	  OUTPUTDIR: /Casino/Poker/Shows/frd/frdCOMMON/frdDEV/gfx/images/Renders/rush_tests/origami_MAIN_Assemble_v3_EDIT_[####].tif
	  AEVERSION: CS5
	RENDERFLAGS: 
	BATCHFRAMES: 5 (6-10)
	    RETRIES: 3 (Fail after 3 retries)
	       PATH: /Network/Library/Fonts:/Applications/Adobe After Effects CS5:/Applications/Adobe After Effects CS5:/Applications/Adobe After Effects CS5:/Applications/Adobe After Effects CS5:/Applications/Adobe After Effects CS5:/usr/local/rush/bin:/usr/local/rush/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/rush/bin

Executing: aerender -mem_usage 60 120 -mp -project "/Casino/Poker/Shows/frd/frdCOMMON/frdDEV/gfx/scripts/dennisw/orDev_typeExploration_037-test.aep" -output "/Casino/Poker/Shows/frd/frdCOMMON/frdDEV/gfx/images/Renders/rush_tests/origami_MAIN_Assemble_v3_EDIT_[####].tif" -comp "origami MAIN Assemble v3 EDIT"  -s 6 -e 10 
Thu Jun 17 18:02:12 r32.eep.com aerender[279] <Warning>: 3891612: (connectAndCheck) Untrusted apps are not allowed to connect to or launch Window Server before login.
Thu Jun 17 18:02:13 r32.eep.com aerender[279] <Error>: kCGErrorFailure: Set a breakpoint @ CGErrorBreakpoint() to catch errors as they are logged.
_RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
PROGRESS:  6/17/10 6:02:19 PM PDT: Starting composition origami MAIN Assemble v3 EDIT.


PROGRESS:  Render Settings: Best Settings
PROGRESS:  Quality: Best
PROGRESS:  Resolution: Full
PROGRESS:  Size: 1920 x 1080
PROGRESS:  Proxy Use: Use No Proxies
PROGRESS:  Effects: Current Settings
PROGRESS:  Disk Cache: Read Only
PROGRESS:  Color Depth: Current Settings
PROGRESS:  Frame Blending: On for Checked Layers
PROGRESS:  Field Render: Off
PROGRESS:  Pulldown: Off
PROGRESS:  Motion Blur: On for Checked Layers
PROGRESS:  Use OpenGL: Off
PROGRESS:  Solos: Current Settings
PROGRESS:  Time Span: Custom
PROGRESS:  Start: 00006
PROGRESS:  End: 00010
PROGRESS:  Duration: 00005
PROGRESS:  Frame Rate: 29.97 (comp)
PROGRESS:  Guide Layers: All Off
PROGRESS:  Skip Existing Files: On
PROGRESS:  
PROGRESS:  Output Module: TIFF Sequence with Alpha
PROGRESS:  Output To: /Casino/Poker/Shows/frd/frdCOMMON/frdDEV/gfx/images/Renders/rush_tests/origami_MAIN_Assemble_v3_EDIT_[####].tif
PROGRESS:  Format: TIFF Sequence
PROGRESS:  Output Info: -
PROGRESS:  Start Frame: 6
PROGRESS:  Output Audio: -
PROGRESS:  Channels: RGB + Alpha
PROGRESS:  Depth: Millions of Colors+
PROGRESS:  Color: Premultiplied
PROGRESS:  Resize: -
PROGRESS:  Crop: -
PROGRESS:  Final Size: 1920 x 1080
PROGRESS:  Profile:  -
PROGRESS:  Embed Profile:  
PROGRESS:   
PROGRESS:  Post-Render Action: None
PROGRESS:   
PROGRESS:   
PROGRESS:   
PROGRESS:  
PROGRESS:  00006 (1): 1 Seconds
PROGRESS:  00007 (2): 1 Seconds
PROGRESS:  00008 (3): 1 Seconds
PROGRESS:  00009 (4): 1 Seconds
PROGRESS:  00010 (5): 0 Seconds
PROGRESS:  6/17/10 6:02:25 PM PDT: Finished composition origami MAIN Assemble v3 EDIT.



PROGRESS:  Total Time Elapsed: 6 Seconds
aerender version 10.0x458
PROGRESS: Launching After Effects...
PROGRESS: ...After Effects successfully launched

--- AERENDER SUCCEEDS

Daniel Browne wrote:
> Unfortunately I've already overwritten the bad frame logs from today's test=
> s. On the off chance that it will be of use you'll find a successful frame =
> log at the end of this message.

	I have plenty of those ;)

	Need the text with the actual error to offer you a good regex
	to search for.

	You can see the usage of LogCheck() by doing a grep on all the submit
	scripts for LogCheck, ie. 'grep LogCheck /path/to/your/rushscripts/*.pl'

	The code for that function is defined in .common.pl, and can take
	one or more regex's as an array, so that you can search for several
	strings in one shot. It also has a separate string that resets the
	search, useful for it to skip over retries.

> To me it looks as if AfterEffects is bound to the static location of the Ma=
> c's /Library as well as the intended install locations of some if not all o=
> f the other pieces of the CS5 suite itself, so it may not be possible at al=
> l to have a central library.

	I don't suppose symbolic links would help? ie. make a symlink(s)
	in the /Library/xxx directory where the extra fonts should go.
	Might be worth a shot.

> Using killall on aerendercore or aerender so far has not solved the zombie =
> problem, so this issue may have gotten worse in CS5 from what you've descri=
> bed.

	Since zombies aren't really processes, they're just the 'exit code'
	of the process, you don't really have to worry about them too much
	unless there's a lot of them.

	Killing them is useless, because they're already dead,
	just like in the movies ;)

	Zombies are created when the parent doesn't reap the exit code,
	which probably means the parent is still around somewhere.

> One aspect of MacOS X that frustrates people who've used other unix pl=
> atforms is that there are situations in which even kill -9 as root will not=
>  terminate a stuck process.

	This actually can happen on all unix platforms.. I used to run
	into this constantly on SGI and linux in the 90's (and to date)
	whenever a process is reading or writing to a hard mounted NFS share
	where the server has gone down or is not responding.

	The hanging process part is OK, you want it to do that, but if
	root comes along with kill -9, it SHOULD die, but it often never
	does. This is because the process is stuck in kernel mode, where
	signals aren't processed immediately the way they are in user mode.
	It can be done, but it's tricky/messy to do in kernel code at all
	the points where a hang could happen, so often processes stuck in
	kernel mode are unkillable.

	That would go for any kernel or driver operation that is 'stuck'
	for some reason, like tape operations or floppy drives...
	other situations where I've seen this occur.


> This is particularly true if you have an OpenGL=
> or graphics driver error. You'd think having created OpenCL that Apple wou=
> ld finally want to do something about that...

	It's all to do with the driver code and how well it's written
	to handle pending signals. It's very tricky to properly detect
	and unravel a kernel mode call that's got pending signals and
	have it react like a user mode operation.

> In the mean time I'll give the LogCheck function a try.

	Yes; see the usage in other submit scripts (like submit-maya,
	submit-nuke and submit-lightwave), and the definition of the
	function in .common.pl


-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)