From: Greg Ercolano <erco@(email surpressed)> Subject: [Q+A] Passing environment variables to render scripts Date: Wed, 01 Dec 2004 11:29:25 -0800 |
Msg# 761 View Complete Thread (1 article) | All Threads Last Next |
> We depend heavily on environment variables for our production pipeline. > It seems that when I submit a job the environment from the submitting > shell isn't available at rendertime (I'm just doing print(%ENV, "\n") > in place of the actual rendering in the submit-renderman.pl script). Correct. Rush runs the renders with a 'bare' environment; it is up to the render script to set or gather up whatever environment it needs to run the renders. There are two common techniques: 1) Set the variables in the render script. This way the script can automatically determine the local OS, and set the variables appropriate for that platform, eg: if ( -d "/windows" ) { $ENV{PATH} = "C:/foo/bar;$ENV{PATH}"; ### WINDOWS } elsif ( -d "/Applications" ) { $ENV{PATH} = "/Applications/foo:$ENV{PATH}"; ### OSX } else { $ENV{PATH} = "/usr/local/bin:$ENV{PATH}"; ### LINUX } 2) Source the environment from centrally maintained rc files. For instance, a render script that uses MAYA and MTOR: #!/bin/csh -f # # RENDER SCRIPT # source /usr/sohovfx/adm/.maya # maintained by sohovfx admins source /usr/sohovfx/adm/.mtor # maintained by sohovfx admins maya -render .. The reason Rush does not send the local environment by default is because a) it would pollute the remote environment with variables that are often not relevant on the remote machines (esp in cross platform environments), and b) it makes handoffs between users buggy; eg. a TD hands off the job to a render watcher, where the jobs submit fine if submitted by the TD, but not by the render watcher because of differences in environments. Or, a TD hands off their rendering to another TD with a different environment. Also, most companies invoke the example submit script GUIs from their desktop, which has no useful environment context. Rush definitely does not do /anything/ with .cshrc or .login files. The only way those could come into play is if you write your own CSH render scripts, and leave out the -f flag from the #!/bin/csh line, causing the user's .cshrc to be sourced. I DONT advise this; definitely specify the -f flag for csh scripts. > Is there an easy way to get the environment into the render context? Yes, as you mention, you can write the environment out from your submit script to eg. a ".env" file in the "log directory" (unique to each render), then source it into the render script using $RUSH_LOGFILE to 'find' the .env file. Another way is to pass simple settings (such as show/shot/sequence info) from the submit script as arguments to the render script, eg: #!/bin/csh -f # # SUBMIT SCRIPT # echo SHOW is $SHOW echo SHOT is $SHOT rush -submit << EOF : command /server/scripts/my-render-script $SHOW $SHOT : ^^^^^ ^^^^^ EOF $1 $2 ..then in the render script, pick up the arguments: #!/bin/csh -f # # RENDER SCRIPT # setenv SHOW $1; echo SHOW is $1 setenv SHOT $2; echo SHOT is $2 : > This is a very useful feature of [other render queue systems] It is not something I have Rush do by default, because it makes problems for handoffs between users (eg. for render watchers or people or handoff before going on vacation), and makes cross platform rendering impossible. Also, most company's user environments are messy, and inconsistent.. they don't port well even across similar platforms. My feeling was, any company that has their user environment together is very aware of technical issues, and has their own custom way of dealing with things, in which case they can deal with passing around specific environment settings, and would probably want full control over that logic. So, to keep things simple, rush's environment is very bare bones, and it's up to the render script to set up whatever is needed (eg. maya specific variables like PATH, PLUG_IN_PATH, LD_LIBRARY_PATH, etc etc) > suspect that I could write the environment out to a file in the > .submit file and then just read it back in in the .render file > (am I correct about that?) but I'd hope for an automatic solution. Correct, though I don't advise doing it wholesale; some variables don't port well, even across machines of the same platform. ($DISPLAY would be one) Usually, either you want to pass a little data, like SHOW/SHOT info, in which case pass these as arguments to the render script. Or, if you want to pass large amounts of data, use a file written out the the render's log directory.. a safe place for data that needs to be passed to all the machines. eg. in a csh script, you could save/load the environment this way: # SUBMIT SCRIPT printenv | sed 's/^/setenv /;s/=/ "/;s/$/"/' > $logdir/.env # RENDER SCRIPT source $logdir/.env > Note that simply sourcing .cshrc files when the render starts is > not an option because as the user works on different shots the > environment changes (ie the environment at submit time is most > certainly different from the environment at login time). If you have logic based on show/shot variables, then pass the show/shot variables to the render script, and have the render script then run the logic that sets the rest of the variables appropriately at render time. Assuming you have things modularized, where 'show' and 'shot' are possibly aliases that set variables, then sources a script that sets other environment variables based on some logic, put this into your render script, eg. #!/bin/csh -f # # RENDER SCRIPT # Expects $1 to be the show name, and $2 to be the shot # source /usr/sohovfs/adm/.stdenv # configures 'show' and 'shot' aliases show $1 # invokes 'show' alias shot $2 # invokes 'shot' alias # At this point, environment is 'configured' maya -render .. I've worked at a lot of companies, both well organized and disorganized, and I designed rush to work for both. |