From: "Mark Smith" <Mark.Smith@capital-fx.co.uk>
Subject: RE: Using Rush in a slightly different way [for Framecycler]
   Date: Tue, 20 Sep 2005 10:57:04 -0700

Thanks for the Python example...

I've had a quick look and it is all making more sense already. I am away
till next week now but will let you know of my progress when I get back.

Interesting to see your views on Python. I haven't spent all that much
time with 'real' scripting languages. I can see from your examples how
many tasks can be made simpler with them. I think I'll have a look at
csh real soon....any good books you can recommend?

Maybe it's about getting the right mix of things. At my last place much
of the core pipeline code was Python....this could be the reason the
submission scripts were in Python too.

I see from the Rush documentation that you can source render
environments in csh but I'm not too sure it would be too easy to do if
you were creating your render environments by sucking things out of a
database. Or maybe I'm wrong?


 
Mark Smith
IT Engineer
Capital FX
20 Dering Street
London W1S 1AJ
United Kingdom
 
T: +44 (0)20 7493 9998
F: + 44 (0)20 7493 9997
-----Original Message-----
From: Greg Ercolano [mailto:erco@(email surpressed)] 
Sent: 19 September 2005 18:59
To: void@(email surpressed)
Subject: Re: Using Rush in a slightly different way [for Framecycler]

>>> Is there a Python version of the submit-generic script?
>> 
>>     No, and I'm not sure a python version of submit-generic
>>     would be as helpful as a python version of one of the other
scripts.
>> 
>>     What I might do is make a simple Python example script
>>     that shows how to submit a job through Rush.
> 
> A simple Python submit script would be brilliant...I'm afraid Perl
sends
> shivers up my spine.

     I converted the "rush/examples/input-example.pl" program into
python,
     and attached it here.

     There's already a .csh and .pl version of the script, so a .py file
     rounds it out nicely ;)

     I'm glad I did this experiment -- it made me look closely at
Python.

     I'm still new to it, so I know my knowledge is limited so far,
     but I'm pretty sure I /won't/ be thinking 'gee, I should port
     all the submit scripts to Python..!" ..it's not a good fit IMHO.

     Though Python's capable, Python seems to be more of a programming
language
     than a scripting language.. ie. it's more like C++ than CSH or
Perl;
     it sticks to strict programming guidelines (which is good, if
writing programs),
     but lacks syntax that makes /scripting/ pleasant.

     For instance, I miss the ability to 'easily' test for the existence
of a file,
     something scripts often do:

         PERL                           PYTHON
	if ( -e "/some/file" )         if ( os.path.exists("/some/file")
):
	if ( ! -e "/some/file" )       if ( not
os.path.exists("/some/file") ):

     Also, the language is so OOP-y, it's like having to specify
absolute paths
     to everything.. makes for redundancy that clutters up code quickly:

	if ( os.path.isdir(foo) ): ..
	if ( os.path.isdir(bar) ): ..
	if ( os.path.isdir(bla) ): ..

     ..all that "os.path" stuff seems redundant readability wise,
     harder to read than the csh/perl equivalent:

	if ( -d $foo ) ..
	if ( -d $bar ) ..
	if ( -d $bla ) ..

     I also miss being able to easily embed variables in strings,
     making commands with lots of arguments harder to read in Python:
	
	$render -f $somefile $sfrm $efrm
# Csh
	system("$render -f $somefile $sfrm $efrm");
# Perl
	os.system(render+" -f "+somefile+" "+str(sfrm)+" "+str(efrm))
# Python

     ..all those (")s and (+)s and str()s make scripting way more
difficult
     than it needs to be.

     Also, I'd really miss being able to do assignments within
conditionals
     something I do often in Perl and C:

         PERL/C                               PYTHON
	while ( $foo = ReadStuff() ) ..      while (1): foo =
ReadStuff(); if ( not foo ): break; ..
	if ( $foo = IsOkay() ) ..            foo = IsOkay(); if ( foo ):
..

     And all those icky ':'s on the end of if/else/while
     seem unnecessary if line breaks are present. In a language where
     white space is significant, I think it would help here.

     I don't mind the whitespace indenting thing as much as I thought I
would,
     and not having to specify $ in front of variables is nice, though
not nice enough
     to loose the ability to embed variables in strings. Would be neat
if Python
     would allow '$'s to be used to expand variables in strings, instead
of that
     icky "%(var)s" syntax.

Mark Smith wrote:

Interesting to see your views on Python. I haven't spent all that much
time with 'real' scripting languages. I can see from your examples how
many tasks can be made simpler with them. I think I'll have a look at
csh real soon....any good books you can recommend?

	CSH is good for simple things.. but runs out of steam quickly
	when you want to do complex text manipulations or subroutines
	as I do in my gui submit scripts.

	But csh is great for making simple command line oriented submit
	scripts, though you often end up having to use external programs
	like sed(1) to do text manipulations, and awk(1) to do things
	like padded zeroes. Or even perl(1) which can do all of those things.

	CSH/TCSH is good to know just for bopping around on the command line..
	my favorite book on learning CSH is this one:
	http://www.amazon.com/exec/obidos/tg/detail/-/013937468X/qid=1127247128/sr=1-2/ref=sr_1_2/002-8701294-1017638?v=glance&s=books

Maybe it's about getting the right mix of things. At my last place much
of the core pipeline code was Python....this could be the reason the
submission scripts were in Python too.

	Python is fine; if you know it, use it.

	The items I cited are not show stoppers at all.. they're just
	nits from an old scripting hound who's picky.

	If all the scripts were already in python, I probably wouldn't
	port them to perl just to make them a little easier to read for me. ;)

I see from the Rush documentation that you can source render
environments in csh but I'm not too sure it would be too easy to do if
you were creating your render environments by sucking things out of a
database. Or maybe I'm wrong?

	You should be able to do anything.

	For instance, to load a csh environment into perl, this simple
	bit of magic spawns a csh to load the settings, then prints them
	out in a format that perl can read back and load into its own environment:

# LOAD A CSH SCRIPT'S ENVIRONMENT SETTINGS INTO PERL ENVIRONMENT
#    $1 = csh script to be 'sourced'
#    Returns: $ENV{} modified as per csh script's settings.
#
sub LoadEnvFromCsh()
{
    my ($rcfile) = @_;
    my $vars = `csh -fc 'source $rcfile; printenv'`;
    foreach ( split(/\n/, $vars) )
        { if ( /(^[^=]*)=(.*)/ ) { $ENV{$1} = $2; } }
}

	And here's a function for loading Bourne Shell (sh, bash)
	style settings from e.g. /etc/profile or /etc/bashrc files:

# LOAD A BASH SCRIPT'S ENVIRONMENT SETTINGS INTO PERL ENVIRONMENT
#    $1 = bash profile to be 'sourced'
#    Returns: $ENV{} modified as per bash profile's settings.
#
sub LoadEnvFromBash($)
{
      my ($rcfile) = $_[0];
      my $vars = `bash -c '. $rcfile; printenv'`;
      foreach ( split(/\n/, $vars) )
          { if ( /(^[^=]*)=(.*)/ ) { $ENV{$1} = $2; } }
}

	I'm sure a similar technique can be done in python.


	Even if the csh is getting settings from a 'database',
	it wouldn't matter. Once they're in the environment,
	they can be accessed with the above technique.

	But I recommend against having render scripts source in
	user .cshrc files. Global .cshrc files are ok, but user .cshrcs
	tend to have errors in them, and are inconsistent from user to user,
	machine to machine.

	Better to have the sysadmin (you?) configure the environment settings
	in the submit script, so that the script works the same for everyone,
	no matter how broken their .cshrc files are ;)



--
Greg Ercolano, erco@(email surpressed)
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Cel: (Tel# suppressed)
Fax: (Tel# suppressed)

For instance, to load a csh environment into perl, this simple
bit of magic spawns a csh to load the settings, then prints them

out in a format that perl can read back and load into its own environment:

# LOAD A CSH SCRIPT'S ENVIRONMENT SETTINGS INTO PERL ENVIRONMENT
#    $1 = csh script to be 'sourced'
#    Returns: $ENV{} modified as per csh script's settings.
#
sub LoadEnvFromCsh()
{
    my ($rcfile) = @_;
    my $vars = `csh -fc 'source $rcfile; printenv'`;
    foreach ( split(/\n/, $vars) )
        { if ( /(^[^=]*)=(.*)/ ) { $ENV{$1} = $2; } }
}

    I'm sure a similar technique can be done in python.

    In a true re-enactment of the blind leading the blind,
    here's a stab at the python equivalent:

--- snip

import os

# LOAD CSH ENVIRONMENT SETTINGS INTO THE CURRENT PYTHON ENVIRONMENT
#    file -- csh script to be 'sourced'
#    Returns: os.environ[] modifed as per csh script's settings
#
def LoadCSHEnvironment(file):
    fp = os.popen("csh -fc 'source "+file+"; printenv'", "r")
    for line in fp.xreadlines():
        line = line.replace("\n","")            # foo=bar\n -> foo=bar
        x = line.find("=")
        var = line[:x]                          # foo=bar -> foo
        val = line[(x+1):]                      # foo=bar -> bar
        os.environ[var] = val                   # set environment variable
    fp.close()

--- snip

    There's probably a way to make that a few lines shorter;
    I'm not too familiar with Python's string manipulation yet.

    Usage of the above might be, for instance:

--- snip

print "--- BEFORE:"
for var in os.environ:
    print "os.environ['%s'] = '%s'" % (var, os.environ[var])

LoadCSHEnvironment("/net/erco/.cshrc")

print "--- AFTER:"
for var in os.environ:
    print "os.environ['%s'] = '%s'" % (var, os.environ[var])

--- snip