From: "Bryan Blevins" <bblevins@(email surpressed)>
Subject: How do I submit jobs with a staggered start?
   Date: Sun, 23 Oct 2005 10:07:32 -0700

Hi all!

Currently, when we submit a job with rush.  All the nodes pick up the job at 
the same time, and kill my little ole server.  All the nodes are fighting 
for data all the way through the render.  What I've been doing is to 
manually online the nodes one at a time, waiting about a minute between 
each.  It works great for my server, and the renders go smooth, but is such 
a pain to do.

Is there a way in Rush to have a slight pause (say 30-60secs) between the 
que frames so that my server can dish out the data to the nodes more evenly?

Thanks

Bryan Blevins - CG Director - Kachew! 

Bryan Blevins wrote:

[posted to rush.general]

Hi all!

Currently, when we submit a job with rush. All the nodes pick up the job at the same time, and kill my little ole server. All the nodes are fighting for data all the way through the render. What I've been doing is to manually online the nodes one at a time, waiting about a minute between each. It works great for my server, and the renders go smooth, but is such a pain to do.

Is there a way in Rush to have a slight pause (say 30-60secs) between the que frames so that my server can dish out the data to the nodes more evenly?

	You can add some stagger code to your submit script so that it
	sleeps a little before starting the render.

	I'd recommend making the amount of sleep time a function of the
	frame number, with a cut off above a certain frame number, though
	just as easily it could be some random value.

	For instance, if you want the first 20 frames to have increasingly
	longer sleep times, you could add these lines to the -render section
	of the script:

----
if ( $ENV{RUSH_FRAME} < 50 )
    { sleep($ENV{RUSH_FRAME} * 2); }
----

	..which will cause frame 1 to sleep 2 seconds, frame 2 to sleep 4 secs,
	frame 10 to sleep 20 seconds, etc.

	Or, you can have frames sleep in blocks:

----
if ( $ENV{RUSH_FRAME} < 50 )
    { sleep( int( $ENV{RUSH_FRAME} / 10 ) * 30 ); }
----

	..where frames 1-9 won't sleep at all, 10-19 will sleep 30 secs,
	20-29 will sleep 60 seconds, etc.

	Or, a possibly better approach is to embed an 'rsh' command in the script
	that polls the server for its current load, and if the load is above some value,
	sleep some amount of time and check again. (below)

	Note: the following code is not tested, but should give you an idea
	on how to approach it. Basically you want to parse out the first load value
	from 'uptime', and if it's above some number you find to be the max load
	you want your server to be, this will cause the render to sleep until the
	load returns to normal.

----
# WAIT FOR SERVER'S LOAD TO GO BELOW SOME MAXIMUM
while ( 1 )
{
    my $max_load = 3;                   # change to taste

    # GET CURRENT LOAD FROM SERVER
    my $server_load = `rsh myserver uptime`;
    if ( $server_load =~ /load averages: (\d+)/ )
    {
        $server_load = $1;
        print "--- SERVER LOAD IS $server_load (MAX=$max_load): ";

        # IF LOAD VALUE LOW ENOUGH, START RENDER
        if ( $server_load <= $max_load )
            { print "OK\n"; last; }
    }
    else
        { print "--- ERROR GETTING SERVER'S LOAD: '$server_load': "; }

    # WAIT AND TRY AGAIN
    print "sleep 20\n";
    sleep(20);
}
----

--
Greg Ercolano, erco@(email surpressed)
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Cel: (Tel# suppressed)
Fax: (Tel# suppressed)

Thanks for the response.  I'm intrigued by the sleep in blocks suggestion. 
As I'm not a programmer/scripter type, how do I go about adding that bit 
when I submit a job?  I tried a simple cut and paste into the 'submit 
options' area of our Maya submit script, but just get an error.

I've forwarded your response to our IT supervisor, Jason in hopes that he 
knows what to do, but if it is simple enough I'd like to add it to my job 
submits now.

>
> You can add some stagger code to your submit script so that it
> sleeps a little before starting the render.
>
> I'd recommend making the amount of sleep time a function of the
> frame number, with a cut off above a certain frame number, though
> just as easily it could be some random value.
>
> For instance, if you want the first 20 frames to have increasingly
> longer sleep times, you could add these lines to the -render section
> of the script:
>
> ----
> if ( $ENV{RUSH_FRAME} < 50 )
>     { sleep($ENV{RUSH_FRAME} * 2); }
> ----
>
> ..which will cause frame 1 to sleep 2 seconds, frame 2 to sleep 4 secs,
> frame 10 to sleep 20 seconds, etc.
>
> Or, you can have frames sleep in blocks:
>
> ----
> if ( $ENV{RUSH_FRAME} < 50 )
>     { sleep( int( $ENV{RUSH_FRAME} / 10 ) * 30 ); }
> ----
>
> ..where frames 1-9 won't sleep at all, 10-19 will sleep 30 secs,
> 20-29 will sleep 60 seconds, etc.
>

Bryan Blevins wrote:

Thanks for the response. I'm intrigued by the sleep in blocks suggestion. As I'm not a programmer/scripter type, how do I go about adding that bit when I submit a job?

	It's a 'scripter type' of change that would involve modifying the
	submit script you're using in a text editor.

	If you feel up to it, you can make a copy of the submit script
	on your file server, and then edit the copy, and use that for
	submitting your current job.

	Or, if you email me the submit script you're using as an attachment,
	I can mail back the modified script which you can save under a different
	name on your file server, and use that for submitting the jobs.
	(My address is erco <at-sign> seriss <.> com )

	BTW, this same question came up on the group as a thread last year:
	http://seriss.com/cgi-bin/rush/newsgroup-threaded.cgi?-viewthread+585+602+603+605+606+613+604

	I would echo Steve Kochak's observation in that thread that this is
	likely symptomatic of a file server that needs to either be tuned
	or replaced with a better box. It sounds like you're skating the edge
	of your file server's ability to handle the load of the number of
	machines you have for network rendering.

	Until tuned or replaced, you're very likely to overload your box
	at random times, depending on how I/O intensive your renders are.
	Possibly too, a throttle can be used at the switch to control the
	load to the server, so that the workstation's requests for data
	from the file server are higher priority than your render farm machines.

	The amount of productivity lost to users waiting for the file server
	often vastly outweighs the cash outlay for purchasing a better server.

	As most folks here know, I maintain there's a magic number of about
	30 or 40 render nodes that can be served by a 'PC based file server'
	before you max it out.

	More than 30 or 40 machines, I strongly recommend the company get a
	'real' file server like a NetApp or BlueArc, which can handle 100's
	of machines rendering. Possibly other products are up to the task
	(I know NetApps and BlueArcs are).. I've heard of folks evalling
	the Max-T Sledgehammer. I've also heard some companies have success
	with Sun and SGI servers, which have better load balancing and more
	robust I/O abilities.

	(It's quite possible a PC based file server can be 'tuned' into handling
	load for >40 hosts, but this surely involves the assistance of the vendor,
	as tuning often involves extreme knowledge of the OS in question,
	and often such knowledge is specific to particular revs of the OS)

I tried a simple cut and paste into the 'submit options' area of our Maya submit script, but just get an error.

	Yes, that won't work.

	The 'Submit Options:' prompt is only for adding rush 'submit commands':
	http://www.seriss.com/rush-current/rush/rush-submit-cmds.html#Submit%20Command%20Reference

--
Greg Ercolano, erco@(email surpressed)
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Cel: (Tel# suppressed)
Fax: (Tel# suppressed)