|> Because of the size of some of our renders, setting them all off a
|> once is putting quite a load on our file server and slowing
|> workstations. Is it possible to set an advanced submit
|option to delay
|> each nodes frame start time?
|
| Trouble is, even if you stagger, eventually once all the
| machines are rendering, they'll all be hitting your server.
I think the main problem we where seeing with big jobs is when all the
render nodes try to load the same data at exactly the same time, this is
when the server hits 100%. Even if we could separate them drawing data by a
few seconds it should help. I agree that this is only likely to help in
situations where actual render times are long compared to server I/O. If a
render takes only a short amount of time to render, it's likely to have
little effect I agree.
|
| If you've got a network large enough to where it slows your
| server down, you may want to consider some kind of network
| flow control at the switches, so that the workstations have
| higher priority to the network than the farm.
I'll look at this, I've been meaning to look at tuning the switches anyway,
and they have got some pretty cool looking features.
|
| To stagger start, you could any one of several things..
| just to give you an idea:
|
| o Submit with the 'maxcpus' set to '1', then
|slowly advance
| the maxcpus value with an external script:
|
| #!/bin/csh -f
| set jobid = $1
| # Every 10 seconds, add two cpus.
|Eventually, removes maxcpus limit
| foreach i ( 2 4 8 10 0 )
| rush -maxcpus $i $jobid -fu
| sleep 10
| end
|
| ..or build this behavior into the submit script.
|
| o Ramp up sleep times in the render script for
|first 20 frames:
|
| if ( $frame <= 20 ) { sleep($frame*5); }
|
| o Enter a loop that polls the server's uptime
|before starting
| a frame to render, eg. in perl:
|
| while ( 1 )
| {
| # Parse output of 'uptime' to get
|load average
| # If load average is high (>8),
|sleep until it gets lower
| my $load=`rsh SERVER uptime | sed
|'s/.*load averages: //;s/ .*//';
| if ( $load > 8.0 ) { sleep(10); next; }
| last;
| }
|
This is cool, I would love to get this working off the CPU load.... I'll let
you know how I go, and uptime should report load fast enough.
|