Rush Render Queue - Command Line Options (C) Copyright 1995,2000 Greg Ercolano. All rights reserved. V 102.42a10 04/10/09 Strikeout text indicates phased out or obsolete operators.

rush -ac

rush -ac <cpuspec..> [jobid..] Add Cpus. Adds cpus to the running job. Examples: Add Cpu Examples rush -ac tahoe@300 Add one cpu on tahoe at 300 priority rush -ac tahoe=2@300 +farm=10@100k Add two cpus on tahoe, 10 cpus for +farm hostgroup rush -ac +any=10@30 Add any 10 available cpus at priority 30 See the Cpus submit command for more info. To remove cpus, see 'rush -rc'.

rush -af

rush -af <framerange..> [jobid..] Add Frames. Adds frames to the current framelist. Adding frames that already exist yields an error. Example: Add Frames Examples rush -af 10-20 Add 10 thru 20 to the current job rush -af 1 5 10-100 800-899 Add frames 1, 5, 10 through 100 and 800 through 899 to the current job rush -af -10--5 Add negative frames: -10 thru -5 rush -af .1-5,.1 Add floating point frames .1 through 5 stepping by .1 (.1, .2, .3 .. 4.9, 5.0) rush -af 20-25=Hold Add 20 thru 25 in Hold state rush -af 30-35:"To be done" Add 30 thru 35, Notes "To be done" rush -af 40-45=Done:"To be done" Add 40 thru 45 as Done with Notes See Frames submit command for more info.

rush -an

rush -an <hostname|+hostgroup..> [jobid..] Add Nevercpus. Prevents the job from rendering on the specified hosts. Example: Add Never Cpus Examples rush -an tahoe +farm Add host tahoe and all hosts in hostgroup +farm as machines to never use. See Nevercpus submit command for more info.

rush -autodump

rush -autodump <off|done|fail|donefail> [jobid..] Automatic Dump. Configures job to automatically dump when there are no more frames to render. Autodump Examples autodump off Don't autodump; job remains when frames are DONE (default) autodump done Job dumps itself when all frames are DONE autodump fail Job dumps itself if job finishes with any FAIL frames autodump donefail Job dumps itself if all frames are DONE or FAIL See AutoDump submit command for more info.

rush -catlog

rush -catlog <file> [file..] [host..] (New in rush 102.42 and up) cat(1) one of the daemon's logs. View one of the various local [remote] daemon logs; access both new and old versions of rushd.log, cpu.acct, etc. 'file' can be any one of: Catlog File Options rushd.log View today's daemon log Orushd.log View yesterday's daemon log (O='Old') cpu.acct View current rush accounting log Ocpu.acct View last rotated (O='Old') accounting log mail.log View the mail.log (if enabled via smtp.debug) What follows are some examples of using 'rush -catlog'.. Catlog Examples rush -catlog rushd.log View the local rushd.log file rush -catlog rushd.log geneva View geneva's daemon log rush -catlog Orushd.log geneva View yesterday's daemon log from host 'geneva' rush -catlog cpu.acct View the local accounting log rush -catlog cpu.acct +any | cpu-utilization.pl -asciigraph Generate a cpu utilization report from all host's accounting logs Caveat: This command deprecates the older '-dcatlog' command.

rush -checkconf

rush -checkconf <filename> Check for errors in rush.conf file.. (Administrative) Checks a rush.conf file for errors. Returns an exit code indicating success or failure; 0=ok, 1=error(s) found. Example: Example CSH Script Using 'rush -checkconf' #!/bin/csh -f ### ### Script to edit the rush.conf file, and rdist it out ### set TMPFILE=/usr/tmp/rush.conf.$$ cp /usr/local/rush/etc/rush.conf $TMPFILE vi $TMPFILE rush -checkconf $TMPFILE if ( $? ) then echo Config file has errors set err=1 else foreach i ( tahoe superior erie ) echo --- $i rdist -c $TMPFILE ${i}:/usr/local/rush/etc/rush.conf end set err=0 endif rm -f $TMPFILE exit $err To use rsync(1), try replacing 'rdist -c' with 'rsync -avz' in the above examples.

rush -checkhosts

rush -checkhosts <filename|-> Checks a hosts file for errors. (Administrative) Returns an exit code indicating success or failure; 0=ok, 1=error(s) found. (New in rush 102.42a9 and up) If no filename specified, rush hosts file is assumed.

rush -command

rush -command 'cmd [args..]' [jobid..] rush -command [jobid..] < file Set Render Command. Changes the command that is executed every frame, normally set in your Submit Script by the Command. Example: If 'cmd [args]' are not supplied, the command will be read from stdin. This is useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into one single line. Command Examples rush -command '/bin/sleep 30' Set command to sleep 30 seconds per frame rush -command < file Set command to contents of 'file' on stdin rush -command '/job/run/myrender' tahoe.21 tahoe.22 Change command for two jobs Note that a full path to the command is normally required. Spaces are used to separate arguments, but the entire command and arguments must be quoted. See also the submit script docs on Command for more info. The command is usually a simple Render Script. Here are some example 3rd Party Render Scripts.

rush -cont

rush -cont [jobid..] Continue a Paused Job. Continues a job that was paused with rush -pause. Pausing a job no longer cancels 'Waitfor'; it was decided in 102.40h that the older behavior of having 'rush -cont' cancel 'WaitFor' was a bug, not a feature. To properly cancel WaitFor (to get the job running) is to use 'rush -waitfor -'.

rush -cp

rush -cp [jobid..] <.JobTid [..]> <@pri> rush -cp [jobid..] <cpuspec [..]> <=#cpus> Change Priority. Changes priority for the cpus specified by 'JobTid' to the new priority '<@pri>'. JobTid values (Job Task IDs) can be found in the 'rush -lc' report. Changing Priority of Cpus Priorities can be changed while cpus are busy. If a busy cpu's priority is changed, the busy task(s) inherit the new priority, which is used for all subsequent scheduling and priority battles. Example: Example of Changing Cpu Priorities % rush -lc CPUSPEC[HOST] STATE FRM PID JOBTID PRI ELAPSED +any=10@100[vaio] Busy 0007 1030 131 100 00:01:45 +any=10@100[vaio] Busy 0008 1031 132 100 00:01:45 +any=10@100[tahoe] Busy 0004 19829 133 100 00:00:05 +any=10@100[tahoe] Busy 0005 19831 134 100 00:00:05 % rush -cp .131 .132 @350k vaio.21 +any=10@100: priority was 100, now 350k vaio.21 +any=10@100: priority was 100, now 350k % rush -lc CPUSPEC[HOST] STATE FRM PID JOBTID PRI ELAPSED +any=10@100[vaio] Busy 0007 1030 131 350k 00:02:03 +any=10@100[vaio] Busy 0008 1031 132 350k 00:02:03 +any=10@100[tahoe] Busy 0004 19829 133 100 00:00:28 +any=10@100[tahoe] Busy 0005 19831 134 100 00:00:28 Caveats: Currently the CPUSPEC column does not reflect the change, though the effective priority (shown in the PRI column) does. GUIs (like IRUSH) make best use of this option, since they hide the mundane task of visually finding JOBTIDs from the -lc report, and transcribing them with dot prefixes for the -lc command. Changing Cpu Limits For Cpus 'rush -cp' can be used to change the number of cpus in a cpuspec. For instance, 'rush -cp +any=50@100 =40' will change '+any=50@100' to '+any=40@100' without killing any running frames. Example: Example of Changing Cpu Limits % setenv RUSH_JOBID vaio.36 % rush -lc CPUSPEC[HOST] STATE FRM PID JOBTID PRI ELAPSED NOTES +any=10@500[tahoe] Busy 0012 11443 343 500 00:00:10 +any=10@500[tahoe] Busy 0013 11445 344 500 00:00:10 +any=10@500[nt-1] Busy 0008 293 345 500 00:00:11 +any=10@500[nt-1] Busy 0009 246 346 500 00:00:11 +any=10@500[vaio] Busy 0015 11489 347 500 00:00:07 +any=10@500[vaio] Busy 0014 11484 348 500 00:00:09 +any=10@500[imac] Busy 0011 22530 351 500 00:00:11 +any=10@500[imac] Busy 0010 22528 352 500 00:00:11 % rush -cp +any=10@500 =3@500 vaio.36 +any=3@500: priority was +any=10@500, now +any=3@500 NOW SET TO 3 CPUS, BUT RENDERS STILL RUNNING, NOTHING IS KILLED.. % rush -lc CPUSPEC[HOST] STATE FRM PID JOBTID PRI ELAPSED NOTES +any=3@500[tahoe] Busy 0021 11453 343 500 00:00:41 +any=3@500[tahoe] Busy 0020 11451 344 500 00:00:41 +any=3@500[nt-1] Busy 0016 280 345 500 00:00:43 +any=3@500[nt-1] Busy 0017 71 346 500 00:00:43 +any=3@500[vaio] Busy 0022 11499 347 500 00:00:39 +any=3@500[vaio] Busy 0023 11501 348 500 00:00:39 +any=3@500[imac] Busy 0018 22534 351 500 00:00:42 +any=3@500[imac] Busy 0019 22536 352 500 00:00:42 TIME PASSES, RENDERS START FINISHING, 3 CPU LIMIT TAKES EFFECT % rush -lc CPUSPEC[HOST] STATE FRM PID JOBTID PRI ELAPSED NOTES +any=3@500[tahoe] Busy 0026 11459 343 500 00:00:04 +any=3@500[tahoe] JobPass - - 344 500 00:01:01 Cpu max (3/3 busy) +any=3@500[nt-1] Idle - - 345 500 00:01:01 Cpu max (4/3 busy) +any=3@500[nt-1] Idle - - 346 500 00:01:01 Cpu max (4/3 busy) +any=3@500[vaio] JobPass - - 347 500 00:01:01 Cpu max (3/3 busy) +any=3@500[vaio] JobPass - - 348 500 00:01:01 Cpu max (3/3 busy) +any=3@500[imac] Busy 0024 22540 351 500 00:00:07 +any=3@500[imac] Busy 0025 22542 352 500 00:00:05

rush -criteria

rush -criteria 'criteria strings' [jobid..] Job Criteria. Sets the criteria string for the job. Use '-' to specify no criteria. See Criteria submit command for more info on how criteria is specified.

rush -d

rush -d [flags..] Rush client side debugging. Enables debugging messages in the 'rush' client program only. (This does not affect the daemon). Supports the same flags as the -dlog option; see the table of logging flags for the complete list of supported flags. Examples: 'rush -d [flags]' Examples rush -lf -d t Enable TCP debug messages (t) for the '-lf' command. rush -laj -d u Enable UDP debugging messages (u) for the '-laj' command. rush -lf -d a Enable ALL debugging messages (a) for the '-lf' command.

rush -dcatlog

rush -dcatlog [host..] NOTE: This option is deprecated, in favor of rush -catlog Daemon cat(1) Log File. View the rushd.log daemon on the local [remote] host. rush -dcatlog prints the current rushd.log (usually today's) rush -dcatoldlog prints the last rotated log (usually yesterday's). Caveat: for reasons strange and unusual, all daemon 'catlog' output has a '>' prepended to each line.

rush -deltaskfu

rush -deltaskfu [..] Delete Task, Force Unconditionally (Administrative) Users should not use this command. Use 'rush -rc' to remove cpus from your job. Deletes individual 'tasks' from a job server or cpu server. Only root and the rush adminuser can use this command. This command has two forms; in one case it takes two arguments where a task (or cpu) is forcibly removed from the job's cpu report, and in the second case a task (or cpu reservation) is forcibly removed from the cpu server's tasklist (i.e. 'rush -tasklist'). Examples: Delete Task Examples rush -deltaskfu vaio.55 117 Delete JOBTID #117 from job vaio.55 rush -deltaskfu vaio.55 183 tahoe    Delete CPUTID #183 (from 'rush -tasklist') on host tahoe JOBTID values can be found in -lc, -lcf and -tasklistfull reports. CPUTID values can be found in -lcf and -tasklistfull reports. Caveat: This command is not intended for casual use. It is to be used by administrators in unusual situations where cpus are stuck due to communications problems between machines. Use this command carefully.

rush -dependon

rush -dependon [-|depjobid[,..]] Set Dependency On Other Jobs. Sets new 'dependon' jobids for the current [specific] job. See the submit script docs on DependOn for more info. Read the examples carefully, to avoid confusing jobid(s) to affect with new jobids for the dependon command; remember to separate all 'dependon' jobids with commas instead of spaces. Caveats: Setting 'dependon' to a new value disables all previous values (i.e., clears them first). Setting dependon to '-' will disable it, but you will have to un-Hold any frames manually. All 'dependon' jobids must have the same hostname as the job being modified. See also, rush -waitfor. To differentiate between the new value for 'dependon', and the jobid(s) to be affected, the new value for 'dependon' must be a comma-separated list of jobids with no spaces. Examples: Examples of using 'rush -dependon' rush -dependon - Disable all dependencies rush -dependon tahoe.37,tahoe.38, Job now depends on 'tahoe.37' and 'tahoe.38'. Trailing comma required rush tahoe.40 -dependon tahoe.37,tahoe.38, tahoe.40 now depends on 'tahoe.37' and 'tahoe.38' rush tahoe.40 -dependon tahoe.37, tahoe.40 now depends on job 'tahoe.37' Trailing comma on tahoe.37 is required

rush -dexit

rush -dexit [remotehost..] Daemon Exit. (Administrative) Tells a daemon to exit immediately via TCP. Acknowledges success. Only root and the rush adminuser can use this command. Automatically saves job checkpoint before exiting.

rush -dexitnow

rush -dexitnow [remotehost..] Daemon Exit Now. (Administrative) Tells daemon to exit immediately, via UDP. There is no acknowledge. Only root and the rush adminuser can use this command. Use only in case of emergency. Automatically saves job checkpoint before exiting.

rush -dlog

rush -dlog <flags> [remotehost..] Daemon Debug Logging (Administrative) Enables certain daemon debug logging flags. Only root and the rush adminuser can use this command. Use 'rush -dlog -' to disable all debugging. Logging Flags These are flags that can be used with: 'rush -dlog' to enable daemon debugging messages in the rushd.log, eg: 'rush -dlog utE' 'rush -d' to enable client side debugging messages to the user's screen, eg: 'rush -d t -lfi' In the rush.conf file's LogFlags These flags can be combined to accumulate logging verbosity. All flags can be enabled by specifying 'a', e.g. 'rush -dlog a'. All flags can be disabled by specifying '-', e.g. 'rush -dlog -' - - A single dash is used to disables all debugging flags a - all b - bump mechanism logging c - select() info d - log duplicate/redundant receipt of packet drops e - events (time oriented, async, lic unpause) f - fork/wait h - hostname lookups j - Log job submissions and job state changes k - Log bumped/killed/usurped tasks l - Logical string evaluations m - memory calculations (RAM) during priority battles, etc n - network commands (udp/tcp) o - connect()/open()/close()/bind()/socket() (low level) p - parse command line arguments, submit scripts q - sequence id errors are printed (default:off) r - reboot management/transactions, and checkpoint purges of old tasks s - signals t - tcp u - udp w - 'waitfor' and 'dependon' checks x - show negative cache events y - yp lookups C - class ToWords/FromWords (PROBABLY OBSOLETE) D - dumpsig debugging (Bluesky: 1.21) F - File loading line-by-line debugging E - Errors not normally displayed (benign, but suspect) J - Job Objects (win2k job objects kill mechanism msgs) K - Actual kill commands executed L - Locks on files M - Microsoft specific debugging N - Newtask and notify advertising O - program loops P - server: Priority mechanism, client: log 'rush -push' mechanism Q - monitor timeouts R - Reaper msgs S - Server/Client context switches T - task/taskack transactions U - update (scheduling, priority mechanism, idle cpu management) W - 'Wakeup' pass states; pass->unpass state changes X - Random UDP message dropping -- TESTING ONLY!! ('a' does not affect this option, it must be specified explicitly) Z - reserved for on the fly debugging problems 0 - snapshot messages @ - PROTOCOL TEST FOR TASK_TIMEOUT. Induces 60 second sleep before sending TASK_START. ('a' does not affect this option, it must be specified explicitly) Examples: Examples: Enabling And Disabling Rushd 'Debug Logging' rush -dlog ut tahoe      Enable UDP (u) and TCP (t) debugging for daemon on host tahoe rush -dlog - tahoe Disable all debugging flags for the daemon on host tahoe

rush -dlogstats

rush -dlogstats [host|+hostgroup..] Daemon Log Statistics (Administrative) Useful for easily determining problems with machines on large networks. Checks the local [remote] daemon log for particular kinds of errors, and logs the number of times each error appears in a table summary. Check Daemon's Log File Statistics % rush -dlogstats +any # ************ Problems ************* **Security* ************* Normal ************** # SID Hostname Alert Fail Error Warn Exit Fork Bump Secur Start Rot Info JobSt JobEn JobKi l 0 geneva 2 0 0 0 0 0 0 0 0 1 0 0 0 0 l 0 rotwang 0 0 0 0 1 0 0 1 1 1 2 0 0 0 l 0 ontario 0 0 0 0 0 0 0 0 1 1 2 1 0 0 This report shows that geneva has two 'ALERT' errors in its daemon log, which might indicate a problem worth investigating with 'rush -catlog rushd.log geneva | grep ALERT'. Debug Log Statistics: Field Descriptions Alert Possible system problem; a systems operation unexpectedly failed. For instance, daemon can't bind() to port, because some other entity is holding the port open. Or a select() that should either timeout or return data, actually failed. Or, a tcp/udp command was malformed. Fail Normal operations, failure condition detected and handled. Error An error condition (usually enabled via 'rush -dlog E') Warn Warn of a non-error condition (ie. use of obsolete ops) Exit Error so fatal, it caused daemon to exit. Fork Unable to fork a new process. Bump A job was bumped (killed) during eg. a priority battle. Secur(e) Attention to sysadmins: someone is trying to circumvent the permissions. eg. someone is changing a job they don't own with '-fu'. Start Daemon was started. Rot(ate) rushd.log was rotated. Info Normal operations information worthy of note. JobSt(art) A job was started. JobEn(d) A job ended. JobKi(ll) A job was dumped.

rush -done

rush -done <framerange|framestate..> [jobid..] Done Frames. Changes frame(s) into the Done state for current [specific] job. If frames are running, they are killed and enter the 'Done' state on completion. The frames to affect can either be specified as a frame range (i.e., 1-100) or as a frame state, for which all matching frames will be affected. Examples: 'rush -done' Examples % rush -done 1-100 'Done' frames 1 through 100 % rush -done fail 'Done' all frames currently Fail % rush -done fail que    'Done' all frames currently Fail or Que   % rush -done all 'Done' all frames (new in 102.42)

rush -donecommand

rush -donecommand <command|-> [jobid..] rush -donecommand [jobid..] < file PHASE OUT. NOTE: This option is deprecated, in favor of rush -jobdonecommand You probably want the 'rush -jobdonecommand' which runs a command when the job finishes rendering the last frame. In very old versions of rush (pre 102.40, released in 2002), the older 'rush -donecommand' did what the newer 'rush -jobdumpcommand' does now; ran a command when the job was dumped. The old name became confusing when it was decided in 2002 to allow separate commands for done vs. dump, so the older command was simply made obsolete, replaced by the newer "rush -jobXXXXcommand" commands with clearer names.

rush -donemail

rush -donemail [email[,email..]] [jobid..] Set Job Done Mail. Changes who to send mail to when job is done. In version 102.40h+, this means when the job renders its last frame, or when the job is dumped. Which means if you requeue frames after a job is done, you will receive an email each time the last frame is rendered. To set several names, separate the names by commas. Each time you set the value, the previous value is overwritten. See DoneMail submit command for more info and examples. See also DumpMail submit command if you want mail sent only once, when the job is actually dumped. Done Mail Examples rush -donemail fred@foo.com,jack@foo.com Fred and Jack will receive mail when the current job is done. rush -donemail fred@foo.com tahoe.37 Fred will receive mail when jobid tahoe.37 is done. rush -donemail - tahoe.37 tahoe.38 Disable all done emails for jobids tahoe.37 and tahoe.38

rush -down

rush -down downhost[,downhost..] jobhost A Machine Is Down. Tells rush jobs a machine is down. If a machine hangs or goes down, rush has no way to know it.. for all rush knows, the remote machine is just under heavy load, and is being unresponsive. Use '-down' to requeue frames in jobs that are stuck in the Run or Start state, because the remote machine is either hung or has been turned off unexpectedly. Normally frames will requeue when the remote is rebooted, but in cases where the machine needs to stay off for a while, the user can use '-down' to manually tell rush the remote is down, and to requeue the hung frames elsewhere. This also helps jobs that won't dump because some frames are 'stuck' on a host that is down. Warning: This command should only be used in extreme situations where job(s) show the frames running on a remote machine that is known to be down, or not running the frame. 'rush -down' Examples rush -down huron,erie tahoe   Tell all jobs on tahoe that machines    'huron' and 'erie' are down rush -down huron,erie +any Tell all hosts that 'huron' and 'erie' are down

rush -dump

rush -dump [jobid|user|user@host..] Dump Job. Dumps the job. Busy frames are killed right away. If no arguments are specified, the RUSH_JOBID variable is used to determine which job to dump. If jobid(s) are specified, those jobs will be dumped. If a user's name is specified, all jobs owned by that user on the local machine will be dumped (e.g., 'rush -dump fred'). In the case of user@host, all jobs owned by 'user' served by 'host' will be dumped. i.e., 'rush -dump fred@tahoe' will dump all jobs owned by fred that were submitted to host tahoe. 'rush -dump' Examples % rush -dump tahoe.37 tahoe.38    Dumps jobs tahoe.37 and tahoe.38 % rush -dump fred Dumps all jobs owned by 'fred' served by the local machine   % rush -dump fred@superior Dumps all jobs owned by 'fred' served by host 'superior'

rush -dumpmail

rush -dumpmail [email[,email..]] [jobid..] Set Job Dump Mail. Changes who to send mail to when job is dumped. To set several names, separate the names by commas. Each time you set the value, the previous value is overwritten. See DumpMail submit command for more info and examples. If you want mail sent whenever the job finishes rendering the last frame, then use instead -donemail. Dump Mail Examples rush -dumpmail fred@foo.com,jack@foo.com Fred and Jack will receive mail when the current job is dumped. rush -dumpmail fred@foo.com tahoe.37 Fred will receive mail when jobid tahoe.37 is dumped. rush -dumpmail - tahoe.37 tahoe.38 Disable all dump emails for jobids tahoe.37 and tahoe.38

rush -end

rush -end [jobid..] End Job. Ends a job. Similar to rush -dump, but waits for running frames to finish (i.e., running frames are not killed).

rush -exitnotes

rush -exitnotes "text message" Exit Notes for frame. Use this from within render scripts instead of 'rush -notes' to set messages for the NOTES field of 'rush -lf' reports This command should ONLY be used from within a running render script to change the notes for the currently running frame. Do NOT use this command to attempt to change frames in other jobs, or different frames within a job; use 'rush -notes' in such cases. The command will not take effect until the process actually exits; this is due to how the notes are delivered to the jobserver as part of the process exit code delivery. Unlike most rush commands, this command does not accept command line arguments to specify the jobid or frame number; it is assumed this command will only be run from within the render script during its execution, and it therefore uses the environment variables RUSH_JOBID and RUSH_FRAME (which are automatically in the render script's environment) to determine which job/frame to affect. For this reason, do not attempt to unset or modify these variables from within the render script, and do not attempt to force-set these variables in attempt to subvert this command to be used in other contexts. Example Use This command is safe to use within render scripts in both error and non-error conditions, since it is a low-bandwidth operation, so it can be used when 100's of machines are all rendering at the same time without negatively impacting the job server (as would be the case if 'rush -notes' were used) Example use of 'rush -exitnotes' #!/bin/csh -f ### RENDER SCRIPT # Render render //jobserver/jobs/inuyasha/s8a/dropshadow/${RUSH_PADFRAME}.rib set err = $status if ( $err != 0 ) then rush -exitnotes "Render failed: exitcode $err" exit 1 endif # Image integrity check imagecheck //jobserver/images/inuyasha/s8a/drop-${RUSH_PADFRAME}.sgi if ( $status ) then rush -exitnotes "Failed image check" exit 1 endif rush -exitnotes "Image check OK" exit 0 In the above case, as each frame finishes, the 'NOTES' field in the 'rush -lf' frames report might look like: Frames Report Showing -exitnotes % rush -lf STAT FRAME TRY HOSTNAME PID JOBID START ELAPSED NOTES Done 0001 2 ontario 19752 ontario.12 05/18,18:51:03 00:01:17 Image OK Done 0002 4 ontario 19763 ontario.12 05/18,18:51:05 00:01:15 Image OK Fail 0003 3 erie 11112 ontario.12 05/18,18:51:03 00:00:27 Failed image check Fail 0004 9 erie 11121 ontario.12 05/18,18:51:44 00:00:25 Failed image check Done 0005 10 cross 19779 ontario.12 05/18,18:52:23 00:00:02 Image OK Done 0006 3 cross 19781 ontario.12 05/18,18:52:23 00:00:02 Image OK Done 0007 2 cyanic 11129 ontario.12 05/18,18:52:22 00:00:25 Image OK Low Latency The use of this command within render scripts is recommended because 'rush -notes' is a much higher latency operation that can overwhelm a job server if used from within render scripts on large networks. 'rush -exitnotes' is very gentle on communications, due to how the notes are delivered back to the server as part of the normal UDP process exit code delivery. This is why the notes will not be seen in the frames report until the render process has actually exited.

rush -fail

rush -fail <framerange|framestate..> [jobid..] Fail Frames. Changes frame(s) into the Fail state for current [specific] job. If frames are running, they are killed and enter the 'Fail' state on completion. The frames to affect can either be specified as a frame range (i.e., 1-100) or as a frame state, for which all matching frames will be affected. Examples: 'rush -fail' Examples % rush -fail 1-100 Fail frames 1 through 100 % rush -fail done Fail all frames currently Done % rush -fail done hold    Fail all frames currently Done or Hold   % rush -fail all Fail all frames (new in 102.42)

rush -frameflags

rush -frameflags <-|keepnotes..> [jobid..] Change Frame Flags. Frame Flags rush -frameflags keepnotes Disables automatic clearing of 'NOTES" field in framelist when a frame starts rendering. rush -frameflags - No flags. (Default) Default behavior is for NOTES field in framelist to be cleared    whenever a frame starts rendering. See also the submit script docs on FrameFlags for more info.

rush -fu

rush [args..] -fu Force User1 The -fu flag augments other operations. It overrides warnings pertaining to lack of permission to do an operation because the user invoking it is not the owner of the job. For instance, if you want to control another person's job, you might get an error, e.g.: Using the 'rush -fu' Flag % rush -an vaio va-229 rush: va-229: you're not owner!   Attempt to add 'vaio' as a nevercpu to someone's job    fails because you're not the job's owner % rush -an vaio va.229 -fu         Add nevercpu 'vaio' Same command with -fu to force it... ...now it works See also the $RUSH_FU environment variable. Note: This feature may be disabled by the system administrator via the DisableFu flag in the Rush Configuration File. 1 This acronym is rumored to have alternate, pejorative expansions.

rush -getoff

rush -getoff [-msg 'text'] [remotehost|+hostgroup..] Get Off A Machine. Similar to 'rush -offline', but kills and requeues any currently running frames before taking the specified host/hostgroups offline. If no hosts/hostgroups are specified, the local machine's rush daemon is taken offline. When a machine is taken offline, frames that are already rendering are allowed to complete before taking the processors offline. Once offline, no new frames will be started. An optional remark can be specified via "-msg 'text'" which will show up in 'rush -lac' reports, eg: Using 'rush -getoff' with a Custom Message % rush -getoff -msg 'Client Demo' ontario ontario: Offline (killing 1 procs) % rush -lac HOST OWNER JOBID TITLE FRM PRI PID ELAPSED REMARKS ---------- -------- -------------- ------------------------- ---- ----- ------- ---------- -------- ontario - - - - - - Offline Client Demo : Examples of 'rush -getoff': 'rush -getoff' Examples % rush -getoff Kills frames running on localhost (requeing them elsewhere) leaving the localhost offline (won't accept any more renders) % rush -getoff tahoe erie      Kills frames running on tahoe and erie (requeing the frames elsewhere)    leaving both hosts offline (won't accept any more renders) % rush -getoff -msg 'Bad Memory' huron    Offline 'huron' with the remark 'Bad Memory' Changes to the online/offline state are appended to the rushd.log. Also: (new in 102.42a9), changes to the online/offline state append 's' entries to the cpu.acct accounting log

rush -hold

rush -hold <framerange|framestate..> [jobid..] Hold Frames. Changes frame(s) into the Hold state for current [specific] job. If frames are running, they are killed and enter the 'Hold' state on completion. When a frame is in the Hold state the frame will not be rendered, and a job will not autodump if there are any Hold frames in the frame list. The frames to affect can either be specified as a frame range (i.e., 1-100) or as a frame state, for which all matching frames will be affected. Examples: 'rush -hold' Examples % rush -hold 1-100 Hold frames 1 through 100 % rush -hold fail Hold all frames currently Fail % rush -hold fail done    Hold all frames currently Fail or Done   % rush -hold all Hold all frames (new in 102.42)

rush -imgcommand

rush -imgcommand 'command [args..]' [jobid..] rush -imgcommand [jobid..] < file Set Image Display Command Used by IRUSH to determine the command to execute whenever the user double clicks on a frame in the IRUSH 'Frames' report. Normally, the command invoked should display the rendered image for that frame. This should be a complete command to view an image in your favorite image viewer, with %d or %04d in place of the frame number. If 'cmd [args]' are not supplied, the command will be read from stdin. This is useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into one single line. The image viewing command must background itself, or you must force it by using '&' (Unix), or 'start /b' (Windows). Examples: 'rush -imgcommand' Examples % rush -imgcommand 'xv /job/images/foo.%04d.sgi &' (Unix) Invokes 'xv' to view images C:\> rush -imgcommand "start /b flip //nt/images/s1.%04d.png" (Windows) Invokes 'flip' in background    to view images See the ImgCommand submit command for more info.

rush -jobcheckpoint

rush -jobcheckpoint [host|+hostgroup..] Save all jobs to the checkpoint file This tells the local host (or remotehost|+hostgroup if specfied) to save a checkpoint now. Checkpoint will be loaded on reboot automatically, if the checkpoint.onboot is set to '1' in the rush.conf file (which is the default).

rush -jobnotes

rush -jobnotes 'notes..' [jobid..] rush -jobnotes [jobid..] < notes.txt rush -jobnotes '-' [jobid..] Set Job Notes. Appends free form strings of 'job notes' to the current [specific] job. Job notes are viewable in 'rush -ljf'. If 'notes..' are not specified, the notes can be supplied on stdin. If 'notes..' are specified on the command line, they must be quoted if they contain spaces. Specifing 'rush -jobnotes -' will clear all previous notes. See Notes submit command for more info. 'rush -jobnotes' Examples rush -jobnotes < foo.txt Set job notes to contents of foo.txt (Previous notes are completely cleared) rush -jobnotes 'This is a test' Set job notes to 'This is a test' You can dynamically modify the job notes (e.g. insert, remove, reformat), using these steps: 1) Use 'rush -ljf' to get the current notes into a local variable or file. 2) Change the notes in your variable or file as needed. 3) Use 'rush -jobnotes -' to clear the previous notes. 4) Use 'rush -jobnotes < file' or pipe in the new notes.

rush -jobremarks

rush -jobremarks 'message_text' [jobid..] User Defined Job Remarks. (New in rush 102.42a9 and up) Includes "message_text" in the "Remarks" field of the rush -lj/-laj reports. In the report, message_text will be enclosed in [brackets] to differentiate the user defined remarks from rush's own remarks (see example below). Historical Note: This feature was actually implemented in an earlier release, but fell out of subsequent releases up thru 102.42a8 due to a source code regression, and wasn't reinstated until 102.42a9. Example: 'rush -jobremarks' % rush -lj STATUS JOBID TITLE OWNER %DONE %FAIL BUSY ELAPSED REMARKS -------- ---------------- ------------------------- -------------- ----- ----- ---- ---------- -------- Run tahoe.1 FG_PLATE erco %0 %0 8 00:05:53 Pause tahoe.2 BG_PLATE erco %10 %0 0 00:05:53 Job paused. Wait tahoe.3 COMP erco %0 %0 0 00:05:53 Wait for other jobs % rush -jobremarks "jason: please pickup" tahoe.1 tahoe.2 tahoe.3 tahoe.1: jobremarks is now 'jason: please pickup' tahoe.2: jobremarks is now 'jason: please pickup' tahoe.3: jobremarks is now 'jason: please pickup' % rush -lj STATUS JOBID TITLE OWNER %DONE %FAIL BUSY ELAPSED REMARKS -------- ---------------- ------------------------- -------------- ----- ----- ---- ---------- -------- Run tahoe.1 FG_PLATE erco %0 %0 8 00:05:53 [jason: please pickup] Pause tahoe.2 BG_PLATE erco %10 %0 0 00:05:53 [jason: please pickup] Job paused. Wait tahoe.3 COMP erco %0 %0 0 00:05:53 [jason: please pickup] Wait for other jobs

rush -jobstartcommand

rush [jobid] -jobstartcommand 'command [args..]' rush [jobid] -jobstartcommand < file rush [jobid] -jobstartcommand '-' Set the command executed when job starts running. See JobStartCommand submit command for more info. If 'cmd [args]' are not supplied, the command will be read from stdin. This is useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into one single line. Setting the jobstartcommand to '-' will disable it. Examples: 'rush -jobstartcommand' Examples rush -jobstartcommand 'perl //server/share/rush/start.pl'   Set job start command to run a perl script rush -jobstartcommand < foo.txt Set job start command to contents of foo.txt   rush -jobstartcommand - disable the start command (Default)

rush -jobdonecommand

rush [jobid] -jobdonecommand 'command [args..]' rush [jobid] -jobdonecommand < file rush [jobid] -jobdonecommand '-' rush [jobid] -jobdonecommand '[options] command [args]' Set the command executed when job done running. See JobDoneCommand submit command for more info. If 'command [args..]' are not supplied, the command will be read from stdin; useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into a single line. Setting the jobdonecommand to '-' will disable it. Examples: 'rush -jobdonecommand' Examples rush -jobdonecommand 'perl //server/share/rush/done.pl' Set job done command to run a perl script rush -jobdonecommand < foo.txt Set job done command to contents of foo.txt   rush -jobdonecommand - Disable the job done command (Default) By default, the jobdonecommand is executed whenever the last running frame finishes. This behavior can be redefined by specifying [options] before the 'command [args..]': -jobdonecommand Options -nolog Disables creation of jobdonecommand.log in the log directory. Here are some examples with [options] specified: Example JobDoneCommand With Options rush -jobdonecommand '-nolog perl //path/mydonecommand.pl' When the 'perl' script runs, no jobdonecommand.log is created. (The output will be thrown away, unless the script redirects its output elsewhere) This is useful if the donecommand needs to delete the entire log directory, without getting 'file is in use' errors.

rush -jobdumpcommand

rush [jobid] -jobdumpcommand '[-nolog] command [args..]' rush [jobid] -jobdumpcommand < file rush [jobid] -jobdumpcommand '-' Set the command executed when job dumps. See JobDumpCommand submit command for more info. If 'cmd [args]' are not supplied, the command will be read from stdin. This is useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into one single line. Specifying the '-nolog' option in front of 'command' will disable the creation of the 'jobdumpcommand.log' file in the LogDir. This is useful to prevent 'file in use' errors on Windows if you want to remove the entire logdir as part of a cleanup operation. Setting the jobdumpcommand to '-' will disable it. Examples: 'rush -jobdumpcommand' Examples rush -jobdumpcommand 'perl //server/share/rush/dump.pl' Set job dump command to run a perl script rush -jobdumpcommand '-nolog perl //server/share/rush/dump.pl' Same as above, disables creation of 'jobdumpcommand.log' rush -jobdumpcommand < foo.txt Set job dump command to contents of foo.txt rush -jobdumpcommand - Disable the dump command (Default)

rush -lac

rush -lac [-s secs] [-r retry] [-c count] List All Cpus. Lists the status of all rush cpus on the network. Report should come up quickly, even on large networks. This report is useful for seeing which cpus are running whose jobs. (New in rush 102.41a and up) [-r secs] is an optional argument specifying the number of times to retry contacting the remote machine, before giving up. Default is 4. [-s secs] is an optional argument specifying the number of seconds between retries for remotes that don't respond right away. Default is 2. [-c count] is an optional argument to specify the number of times to automatically regenerate the report (for continuous updates). A value of 0 indicates infinite updating. Default is 1. Example report: 'rush -lac' Report % rush -lac HOST OWNER JOBID TITLE FRM PRI PID ELAPSED tahoe - - - - - - Online tahoe - - - - - - Online superior erco vaio.50 FOREST_RENDER 0034 100 4026 00:10:01 superior erco vaio.50 FOREST_RENDER 0035 100 4027 00:10:02 superior tamu ibm4.208 AUTO_CAD 0101 300k 2031 02:43:31 superior tamu ibm4.208 AUTO_CAD 0102 300k 2035 02:43:31 erie erco vaio.50 FOREST_RENDER 0041 900k 14560 00:10:15 erie erco vaio.50 FOREST_RENDER 0042 900k 14561 00:10:15 michigan - - - - - - Offline michigan - - - - - - Offline michigan - - - - - - Offline michigan - - - - - - Offline *** NO RESPONSE FROM: *** ontario huron Note that machines that did not respond are listed in the summary at the bottom of the report; all lines of these summaries are prefixed by '***'. All Cpus Report Format The administrator has a certain amount of control over the output format of the 'All Cpus' report via the rush.conf file, specifically with the AllCpusFmt, AllCpusHeader1, AllCpusHeader2 commands.

rush -lacf

rush -lacf [-s secs] [-r retry] [-c count] List All Cpus Full. Lists the full info of all rush cpus on the network. This report may take a while to run. This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging. (New in rush 102.41a and up) [-r secs] is an optional argument specifying the number of times to retry contacting the remote machine, before giving up. Default is 4. [-s secs] is an optional argument specifying the number of seconds between retries for remotes that don't respond right away. Default is 2. [-c count] is an optional argument to specify the number of times to automatically regenerate the report (for continuous updates). A value of 0 indicates infinite updating. Default is 1.

rush -lah

rush -lah [-t timeout] [hostname..|+hostgroup..] List All Hosts. Lists the Ram/Criteria configuration for all hosts in the render queue, which is loaded directly from the $RUSH_DIR/etc/hosts file. Example: 'rush -lah' Report % rush -lah IP Hostname Ram Cpus MinPri Criteria 10.100.100.1 tahoe 512 2 0 +any,irix 10.100.100.2 superior 512 2 0 +any,irix,+dante 10.100.100.3 erie 512 2 0 +any,irix,+dante 10.100.100.4 ontario 512 2 0 +any,irix,+dante 10.100.100.5 vaio 128 1 0 +any,linux,intel 10.100.100.6 rotwang 128 1 100 +any,linux,intel     Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds. If hostname is specified, i.e., 'rush -lah hostname', the output comes from the cache of the daemon running on the specified host. This is useful in determining hostname caching problems. If a +hostgroup is specified, i.e. 'rush -lah +farm', the output comes from the cache of the daemons in that hostgroup. (New in rush 102.42 and up) A daemon's current negative hostname cache will be shown at the bottom of an -lah report, if the 'rush -lah <hostname>' syntax is used. The following example shows 3 negative cache entries ("geneeva", "thoe", and "foobar") as being cached by the daemon on host 'tahoe': Negative Cache % rush -lah tahoe Cache on tahoe: IP Hostname Ram Cpus MinPri Criteria 192.168.0.10 geneva 1024 2 0 +any,+w2k,+work 192.168.0.3 rotwang 100 2 0 +any,+erco,+linux,linux,linux6.0,intel,dante 192.168.0.9 ontario 100 2 0 +any,+erco,+linux,linux,linux6.0,intel,dante 192.168.0.14 meade 512 2 0 +any,+erco,+linux,linux 192.168.0.7 tahoe 512 2 0 +any,+erco,+linux,linux *** Negative Cache: geneeva (expires 00:12:38 secs from now) *** Negative Cache: thoe (expires 00:14:53 secs from now) *** Negative Cache: foobar (expires 00:14:53 secs from now) (New in rush 102.42a6 and up) When too many hosts have been added to the rush/etc/hosts file than one is licensed for, rush simply ignores the extra hosts, and the 'rush -lah' report will warn about the extras in its report: Warnings for Unlicensed Hosts % rush -lah IP Hostname Ram Cpus MinPri Criteria 192.168.0.10 geneva 1024 2 0 +any,+w2k,+work 192.168.0.3 rotwang 100 2 0 +any,+erco,+linux,intel,dante 192.168.0.9 ontario 100 2 0 +any,+erco,+linux,intel,dante [..] 192.168.5.132 r0129 100 1 0 +any,+sgi 192.168.5.133 r0130 100 1 0 +any,+sgi *** ERROR: TOO MANY HOSTS IN RUSH HOSTS LIST [24 found, 20 max] *** ERROR: The following hosts are being ignored by rush.. *** 192.168.5.134 r0131 100 1 0 +any,+sgi *** 192.168.5.135 r0132 100 1 0 +any,+sgi *** 192.168.5.136 r0133 100 1 0 +any,+sgi *** 192.168.5.137 r0134 100 1 0 +any,+sgi

rush -lahf

rush -lahf [-t timeout] [hostname..] List All Hosts Full. Full hosts report for all hosts on local [remote] host. This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging. Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

rush -laj

rush -laj [-s secs] [-r retry] [-c count] [-r retries] [hostname..|+hostgroup..] List All Jobs. Queries all hosts on the network for what jobs are running. (All hosts being all hosts in the "+any" hostgroup) (New in rush 102.41a and up) [-r secs] is an optional argument specifying the number of times to retry contacting the remote machine, before giving up. Default is 4. [-s secs] is an optional argument specifying the number of seconds between retries for remotes that don't respond right away. Default is 2. [-c count] is an optional argument to specify the number of times to automatically regenerate the report (for continuous updates). A value of 0 indicates infinite updating. Default is 1. (New in rush 102.41a and up) Optional hostnames or +hostgroups can be specified to list jobs served by only those hosts. If unspecified, all hosts in the +any group is used. 'rush -laj' Report % rush -laj STATUS JOBID TITLE OWNER %DONE %FAIL BUSY ELAPSED Run erie.167 100/340/master tamu %18 %0 8 01:15:28 Run erie.169 100/410/master tamu %75 %0 20 01:11:21 Run erie.170 100/390/master tamu %10 %34 4 01:10:44 Run erie.171 100/360/master tamu %10 %34 4 01:02:43 Run huron.454 W:090/440/all krement %100 %0 0 26:25:31 Run huron.457 W:090/425/master krement %100 %0 0 26:11:59 Run huron.476 W:090/425/master krement %100 %0 0 23:40:37 Run huron.519 Q:090/460/master krement %5 %0 8 00:21:27 Pause tahoe.76 230/250/layout_lock erco %2 %0 0 01:52:08 Run allek.790 qr2 tje %80 %1 4 06:58:20 Run allek.794 180/260/DAY tje %80 %1 2 06:38:48 Run allek.796 180/250/DAY tje %0 %0 0 06:34:21 Run allek.801 qr2 tje %0 %0 0 05:27:40 Run vaio.663 230/tdcheck/roomBC woz %80 %20 0 00:44:53 All Jobs Report Format The administrator has a certain amount of control over the output format of the 'All Jobs' report via the rush.conf file, specifically with the AllJobsFmt, AllJobsHeader1, AllJobsHeader2 commands.

rush -lajf

rush -lajf [-t timeout] List All Jobs Full. Full job report for all jobs on the network. This report may take a while to run. This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging. Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

rush -lc

rush -lc [-t timeout] [jobid..] List Cpus. Lists cpus for current [specific] job. Lists a job's cpus, and what they're doing (or not doing). Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds. Example: 'rush -lc' Report % rush -lc # List Cpus assigned to the current job CPUSPEC[HOST] STATE FRM PID JOBTID ELAPSED NOTES huron=4@800k JobPass - - 178 00:00:00 This is a 'neverhost' huron=4@800k JobPass - - 179 00:00:00 This is a 'neverhost' huron=4@800k JobPass - - 180 00:00:00 This is a 'neverhost' huron=4@800k JobPass - - 181 00:00:00 This is a 'neverhost' +any=10@100[howland] Busy 0080 14733 165 00:00:01 +any=10@100[howland] Busy 0078 14731 166 00:00:01 +any=10@100[howland] Busy 0079 14732 167 00:00:01 +any=10@100[toronto] Idle/Nak - - 171 00:00:00 - +any=10@100[toronto] Idle/Nak - - 172 00:00:00 - +any=10@100[toronto] Idle/Nak - - 173 00:00:00 - +any=10@100[vaio] Busy 0074 4107 174 00:00:03 +any=10@100[vaio] Busy 0071 4100 175 00:00:03 +any=10@100[vaio] Busy 0072 4101 176 00:00:03 +any=10@100[vaio] Busy 0073 4106 177 00:00:03 List Cpus Headings Heading Description CPUSPEC[HOST] The cpu specification supplied by the user during submit. If hostgroups are specified (i.e. +any), the group is broken out into individual cpus, with the hostname of each processor shown in [brackets]. STATE The state the cpu's 'task reservation' is in. In the following, 'cpuserver' means the remote host serving the cpu. Idle -- task idle, waiting for cpuserver Idle/Nak -- job is still trying to reach the cpuserver Avail -- cpuserver wants us to run a frame Start -- found frame to start, reply to cpuserver Run -- cpuserver telling us frame started ok Busy -- frame is busy, reply to cpuserver CpuPass2 -- replies to cpuserver, remains in state JobPass -- job told cpuserver we can't use its cpu (see NOTES for why) Unpass -- remote reverting to idle state (cosmetic) FRAME The frame that is being rendered. PID The process id on HOST for the running render script. This is also the 'process group leader'. JOBTID The 'job task id' for this processor. Only useful as an argument commands like 'rush -rc'. ELAPSED The wall clock elapsed time in hours:minutes:seconds for the rendering frame. NOTES Various notes regarding the state of the job. These notes can sometimes be stale; the 'STATE' information is a more accurate representation of the state of the cpu.

rush -lcf

rush -lcf [-t timeout] [jobid..] List Cpus Full. A 'full' report for all of the current [specific] job's cpus. This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging. Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

rush -lf

rush -lf [-t timeout] [jobid..] List Frames. Lists Frames for current [specific] job. Lists a job's frames to see what state they're in. Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds. 'rush -lf' Report % rush -lf              # List Frames for the current job STATUS FRAME TRY HOSTNAME PID JOBID START ELAPSED NOTES Run 5000 1 vaio 18092 tahoe.34 02/26,02:15:40 00:00:21 - Run 5001 1 vaio 18093 tahoe.34 02/26,02:15:40 00:00:21 - Run 5002 1 vaio 18100 tahoe.34 02/26,02:15:50 00:00:11 - Que 5003 0 - 0 tahoe.34 00/00,00:00:00 00:00:00 - Que 5004 0 - 0 tahoe.34 00/00,00:00:00 00:00:00 - Que 5005 0 - 0 tahoe.34 00/00,00:00:00 00:00:00 - The columns contain the following information: List Frames Examples Heading Description Stat The status of the frame. Can be any of: Que - The frame is queued, ready to be run by an available cpu Run - The frame is busy running Done - The frame completed successfully Fail - The frame failed to run successfully (exit code was 1) Hold - The frame will not run until changed manually to Que The following are transitional states for frames, which you should only see for a short time: Strt - Frame just started to run, no process id yet Die - The frame is being killed, but hasn't received an exit code back yet Frame The frame number, padded 4 digits. (Or as otherwise specified by 'framepad' in the rush.conf file.) Try The number of times this frame has been tried. Includes requeues due to exit(2), but does not include priority bumps. Hostname The name of the host that is running (or that ran) the frame. PID The process id of the render script. This is also the process group leader for the frame's process hierarchy. (In windows, the top level 'Job Object') Start The date/time the render began running. Elapsed The wall clock elapsed time in hours:minutes:seconds. Notes Free form notes text, set by the user's invocation of rush -notes, e.g. 'rush -notes 0012:"License Error"' Frames Report Format The administrator has a certain amount of control over the output format of the 'frames' report via the rush.conf file, specifically with the FrameFmt, FrameHeader1, FrameHeader2, FramePad commands.

rush -lff

rush -lff [-t timeout] [jobid..] List Frames Full. A 'full' report of all frames' state information. This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging. Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

rush -lfi

rush -lfi [-t timeout] [jobid..] List Frame Information. Lists Frame info for current [specific] job, including average render times, ETAs, percent Busy/Done/Fail, etc. Example: 'rush -lfi' Report % rush -lfi vaio.139               # List Frame Info for job 'vaio.139' Average Total Jobid State Total Perc Elapsed Elapsed Average ETA ------------ ----- ----- ---- ---------- ---------- ------------------------ rotwang.3 Que 2 %4 - - - rotwang.3 Run 2 %4 - - - rotwang.3 Done 46 %92 00:00:11 00:08:36 Thu Feb 12 17:47:16 2004 rotwang.3 Fail 0 %0 - - - rotwang.3 Hold 0 %0 - - - rotwang.3 --- rotwang.3 Render Elapsed: 00:02:25 rotwang.3 Render Start: 02/12,17:44:45 rotwang.3 Render End: 02/12,17:47:10 Jobid The jobid for this job. (If multiple jobs are being listed, this column shows which info goes with which job) State The various states the frames can be in, and to the right, the accumulated data for all the frames in each of those states. Total The total number of frames in each 'State'. Perc The percentage of frames in each 'State'. Average Elapsed The average render times. This is the sum of all the Done frames' render times divided by the total Done frames. (Currently only 'Done' state frames are shown) Total Elapsed The total elapsed time for the frames; a sum of all the 'Elapsed' times in the 'Frames' report. (Currently only 'Done' state frames are shown) Average ETA This is the estimated time of completion for the job, based on the average render time for frames rendered successfully so far vs. the wall clock time spent running the job. e.g.: oldest_start = "The oldest start time for all Done frames" recent_end = "Most recent end time for all Done frames" time_spent = ( recent_end - oldest_start ) time_per_frame = time_spent / total_done_frames time_to_go = time_per_frame * ( total_que + total_hold ) This time will vary widely, since completion time is dependent on the cpu availability, which is always changing. Render Elapsed The elapsed time for rendering, ie. how long the TD had to wait for the job to finish rendering. The difference between the 'Render Start' and the 'Render End' times. Render Start The 'Start Time' for the first frame. Render End The 'End Time' for the last frame that finished rendering. Warning: the ETA is meant for ballpark estimates only, and is not meant to be taken literally. Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

rush -lhc

rush -lhc [criteria] List Host Criteria. Shows all [specific] host criteria your sysadmin has configured in rush. If no arguments are specified, all host criteria are listed. Otherwise, arguments can be the criteria names to limit the report to display. Both criteria and host names are sorted alphabetically. 'rush -lhc' Reports % rush -lhc       # Lists all criteria when no arguments specified intel: vaio rotwang irix: erie ontario superior linux: rotwang tahoe vaio % rush -lhc linux       # Lists only hosts matching 'linux' criteria   linux: rotwang tahoe vaio

rush -lhg

rush -lhg [+hostgroup..] List Host Groups. Shows all [specific] hostgroups your sysadmin has configured in rush. If no arguments are specified, all hostgroups are listed. Otherwise, arguments can be the names of hostgroups to limit the report to display. Both hostgroups and hostnames are sorted alphabetically. 'rush -lhg' Reports % rush -lhg       # Lists all hostgroups when no arguments specified   +any: erie ontario rotwang superior tahoe vaio +dante: erie ontario superior +intel: rotwang tahoe vaio +irix: erie ontario superior +linux: rotwang tahoe vaio % rush -lhg +dante       # Lists only the '+dante' hostgroup +dante: erie ontario superior

rush -lj

rush -lj [-fmt string] [-t timeout] [remotehost] List Jobs. Shows all jobs running on the local [remote] host. Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds. 'rush -lj' Reports [you@erie] % rush -lj                   # List Jobs on local workstation STATUS JOBID TITLE OWNER %DONE %FAIL BUSY ELAPSED REMARKS -------- -------------- ------------------------- -------- ----- ----- ---- ---------- -------- Run erie.167 100/340/master tamu %18 %0 8 01:15:28 Run erie.169 100/410/master tamu %75 %0 20 01:11:21 Run erie.170 100/390/master tamu %10 %34 4 01:10:44 Run erie.171 100/360/master tamu %10 %34 4 01:02:43 [you@erie] % rush -lj tahoe             # List Jobs on remote host tahoe STATUS JOBID TITLE OWNER %DONE %FAIL BUSY ELAPSED REMARKS -------- -------------- ------------------------- -------- ----- ----- ---- ---------- -------- Run tahoe.5 100/540/master izzie %25 %0 4 00:30:01 Run tahoe.6 100/540/comp izzie %25 %0 4 00:30:02 Run tahoe.10 100/640/master izzie %10 %0 4 00:22:48 Pause tahoe.11 100/640/comp izzie %12 %0 4 00:22:49 Job paused. The default columns contain the following information: Job Report Headings Heading Description Status The status of the job. Here's a state diagram showing the various states the job status transitions through: Click on the image to see a larger version. Wait - The job is waiting for another job or time to finish (WaitFor, -waitfor) Pause - The job has been paused with -pause Run - The job is available to run frames Done - The job is done, all the frames rendered successfully Fail - The job is done, some frames failed The following are transitional states for the job, which you should only see for a short time: End - the job was ended with -end Dump - the job was dumped with -dump Done - all frames Done, about to run JobDoneCommand, -jobdonecommand DoneCmd - the configured JobDoneCommand is running DumpCmd - the configured JobDumpCommand is running Remove - about to be removed from the system Jobid The job's jobid. Title The job's title (Title, -title). Owner The user that submitted the job. %Done The percentage of frames that are in the Done state. %Fail The percentage of frames that are in the Fail state. Busy The number of frames currently in the Run state. Elapsed The elapsed time in HH:MM:SS since the job started running. Remarks This field may contain messages like 'Job paused.' or 'Wait for other jobs'. Jobs Report Format The administrator has a certain amount of control over the output format of the 'Jobs' report via the rush.conf file, specifically with the JobFmt, JobHeader1, JobHeader2 commands.

rush -ljf

rush -ljf [-t timeout] [remotehost] List Jobs Full. Lists all jobs running on the local [remote] host, showing 'full information' in a multi-line format. This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging. Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds. 'rush -ljf' Report [erco@tahoe] % rush -ljf tahoe.9 Jobid: tahoe.9 State: Done ActState: Run Title: HONDA DefPriority: 1 LogDir: //server/jobs/HONDA/SC4A/rushlogs LogFlags: ImgCommand: xv //server/jobs/HONDA/SC4A/imgs/sc4a-%04d.tif Ram: 10 Nice: 10 MaxTime: 00:10:00 FrameFlags: Command: //server/jobs/HONDA/SC4A/render-vehicle 640 480 ChkCommand: JobDoneCommand: JobStartCommand: JobDumpCommand: AutoDump: off User: erco (1000/100) DoneMail: - StartDate: Mon May 14 23:54:53 2001 EndDate: - Elapsed: 11:22:34 Frames: 50 Cpus: superior=2@100 +farm=20@10 Notes[0]: - Criteria:

rush -licpause

rush -licpause [remotehost] Pause Job Into A License Pause State. Causes the job to pause for the number of seconds specified by the submit command 'LicPauseSecs' or by the most recent 'rush -licpausesecs' command. Normally 'rush -licpause' is executed from within a render script, when it detects the renderer failed with a license error, causing the job to pause for typically a minute or two, then unpauses itself again. For info on how to use 'rush -licpause' mechanism, see the 'How To' documentation on Render License Errors. 'rush -licpause' will be ignored if the job has already been paused with 'rush -pause'. 'rush -licpause' Example : (your render script) : ## YOUR RENDER COMMANDS HERE render /job/MYSHOW/MYSHOT/ribs/foo.rib set $err = $status if ( $err == $PRMAN_LIC_ERROR ) then # LICENSE ERROR OCCURRED -- PAUSE JOB, REQUEUE FRAME rush -licpause rush -notes ${RUSH_FRAME}:'License Error' exit 2 endif :

rush -licpausesecs

rush -licpausesecs secs [jobid..] Set Pause Time for License Errors Sets the amount of time a job remains in the 'Pause' state when 'rush -licpause' is invoked from a render script. For info on how to use 'rush -licpause' mechanism, see the 'How To' documentation on Render License Errors.

rush -log

rush -log [-tail] <framerange> [jobid..] View Frame Error Logs. Prints the frame logs for the specified frame range. The '-tail' option can be specified to continuously output appended data as the log file grows. Using the job's 'logdir' value, prints the logs found in that directory for the frames you specify, e.g.: 'rush -log' Example % rush -log 1-10,2 20   View logs for frms #1-10 on 2s, and frm #20

rush -logdir

rush -logdir pathname [jobid..] Log Directory. Changes the job's log directory, where frame log files are written to. This does not affect commands that are already running or have run. i.e., old logs are not moved to the new location, or renamed on the fly. When changing the logdir with -logdir, no checks are made to see if 'path' is a valid directory. '%s' can be used to insert the current jobid into the logdir path. Note: when the logdir is changed from the command line (rush -logdir) and has an embedded '%s', the directory is not created for you. Auto-creation of the logdir only occurs during job submission to avoid the "catch-22" scenario where the caller can't create the directory with a jobid in it if the jobid doesn't exist yet. Embedding the Jobid in the Log Directory's Path rush -logdir //server/jobs/logs/%s    This sets the LogDir to: //server/jobs/logs/tahoe.34   See LogDir for more information.

rush -logext

rush -logext -|.txt [jobid..] Log Extension. Changes the filename extension used on all the text files saved into the job's log directory. The default is normally no extension, however the sysadmin can change the network-wide default via the framelog.ext in the rush.conf file. Changing the log extension while a job is running will only affect the logs for frames that haven't started rendering yet; frames that have already rendered or are rendering will still have the old filename extension. It is for this reason it's usually better to either submit the job with the correct 'logext' command, or have the sysadmin change the network wide default in the rush.conf file. Changing Log Filename Extensions rush -logext - tahoe.55 Disables filename extensions for log files. Example: //server/some/path/logs/0001 //server/some/path/logs/0001.old //server/some/path/logs/framelist //server/some/path/logs/jobinfo rush -logext .txt tahoe.55 Causes all new frame logs to have ".txt" appended. Example: //server/some/path/logs/0001.txt //server/some/path/logs/0001.old.txt //server/some/path/logs/framelist.txt //server/some/path/logs/jobinfo.txt Caveats Changing the log extension while a job is running will not affect frames already rendered, or are in progress. Changing this setting while a job is running will affect irush(1): irush will use the new log extension setting to view all logs, effectively preventing it from seeing logs for frames rendered with the old setting. It is for these reasons it's better to set the logext either as part of the job submission, or have the sysadmin set the network wide default in the rush.conf file. See Also framelog.ext -- network wide default logext -- submit command

rush -logflags

rush -logflags <-|keeplast|keepall> [jobid..] Log Flags. Changes the log flags for the job. Examples: 'rush -logflags' Examples rush -logflags - Logs are overwritten (Default) rush -logflags keepall   Keep all logs; concatenate old to *.old (e.g. 0012.old) rush -logflags keeplast    Like 'keepall', but only keeps last log (don't concatenate) See LogFlags for more information.

rush -maxcpus

rush -maxcpus <#cpus> [jobid..] Sets the maximum number of cpus a job can use. Puts a cap on the entire job. If maxcpus is set to 0, there is no limit. MaxCpus rush -maxcpus 0 Disabled -- no limit rush -maxcpus 15 Limit job to no more than 15 cpus busy rendering See MaxCpus for more information.

rush -maxtime

rush -maxtime <hh:mm:ss> [jobid..] Sets maximum time for rendering frames. Value is in HH:MM:SS (hours:minutes:seconds). The system will automatically kill frames that exceed the specified maxtime. You can control what happens to the frame when maxtime expires using MaxTimeState. See the MaxTime submit command for more info.

rush -maxtimestate

rush -maxtimestate <que|fail|done|hold> [jobid..] What happens when MaxTime expires. By default, rush requeues (Que) a frame when its MaxTime timer expires. You can change this to any of: 'rush MaxTimeState Values rush -maxtimestate que Frames become "Que" when timer expires rush -maxtimestate fail Frames become "Fail" when timer expires rush -maxtimestate done Frames become "Done" when timer expires rush -maxtimestate hold Frames become "Hold" when timer expires

rush -nice

rush -nice <niceval> [jobid..] Set Job's Niceness. Sets the nice value for the job; 0 runs as normal user process, higher values (10, 20) increase niceness to others. See Nice submit command for more info.

rush -notes

rush -notes <framerange>:'notes..' [jobid..] Frame Notes. Sets Notes for frame(s) on current [specific] job. These notes will show up in the rightmost column of 'rush -lf' reports. NOTE: Use 'rush -exitnotes' instead if you need to set the notes from within a render script for the currently rendering frame. For more info, see the high latency section below. Examples: 'rush -notes' Examples rush -notes 155:"license error"   set notes for frame 155 to 'license error' rush -notes 200-250:"redo" set notes for frames 200 thru 250 to "redo" Here's an example showing how to set the notes for frames 3 through 5 for a job: Setting The 'Notes' For Frames % rush -notes 3-5:"Redo For Client" [ontario.4] 0003: Redo For Client [ontario.4] 0004: Redo For Client [ontario.4] 0005: Redo For Client % rush -lf on.4 STAT FRAME TRY HOSTNAME PID JOBID START ELAPSED NOTES Done 0001 1 ontario 7762 ontario.4 02/23,16:17:26 00:00:11 Done 0002 1 ontario 7764 ontario.4 02/23,16:17:26 00:00:11 Done 0003 1 ontario 7777 ontario.4 02/23,16:17:38 00:00:12 Redo For Client Done 0004 1 ontario 7779 ontario.4 02/23,16:17:38 00:00:12 Redo For Client Done 0005 1 ontario 7787 ontario.4 02/23,16:17:51 00:00:12 Redo For Client   High Latency Caveat 'rush -notes' is a high-latency command that should not be used casually in render scripts, especially on large networks where potentially 100's of machines can all be running the command at once, overwhelming the jobserver daemon by backlogging it with numerous TCP connections. As of Rush Version 102.42a9, the new 'rush -exitnotes' command is better suited for use in render scripts to alter the notes for the currently rendering frame; it avoids hitting the jobserver with TCP, instead contacting the render node's daemon to schedule the message for delivery as part of the renderer's exit code delivery mechanism, which has no load impact on the job server. When should 'rush -notes' be used? Only in other contexts, such as external applications (like 'irush' and 'www-rush') where setting the frame notes of different jobs is needed, or in the rare case where a running render script needs to change the notes of frame(s) OTHER than the one currently rendering, and in that case only when there's not much potential for many of the 'rush -notes' commands to be running in parallel.

rush -offline

rush -offline [-msg 'text'] [remotehost|+hostgroup..] Offline My Machine. Offline's the rush daemon on the specified hosts/hostgroups. If hostnames/hostgroups aren't specified, the local machine's rush daemon is taken offline. When a machine is taken offline, frames that are already rendering are allowed to complete before taking the processors offline. Once offline, no new frames will be started. 'rush -offline' is similar to 'rush -getoff', differing only in that -offline allows any currently rendering frames to complete before taking the processors offline, whereas 'rush -getoff' will bump any running renders, freeing up the processors right away. The bumped frames will be requeued. An optional remark can be specified via "-msg 'text'" which will show up in 'rush -lac' reports, eg: Offline With Custom Message % rush -offline -msg 'Bad Memory' ontario ontario: Offline % rush -lac HOST OWNER JOBID TITLE FRM PRI PID ELAPSED REMARKS ---------- -------- -------------- ------------------------- ---- ----- ------- ---------- -------- ontario - - - - - - Offline Bad Memory : Examples of using 'rush -offline':  'rush -offline' Examples  rush -offline   offline all cpus on local host rush -offline tahoe offline all of tahoe's cpus rush -offline +farm   offline the entire render farm rush -offline -msg 'Failing drive' tahoe   offline 'huron' with the remark 'Failing drive' To return processors to the online state, use rush -online. Changes to the online/offline state are appended to the rushd.log. Also: (new in 102.42a9), changes to the online/offline state append 's' entries to the cpu.acct accounting log

rush -online

rush -online [-msg 'text'] [remotehost|+hostgroup..] Online My Machine. Online's the rush daemon on the specified hosts/hostgroups. If hosts/hostgroups are not specified, the local machine's rush daemon is put online. When a machine is online, frames will immediately be allowed to start rendering on the available processors. The '-msg' flag is not recommended for use with -online, due to the message's persistence in 'rush -lac' reports when online, but is supported for symmetry with the -getoff/-offline commands. Examples of using 'rush -online': 'rush -online' Examples rush -online   Online all cpus on local host rush -online tahoe Online all cpus on host tahoe rush -online +farm   Online all cpus in the '+farm' hostgroup    To take hosts offline, use either rush -offline or rush -getoff. Changes to the online/offline state are appended to the rushd.log. Also: (new in 102.42a9), changes to the online/offline state append 's' entries to the cpu.acct accounting log

rush -pause

rush -pause [jobid..] Pause Job Pauses the current [specific] job. Busy frames will be allowed to finish, no new frames will be started. To pause a job and kill the running frames, use:  Pause a Job, Kill Running Frames  rush -pause && rush -que run To continue a job from pause, use rush -cont.

rush -ping

rush -ping [-t timeout] [remotehost|+hostgroup..] Ping Rush Daemon. (Administrative) Pings the local [remote] daemon to see if it's running, and what it's doing. Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

rush -priority

rush -priority <prival> [jobid..] Change Default Job Priority. Changes the default priority for the job, which affects all 'cpus' specifications that do not indicate priority values (and therefore use the default priority). Example-- If a job submitted the following 'cpus' specifications: Example Submit Using 'priority' % rush -submit << EOF priority 10 : : cpus erie=2@100     cpus tahoe=4 cpus +any=10 EOF ..this would create a job with 2 cpus on erie at a priority of 100, and all other cpus at a priority of 10. Changing this job later with 'rush -priority 20' would cause all cpus (except erie, which specified a priority) to change their priorities to 20. See the Priority submit command for setting the initial default priority. See Priority Description for a description of how priority values work.

rush -push

rush -push [-s secs] [-c tries] <file[,file..]> [host|+hostgroup] Push File(s) To Remotes. (Administrative) Pushes rush configuration files (such as rush.conf, hosts, etc) to remote servers. Used to distribute rush config files quickly. -s secs - Specifies the number of seconds to wait for replies from remotes. Default is 5. -c tries - Number of retries made if no response in the number of seconds specified (-s secs). Default is only 1 try is made. Files to be 'pushed' must be specified by name. Only files in the rush etc directory can be 'pushed'. Specify filenames only, not full paths. More than one file can be specified, if separated by commas. "file" may only contain the characters "A-Z", "a-z", "0-9", "-", "_", and a single ".". This prevents abusive use of -push, since only files in $RUSH_DIR/etc are affected. Only root and the rush adminuser can use this command. The sysadmin can enable/disable 'rush -push' operations via a setting in the rush.conf file, using the AllowPush flag. 'rush -push' Examples % rush -push hosts,rush.conf Push hosts and rush.conf file to all hosts % rush -push rush.conf,license.dat     Push rush.conf and license.dat to all rush hosts   % rush -push rush.conf +linux     Push rush.conf to only hosts in the +linux hostgroup % rush -push rush.conf tahoe huron     Push rush.conf to hosts 'tahoe' and 'huron' only % rush -push rush.conf,hosts,license.dat tahoe huron     Push rush.conf, hosts, and license.dat to hosts 'tahoe' and 'huron' only

rush -que

rush -que <framerange|framestate..> [jobid..] Re-Queue Frames Changes frame(s) into the Que state for current [specific] job. If frames are running, they are killed and enter the 'Que' state on completion. The frames to affect can either be specified as a frame range (i.e., 1-100) or as a frame state, for which all matching frames will be affected. Examples: 'rush -que' Examples % rush -que 1-100 Que frames 1 through 100 % rush -que fail Que all frames currently Fail % rush -que fail done    Que all frames currently Fail or Done   % rush -que all Que all frames (new in 102.42)

rush -ram

rush -ram <ramval> [jobid..] Set Job's Ram Use. Sets the Ram use (in MB) for the job. See Ram submit command for more info.

rush -ramlist

rush -ramlist [remotehost..] View the ram usage of rush jobs for localhost [remotehost] machine. Shows a report of all running rush processes on the machine, including how much ram each has requested to reserve, and a running total for the available ram left over. 'rush -ramlist' Report % rush -ramlist tahoe STATE JOBID/TITLE PRI RAMUSE NOTES Busy neverland.57,grkenttest 1 800 Busy neverland.58,pflare 1 200 ------ 1000 Total ram on tahoe: 1024 Available ram on tahoe: 24

rush -rc

rush -rc <cpuspec|tidspec|hostname> [jobid..] Remove Cpu Removes Cpu from cpu list of current [specific] job. If the cpus being removed are busy running frames, the frames are killed and re-submitted elsewhere. Cpus can be removed in one of several ways: Individually by the JOBTID number, e.g. rush -rc .32 By hostname (or host alias), e.g. rush -rc tahoe, rush -rc ta By a particular cpu specification as shown in the 'rush -lc' report, e.g., rush -rc tahoe=4@200 To remove a cpu via 'JOBTID' values (e.g., 'rush -rc .334') you must precede each value with a period. When you delete by JOBTID, you are deleting single cpus from the 'rush -lc' report. Note: this report includes the JOBTID values so you can see which values to delete. If the cpu you delete is part of a larger specification, (e.g., tahoe=4@12), then the cpu count for the spec will be modified and the cpu count in that spec will be decremented as well (e.g., tahoe=3@12) If you remove a hostname (e.g., 'rush -rc tahoe') then all cpu specifications that have that host name (e.g., tahoe=3@100) will be removed. Also, any hostgroups that expand to include that host will have that host removed from the expansion (e.g., +any=3@100, which includes tahoe). (New in rush 102.42 and up) Hostname aliases can also be used to remove hosts. In previous versions, hostname aliases were not guaranteed to work for 'rush -rc'. Removing a Host % rush -lc CPUSPEC[HOST] STATE FRM PID JOBTID PRI ELAPSED NOTES howland=2@900 JobPass - - 171 900 00:00:00 No more frames howland=2@900 JobPass - - 172 900 00:00:00 No more frames +any=1@1[howland] JobPass - - 147 1 00:00:01 No more frames +any=1@1[howland] JobPass - - 148 1 00:00:00 No more frames +any=1@1[nt-1] JobPass - - 149 1 00:00:31 This is a 'neverhost' +any=1@1[nt-1] JobPass - - 150 1 00:00:31 This is a 'neverhost' +any=1@1[rotwang] JobPass - - 151 1 00:00:01 No more frames +any=1@1[rotwang] JobPass - - 152 1 00:00:01 No more frames % rush -rc howland nt-1.15 'howland=2@900': Removed nt-1.15 '+any=1@1': 2 task(s) deleted % rush -lc CPUSPEC[HOST] STATE FRM PID JOBTID PRI ELAPSED NOTES +any=1@1[nt-1] JobPass - - 149 1 00:00:37 This is a 'neverhost' +any=1@1[nt-1] JobPass - - 150 1 00:00:37 This is a 'neverhost' +any=1@1[rotwang] JobPass - - 151 1 00:00:08 No more frames +any=1@1[rotwang] JobPass - - 152 1 00:00:08 No more frames If you remove a cpu specification (e.g., 'rush -rc +any=3@100'), it must match character-for-character the entry shown in the 'rush -lc' report for the job: Removing a Hostgroup % rush -ac tahoe@100 # Add a cpu. % rush -rc tahoe@100 # Now try to remove it 'tahoe@100' no such cpu specification # FAILED: need to use spec shown in 'rush -lc' % rush -lc # Look at 'rush -lc' report CPUSPEC STATE FRM PID ELAPSED .. # More complete specification in report. tahoe=1@100 Run 0002 26747 00:00:11 .. % rush -rc tahoe=1@100 # Remove using spec shown in report 'tahoe=1@100' removed # It works

rush -reload

rush -reload <hosts[,rush.conf]> [host|+hostgroup..] (New in rush 102.42 and up) Reload daemon's files (rush.conf, hosts..). Forces the local [remote] daemon to reload the specified file(s). The first argument after -reload may be either "hosts" or "rush.conf", or a comma separated combination of the two, eg. "hosts,rush.conf". Whenever 'hosts' is specified, the rush/etc/hosts file is reloaded by the local [remote] daemon, its IP cache flushed, and the negative cache is cleared. 'rush -reload' Examples rush -reload hosts Tells the local machine's rushd to reload its hosts file flushing its IP cache and negative cache. rush -reload hosts +any Tells all machines to reload their hosts files and flush their IP and negative caches rush -reload rush.conf +any Tells all machines to reload their rush.conf files   rush -reload hosts,rush.conf +any Tells all machines to reload hosts and rush.conf and flush their IP and negative caches rush -reload hosts tahoe geneva Tells only 'tahoe' and 'geneva' to reload hosts files and flush their IP and negative caches Caveats In DNS environments, using 'rush -reload hosts +any' should be safe for your DNS server, though it will keep it steadily busy while the command is updating each host; each host will reload the hosts file, causing all the hosts to resolve to update the IP cache. If your hosts file is large, this will cause a lot of requests to your DNS server from each machine while the hosts file is reloaded and re-cached. Fortunately, the 'rush -reload' operates serially; it hits each machine one at a time, and waits for the remote to load the hosts file completely before moving to the next machine, ensuring there's only one machine reloading the hosts file at any given moment in time.

rush -reorder

rush -reorder <framerange..> [jobid..] Reorder Frames. Reorders frames in the frame list to the new order specified. <framerange> specifies the new order of the frames in the frame list. Several frame ranges can be specified. Unspecified frames are simply left alone. Changing the order of the framelist affects the order frames are rendered in, since frames are issued from the top of the list, down. Example-- If you have a job with 10 frames in the frame list that are in normal 1 thru 10 order, you can use 'rush -reorder' to get different orderings, such as these: 'rush -reorder' Examples rush -reorder 10-1 becomes 10 9 8 7 6 5 4 3 2 1 rush -reorder 1-10,2 2-10,2   becomes 1 3 5 7 9   2 4 6 8 10   rush -reorder 5 6 7 8 9 10 1 2 3 4   becomes exactly that order: 5 6 7 8 9 10 1 2 3 4

rush -reserve

rush -reserve [-server <host>] <cpuspec> [ramval] Reserve Cpus. Reserves cpus by creating a 'dummy' job that holds the cpus. The job's title will be RESERVE. Reservation jobs are like any other-- to remove a reservation, dump the job. To change the reservation, you can use IRUSH or the usual rush commands like -ac/-rc to modify the job. 'cpuspec' indicates which processors are reserved, and at what priority. It is an error not to include a priority value in the cpuspec. The optional '-server <host>' can be specified, to make 'host' the job server for the reserve job, instead of the default which is to have the job server be on the same host as the cpus being reserved. A reservation job with a priority of '500' will prevent jobs with priorities less than 500, but yields to jobs higher than 500k. Including the 'k' flag in your reservation will also bump any renderings that are running on your machine that are lower than 500 priority. See Priority Description for more info on how rush priorities work. As an example of reserving cpus on your workstation, let's assume you sit at a dual proc workstation called 'tahoe': To reserve both processors of your dual proc workstation at a priority of '500', use:  Reserve Both Processors On A Dual Processor Workstation  rush -reserve tahoe=2@500k   Reserve both cpus at 500k priority   To submit a job that actually renders on your own workstation, such that one processor is used for rendering, leaving the other for your own interactive use, just submit with a cpu specification that requests your workstation at a priority higher than your reservation, with the 'kill' flag, e.g. "tahoe=1@510k". To get both cpus, just request both processors, e.g. "tahoe=2@510k". To remove the reservation for your machine, simply dump the reservation job. To prevent all jobs from rendering on your workstation-- even your own-- make your reservation at a priority of 999. Or, just use 'rush -offline' and/or 'rush -getoff'. The optional 'ram value' allows one to reserve an amount of ram per processor. This is useful on multiproc machines where you want to prevent large jobs from rendering on the few processors left unreserved. Increasing the ram value you reserve will decrease the size of jobs rush will allow to run on the unused processors. If 'cpuspec' doesn't include the number of processors, one processor is assumed. If 'ramval' is unspecified, a value of '1' is assumed. 'rush -reserve' Examples rush -reserve tahoe@998 Reserve 1 cpu on tahoe@998 rush -reserve tahoe=2@998 128   Reserve 2 cpus @998, 128MB of ram to each rush -reserve -server univac tahoe=2@998 128   Same as above, but jobserver will be 'univac'   rush -reserve tahoe=2@500 128 Reserve 2 cpus @500, 128MB of ram to each

rush -rf

rush -rf <framerange..> [jobid..] Remove Frames. Removes frames from the frame list of the current [specific] job. If frames are Busy at the time they are removed, they will first be killed, then scheduled for removal when confirmation arrives from the remote machine(s). 'rush -rf' Examples rush -rf 10-15 Removes 10 thru 15 from the current job rush -rf 108 110 800-899 Remove frames 108, 110, and 800-899 from the current job rush -rf -10--5 Removes negative frame range: -10 thru -5 rush -rf .1-5,.1 Removes floating point frames .1 through 5    stepping by .1 (.1, .2, .3 .. 4.9, 5.0)

rush -rn

rush -rn <hostname|+hostgroup..> [jobid..] Remove Nevercpus. Removes Nevercpus for current [specific] job.

rush -rotate

rush -rotate [rushd.log|cpu.acct] [remotehost|+hostgroup..] Rotate Rush Log Files (Administrative) Rotates the rushd.log file by default on local [remote] host. (New in 102.42) The cpu.acct log can also be rotated if the optional argument 'cpu.acct' is specified. Can be used in crontab scripts to rotate logs on local or remote machines. The rushd.log is normally automatically rotated with the LogRotateHour command in the rush.conf file, however that can be disabled so that you can configure your own weekly or monthly log rotations via a crontab using 'rush -rotate'. 'rush -status' Examples rush -rotate rushd.log Rotates the rushd.log on local machine rush -rotate rushd.log tahoe ontario    Rotates rushd.log on remote hosts 'tahoe' and 'ontario' rush -rotate rushd.log +any Rotates rushd.log on all machines rush -rotate cpu.acct Rotates cpu.acct log on local machine rush -rotate cpu.acct tahoe ontario    Rotates cpu.acct log on remotes 'tahoe' and 'ontario' rush -rotate cpu.acct +any Rotates cpu.acct on all machines Only root and the rush administrators can use this command.

rush -status

rush -status [-jobs] [-cpus] [-usage] [-s secs] [-c count] [remotehost|+hostgroup..] Status of Host. (Administrative) 'rush -status' quickly reports the status of jobs and cpus on the local [remote] hosts. Commonly used as 'rush -status' to get the status of the local host, or 'rush -status +any' for the status of all hosts. If none of the options '-jobs', '-cpus' or '-usage' are specified, all are assumed. ie. 'rush -status' is the same as 'rush -status -jobs -cpus -usage'. Output is intended for other programs to parse and regenerate. 'rush -status' Options -jobs Shows job info ('j' entries). -cpus Shows cpu info ('p' entries). -usage Shows usage of cpu, memory and swap ('C', 'M' and 'S' entries). -s secs   Sets the amount of time to wait for responses from remotes.   Default is 3. -c count Sets the number of times to continuously update the report. Default is 1. -c 0 creates a continuous update. The output contains several records of information for each host, one record per line. Host records start with an 'h' record, and terminate with a line of '--'. 8 different types of data records are possible. Data record types are defined by the first character in each record line, and can be one of: Line Prefixes Prefix Description h Hostname header. Leads off records for the specified host. d Daemon information. Info about the running daemon. j Job information. One line per job. (-jobs) p Processor status. One line per processor. (-cpus) C Cpu status. (-usage) M Memory status. (-usage) S Swap status. (-usage) X Error message (eg. "X PdhGetFormatted[usr0]: query not validated") Each record has its own fields: h <hostname> d <sequence-id> <daemon> <version> <PID=pid> <online state> <jobs> <busy procs> <total procs> j <owner> <jobid> <job title> <job state> <elapsed> <percent done> <percent fail> <# frms busy> <"job remarks"> p <owner> <jobid> <job title> <frame> <priority> <pid> <elapsed> C <ncpus> <usr%> <sys%> [<usr%> <sys%> ..] M <total mb> <used%> S <total mb> <used%> <swapin*> <swapout*> <'bad'swapping*> * - These values are in 'pages', eg. pages per second 'rush -status' Examples rush -status Shows the status of the local host rush -status +any Shows the status of all hosts rush -status -usage +any Shows only the cpu usage of all hosts rush -status -jobs +any Shows only the job info for all hosts rush -status -jobs -cpus +any Shows only the job and cpu info for all hosts rush -status -jobs -cpus -usage +any   Same as 'rush -status +any'

rush -submit

rush -submit [remotehost] Submit Job Submits a job to be managed by the local [remote] job server. Reads stdin for Rush Submit Commands which define the job submission. Optional [remotehost] can be specified to specify some other machine to be the server for the submitted job. By default, the machine that 'rush -submit' runs on becomes the machine that manages this job. This is usually the user's workstation. This ensures there's no "single point of failure" for the render queue, and distributes the overall load of managing jobs to multiple machines. For examples of how to make custom scripts that use 'rush -submit', see submit scripts in various languages and the rush command line tutorial which shows how to submit and manage jobs from the command line. Simple Example % rush -submit << EOF title MY_JOB ram 1 frames 1-10 logdir //path/to/an/empty/directory command echo This is a test cpus +any=5@1 EOF When this command completes successfully, it returns an exit code of 0, and the jobid of the submitted job is printed on stdout, which can be used to monitor the job. Warnings may also be printed. If the job failed to submit, a non-zero exit code is returned, and the reason for the failure will be printed on stderr. If you're writing a GUI application that submits jobs, you should keep stdout AND stderr of the 'rush -submit' command, and present the output to the user in case of error or warnings. Here's a python example: Python Example: How to parse errors from 'rush -submit' import os,re,string # Temp file for output tmpfile = "/tmp/.rush-submit-" + str(os.getpid()) # Submit the job, save stdout+err to 'tmpfile' submit = os.popen("rush -submit > " + tmpfile + " 2>&1", 'w') submit.write("title MYJOB\n" + "frames 1-10\n" + "command ls\n" + "cpus +any=5@1\n") submit.close() # Read all lines of stdout/err from 'rush -submit' f = open(tmpfile, 'r') submitout = string.join(f.readlines()) f.close() os.remove(tmpfile) # remove tmp file # Parse jobid (if any) try: jobid = re.search("RUSH_JOBID.(\S+)",submitout).groups()[0]       except: jobid = "" # If no jobid found, submit failed if ( jobid == "" ): ShowDialogToUser("SUBMIT FAILED\n" + submitout) sys.exit(1) else: ShowDialogToUser("SUBMIT WORKED: JOBID IS " + jobid)

rush -tasklist

rush -tasklist [remotehost] Task List. Lists scheduled tasks on local [remote] cpu server. This is the scheduler's sorted list of tasks wanting to run on the host in question. Tasks are sorted in the order they will be executed; the topmost task is the next one to be rendered. Tasks that are busy are sorted to the bottom. Tasks that were denied by either the job, or by cpu criteria, are also sorted to the bottom. Tasks are sorted into 'groups' of the same priority, and will round robin. For a 'full' version of a tasklist report, use 'rush -tasklistfull'.

rush -tasklistfull

rush -tasklistfull [remotehost] Task List Full Report. A 'full report' version of -tasklist, where data is presented as "key:value" pairs that are easily parsed by scripts. For more info, see -tasklist.

rush -title

rush -title <text> [jobid..] Job Title Sets title for current [specific] job. For more information, see Title.

rush -trs

rush -trs [csh|perl] Template Render Script Prints the template render script on stdout. This can be redirected to a file, to quickly create a working script, which should then be customized using a text editor. You can choose which scripting language your prefer; optional arguments 'csh' or 'perl' can be specified. If none are specified, 'perl' is the default.  Create a Render Script  % rush -trs > render_me # Save template as 'render_me'    % chmod +x render_me # Make executable (unix) % vi render_me # Edit the file to customize.. You will at least need to setup the render commands (the '### YOUR RENDER COMMAND(S) HERE' section) at minimum. Administration Notes The systems administrator can customize the template render scripts by modifying the $RUSH_DIR/etc/templates file, which 'rush -trs' simply prints out.

rush -try

rush -try <newcount> <framerange..> [jobid..] Set Try Count Changes the 'Try' count for the frames specified. A 'try count' is maintained for each frame in a job's frame list. This value is shown in the 'Try' column of 'rush -lf' reports, and is passed to render scripts via the $RUSH_TRY environment variable.  Example: Changing 'Try' Counts  % rush -lf # List Frames to see try counts STAT FRAME TRY HOSTNAME PID START ELAPSED NOTES Done 0001 4 superior 4144 10/04,03:29:10 00:00:18 Done 0002 4 tahoe 4160 10/04,03:29:30 00:00:16 Done 0003 4 placid 4163 10/04,03:29:47 00:00:16 Done 0004 6 huron 4168 10/04,03:30:04 00:00:16 Done 0005 4 finger 4171 10/04,03:30:21 00:00:16 % rush -try 0 1-5 # Reset Try count to zero 0001: tries was 4, now 0 0002: tries was 4, now 0 0003: tries was 4, now 0 0004: tries was 6, now 0 0005: tries was 4, now 0 % rush -lf # List Frames to see changes STAT FRAME TRY HOSTNAME PID START ELAPSED NOTES Done 0001 0 superior 4144 10/04,03:29:10 00:00:18 Done 0002 0 tahoe 4160 10/04,03:29:30 00:00:16 Done 0003 0 placid 4163 10/04,03:29:47 00:00:16 Done 0004 0 huron 4168 10/04,03:30:04 00:00:16 Done 0005 0 finger 4171 10/04,03:30:21 00:00:16

rush -tss

rush -tss [csh|perl] Template Submit Script. Prints the template submit script on stdout. This can be redirected to a file, to quickly create a working script, which should then be customized using a text editor. You can choose which scripting language your prefer; optional arguments 'csh' or 'perl' can be specified. If none are specified, 'perl' is the default.  Create a Submit Script  % rush -tss > submit_me # Save template as 'submit_me'    % chmod +x submit_me # Make executable (unix) % vi submit_me # Edit the file to customize.. You will, for instance, need to customize the command, title, cpus, and ram values at minimum. Administration Notes The systems administrator can customize the template render scripts by modifying the $RUSH_DIR/etc/templates file, which 'rush -tss' simply prints out.

rush -uping

rush -uping [-c count] [-s secs] [remotehost..] UDP Ping. (Administrative) UDP Pings the local [remote] daemon. Useful for checking for UDP packet dropping. If [remotehost] is unspecified, the local host is assumed. Options: -c <count> Sets the number of udp transmissions sent to the remote, using the number of seconds (-s secs) between each transmission. If -c is unspecified, the default is 1. Zero can be specified for <count> to send udp transmissions indefinitely, or until you hit ^C. -s <secs> Sets the number of seconds between transmissions. If -s is unspecified, the default is 5.  'rush -uping' Example Report  % rush -uping -c 5 -s 1 tahoe >> UDP Packet Burst Response Test >> Hit ^C when done. tahoe: uping 1 236020307 (0.0026 secs roundtrip)   tahoe: uping 2 237020120 (0.0022 secs roundtrip) tahoe: uping 3 238020080 (0.0022 secs roundtrip) tahoe: uping 4 239020077 (0.0023 secs roundtrip) tahoe: uping 5 240019993 (0.0022 secs roundtrip)

rush -waitfor

rush -waitfor [-|waitjobid|timeval,[..,..]] [jobid] Wait For Other Jobs or Times Sets the 'waitfor' jobids or times for the current [specified] job. See the submit script docs on WaitFor for more info. 'timeval' specifies a time to wait for. 'timeval' can be any of: WaitFor Time Values +8h 8 hours from now +60m 60 minutes from now +08:30:00 8 hours 30 minutes from now 17:00 at 5pm 5:30p at 5:30pm 5:30pm at 5:30pm 17:45,07/22/2002 at 5:45pm on 07/22/2002 12:30p,07/22/2002 at 12:30pm on 07/22/2002 The 'WaitFor' time value will show up in 'rush -ljf' reports in two places; once in the 'WaitForTime:' field as an absolute time in MM/DD/YY,HH:MM:SS format, the other in the 'WaitFor:' field as a relative time in +HH:MM:SS format if the time has not yet expired. For a tutorial on how to have one job wait for another, see Chaining Jobs. Read the examples carefully to avoid confusing the [jobid] with the [waitjobids]. Remember to specify commas to separate the [waitjobids] instead of spaces. If only one [waitjobid] is being specified, add a trailing comma. CAVEATS Using 'rush -waitfor' clears all previous waitfor jobids and times Setting 'waitfor' to multiple jobids AND times means ALL conditions must be met before job will start. Specifying 'waitfor -' will disable the job from waiting for other jobs and times. The jobs being waited for must be served by the same machine as the job doing the waiting. (i.e. the waitfor jobids and the job doing waiting must all be on the same jobserver) Note the strange syntax where commas are mandatory when specifying the first argument after '-waitfor'. This prevents the parser from being confused as to which is the 'waitjobid' and which is the jobid to affect. To differentiate between the new value of 'waitfor' and the jobid(s) to be affected, the new value of 'waitfor' must be a comma-separated list of jobids or times with no spaces. Examples: 'rush -waitfor' Examples rush -waitfor - # Disable all waitfor's rush -waitfor tahoe.37,tahoe.38, # Job waits for 'tahoe.37' and 'tahoe.38' to complete *** rush tahoe.40 -waitfor +8h, # tahoe.40 won't start running until 8 hours from now rush tahoe.40 -waitfor tahoe.37,tahoe.38, # tahoe.40 waits for 'tahoe.37' & 'tahoe.38' to complete *** rush tahoe.40 -waitfor tahoe.37, # tahoe.40 now waits for 'tahoe.37' to complete *** # Trailing comma on tahoe.37 is important! rush tahoe.40 -waitfor 5:30p,tahoe.37,tahoe.38, # tahoe.40 won't start until 5:30pm and # until 'tahoe.37' and 'tahoe.38' complete *** *** Depends on setting of 'rush -waitforstate'. See also, rush -dependon, rush -waitforstate

rush -waitforstate

rush -waitforstate <done|donefail|fail|dump> [jobid] Controls circumstances that cause WaitFor to trigger. See the submit script docs on WaitForState for more info. Examples: 'rush -waitforstate' Examples rush -waitforstate dump # Causes 'waitfor' to wait on other job(s) to Dump themselves (Default) rush -waitforstate fail # Causes 'waitfor' to wait on other job(s) to be Fail rush -waitforstate donefail # Causes 'waitfor' to wait on other job(s) to either be Fail or Done rush -waitforstate done # Causes 'waitfor' to wait on other job(s) to be Done

Frame State Table

This is the list of states a frame can be changed to, or searched for, eg. 'rush -done fail' changes all the Fail frames to 'Done'. Frame States   que   Frame is in the Que state, ready to be rendered done Frame is in the Done state, it completed successfully. fail Frame is in the Fail state, it failed with an error. hold Frame is in the Hold state, it won't be rendered until it is changed to a different state. all (new in 102.42) Valid only for searching only. Can be used to match 'all' frames, eg: 'rush -done all'