Rush Logo Rush Render Queue - Command Line Options
V 103.08p3 09/01/17
(C) Copyright 2008, 2017 Seriss Corporation. All rights reserved.
(C) Copyright 1995,2000 Greg Ercolano. All rights reserved.

Strikeout text indicates phased out or obsolete operators.
Text in red indicates new features; hovering over the text shows the version number.

   rush -acct_archive  
rush -acct_archive [-dbasedir /path]

Rush Accounting Data Archival

Lets you write scripts that take monolithic Rush cpu.acct files (from all versions of Rush) and write them to a file server directory, breaking the data out into a year/month/day directory tree by hostname. This data can then be loaded by the Rush python modules, e.g. RushAcct.py, etc.

In Rush 103.00, you can use the new rush.conf file command cpuacct.dbasedir to tell rush to automatically archive cpu accounting data to the file server year/month/day directory tree. This new '-acct_archive' flag is provided just to take old monolithic files and commit them to this new ascii based directory layout.

If -dbasedir is specified, the pathname argument should point to a file server directory where the data will be broken out into year/month/day subdirectories (which are created automatically).

If -dbasedir is *not* specified, the default path set in the rush.conf file for cpuacct.dbasedir will be used.

    -acct_archive Examples
    rush -catlog cpu.acct +any -t 5 > alldata.acct
    rush -acct_archive -dbasedir /yourserver/rush-acct-dbase < alldata.acct
    Take all the cpu.acct files from all the rush machines
    and combine them all into a single temp file, then archive
    the data to the directory tree at
    /yourserver/rush-acct-dbase/
    rush -acct_archive -dbasedir /yourserver/rush-acct-dbase < /usr/local/rush/var/cpu.acct Take the single cpu.acct data from
    /usr/local/rush/var/cpu.acct
    and write it to the server directory tree at
    /yourserver/rush-acct-dbase/

See Also

   rush -ac  
rush -ac <cpuspec..> [jobid..]

Add Cpus.

Adds cpus to the running job. Examples:

    Add Cpu Examples
    rush -ac tahoe@300 Add one cpu on tahoe at 300 priority
    rush -ac tahoe=2@300 +farm=10@100k Add two cpus on tahoe, 10 cpus for +farm hostgroup
    rush -ac +any=10@30 Add any 10 available cpus at priority 30

For more details on the different syntax of cpu specifications, see the Cpus submit command.

See Also

    Cpus -- Submit command to add cpus to a job
    rush -rc' -- Command line flag to remove cpus

   rush -af  
rush -af <framerange..> [jobid..]

Add Frames.

Adds frames to the current framelist. Adding frames that already exist yields an error. Example:

    Add Frames Examples
    rush -af 10-20 Add 10 thru 20 to the current job
    rush -af 1 5 10-100 800-899 Add frames 1, 5, 10 through 100
    and 800 through 899 to the current job
    rush -af -10--5 Add negative frames: -10 thru -5
    rush -af .1-5,.1 Add floating point frames .1 through 5
    stepping by .1 (.1, .2, .3 .. 4.9, 5.0)
    rush -af 20-25=Hold Add 20 thru 25 in Hold state
    rush -af 30-35:"To be done" Add 30 thru 35, Notes "To be done"
    rush -af 40-45=Done:"To be done" Add 40 thru 45 as Done with Notes

See Also

    Frames -- Submit command to add frames to a job

   rush -an  
rush -an <hostname|+hostgroup..> [jobid..]

Add Nevercpus.

Prevents the job from rendering on the specified hosts. Example:

    Add Never Cpus Examples
    rush -an tahoe +farm   Add host tahoe and all hosts in hostgroup  
    +farm as machines to never use.

See Also

    Nevercpus -- Submit command to add 'never cpus' to a job

   rush -args  
rush -args 

Read rush arguments from stdin.

(103.00 and up)
This allows you to make one invocation of rush, and send a stream of command line arguments on stdin, one set of rush commands per line.

This is an optimization to allow execution of many rush commands without the overhead of rush reloading the rush.conf and hosts files for each operation. (At some facilities, the hosts file can be so large as to incur latency for each command executed).

For instance, the following 3 commands:

    rush -cp -fu linux001.23 .922742 .922743 .922744 .922745 @1
    rush -cp -fu linux001.24 .923061 .923062 .923064 @1
    rush -cp -fu linux001.28 .925878 .925879 .925880 @1
        

..can be equivalently executed from a shell script as:

    rush -args << EOF
        -cp -fu linux001.23 .922742 .922743 .922744 .922745 @1
        -cp -fu linux001.24 .923061 .923062 .923064 @1
        -cp -fu linux001.28 .925878 .925879 .925880 @1
    EOF
        

..such that only one instance of rush is started (thus only loading the rush hosts and rush.conf files once), and all three "-cp" commands are executed one after the other.

This can be useful with scripts that want to run several rush commands in succession through a pipe. For instance, a python example:

    Example use of 'rush -args in Python'
      #!/usr/bin/python
      import subprocess
      pipe = subprocess.Popen("rush -args", shell=True, stdin=subprocess.PIPE).stdin   
      print >>pipe, "-lj";            # List Jobs
      print >>pipe, "-lf  tahoe.1";   # List Frames
      print >>pipe, "-lfi tahoe.1";   # List Frame Info
      print >>pipe, "-lc  tahoe.1";   # List Cpus
      pipe.close()
        

Whitespace and Comments

Leading white space, blank lines, and lines starting with '#' are ignored. This allows arguments and comments to be indented, and blank lines to be used for programming clarity.

Example:

    Example use of 'rush -args'
    rush -args << EOF
        # DUMP STALE JOBS
        -dump -fu eagle.37 eagle.38 eagle.39 eagle.50 eagle.58        
        -dump -fu tahoe.3 tahoe.9 tahoe.10
    
        # OFFLINE WORKSTATIONS
        -offline +work +previz
    
        # PING THE TAPE BACKUP SERVERS
        -ping shasta tahoe bluestone -t 3
    EOF
        

Error Handling

If any of the commands returns a non-zero exit code, "ARGS_ERROR" messages of the following format indicate the line number and exit code of the failed command returned, and the final exit code of 'rush -args' will be 1.

This allows your program to see which line failed, what the actual exit code of that line was. This error message format is guaranteed not to change, so as to be parsable by your own program. The format of these errors follows the existing standard for rush error messages to start with 'rush:', in this case "rush: ARGS_ERROR" uniquely identifies '-args' specific exit code messages, e.g.:

    
    rush: ARGS_ERROR at line <<LINE#>> of <<FILENAME>>, exitcode <<EXITCODE>>: <<LINE_CONTENTS>>
        

So a literal example stderr for a failed command due to a bad hostname might be:

    
    rush: 'tahoo' unknown host                                         <-- Output from the failed command
    rush: ARGS_ERROR at line 10 of stdin, exitcode 1: -getoff tahoo    <-- parsable error message from 
                                                                           'rush -args' showing exit code
        

Quoting

The commands you specify are NOT parsed by a shell. So shell specific characters like: <,|,>,&,$,% ..are not treated specially. The only characters that are treated specially are (") and (\):

  • (") can be used to protect arguments that contain spaces
  • (\) can be used to protect quotes
  • (\\) can be used to protect the backslash

So for instance:

    Example use of 'rush -args'
    #!/bin/sh
    rush -args << EOF
        -online emporia zoar eagle bluestone -msg "New drives installed"
        -getoff tahoe superior huron ontario -msg "Down for repairs"
        -command "python //path/to/my_script.py -arg1 -arg2 -arg3" tahoe.34      
        -command "cmd /c del c:\\temp\\myfile.txt" tahoe.34
    EOF
        

..shows how to protected spaces in -online/offline's '-msg' flag, and in -command's arguments, the last example showing how to double-up backslashes to specify pathnames containing backslashes.

Caveats

  • When -args is specified, you can specify *any* of the rush command line arguments EXCEPT -args. This prevents recursion and a confusion of multiple inputs.
  • Lines passed on stdin to -args can optionally have the 'rush' commands included. This way a list of rush commands can be copy/pasted directly into -args, without having to delete all the 'rush' commands. For instance, the following do the same thing:

      Example use of 'rush -args'
      # WITH 'rush' SPECIFIED
      rush -args << EOF
          rush -getoff tahoe superior
          rush -dump   tahoe.34 tahoe.35       
      EOF
      
      # WITHOUT 'rush' SPECIFIED
      rush -args << EOF
          -getoff tahoe superior
          -dump   tahoe.34 tahoe.35            
      EOF
      

   rush -autodump  
rush -autodump <off|done|fail|donefail> [jobid..]

Automatic Dump.

Configures job to automatically dump when there are no more frames to render.

    Autodump Examples
    autodump off Don't autodump; job remains when frames are DONE (default)
    autodump done Job dumps itself when all frames are DONE
    autodump fail Job dumps itself if job finishes with any FAIL frames
    autodump donefail Job dumps itself if all frames are DONE or FAIL

See Also

    AutoDump -- Submit command to set the autodump behavior for the job

   rush -bootcheck  
rush -bootcheck [options]

Use as a 'bootcheck' command.

(New in 103.00)
When the daemon starts, it can optionally use a bootcheck_cmd to check the state of the machine. 'rush -bootcheck' provides a way to check the state of the machine for common issues when a scripting language is not available, e.g. on Windows where perl and python might not be installed.

The common use would be: "rush -bootcheck -default", which does some default checks. However one can specify the individual -bootcheck arguments to run specific checks in a particular order, e.g.

    Example use of 'rush -bootcheck'
    [jdoe@tahoe] $
    [jdoe@tahoe] $ rush -bootcheck -ip_not 127. -ip_not 169.254 -hostname_not localhost -pingcheck   
    
    --- BOOTCHECK: IP address is '192.168.1.34' (and not '127.')
    --- BOOTCHECK: IP address is '192.168.1.34' (and not '169.254')
    --- BOOTCHECK: hostname is 'tahoe' (and not 'localhost')
    PING tahoe (192.168.1.34) 56(84) bytes of data.
    64 bytes from tahoe (192.168.1.34): icmp_seq=1 ttl=64 time=0.028 ms
    64 bytes from tahoe (192.168.1.34): icmp_seq=2 ttl=64 time=0.037 ms
    64 bytes from tahoe (192.168.1.34): icmp_seq=3 ttl=64 time=0.030 ms
    
    --- tahoe ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 1999ms
    rtt min/avg/max/mdev = 0.028/0.031/0.037/0.007 ms
    --- BOOTCHECK: ping check succeeded (exit=0)
    --- BOOTCHECK: DONE/OK
    [jdoe@tahoe] $
        

..which runs some common checks in verbose mode. Note the operations are run in order from left to right.

Here's a list of all the -bootcheck arguments:

    'rush -bootcheck' Optional Arguments
    -help Help report showing all options
    -d Toggles the debug flag. When on, prints debugging info.
    Default is off.
    -v Toggles the verbose flag. Prints more verbose output.
    Default is off.
    -m [#retries] Sets the maximum number of retries before failing.
    Use 0 to retry forever (default).
    -s [#secs] Sets the number of seconds delay between retries.
    Default: 5
    -hostname_not string Tests that the machine's hostname is not 'string'.
    Entire string must match. Case is ignored.
    -ip_not string Tests that the machine's ip address is not string.
    Only first part of 'string' must match.
    So if the ip is "127.0.0.1" and string is "127.0", it's a match.
    -pingcheck Tests that the machine's hostname is 'pingable'.
    Runs ping to check if the machine's IP is valid and reachable.
    -default Replaces the entire command line with these arguments:
        rush -bootcheck
             -hostname_not localhost     # RFC 6761 (6.3)
             -ip_not 127.                # RFC 5735 (3), RFC 1122 (3.2.1.3)
             -ip_not 169.254             # RFC 5735 (3)
             -pingcheck
                
    Specifying e.g. "-d" before "-default" will enable debugging
    while the default flags are being processed.

The options are run left-to-right in the order specified, so if you want to enable -v for only certain options, you can surround them with -v to toggle verbose on and off.

USE CASE
Again, the intended use for this command is with the 'bootcheck_cmd' in the rush.conf file. However, it can be run interactively to see what it does. The goal of the bootcheck_cmd is to wait until the machine is ready for rushd to start. So if it encounters an error during a test, it should wait a little and retry, the assumption being the problem will clear itself over time. For instance, in the rush.conf you might have:

    os=windows bootcheck_cmd "c:/rush/bin/rush -bootcheck -hostname_not localhost"
    
..which will cause the daemon to not start until the machine's hostname doesn't resolve to "localhost". Why do such a test?

The hostname might resolve to "localhost" during early stages of boot, then acquires a proper network hostname after a delay of seconds or minutes, depending on the machine's OS, when the network drivers have further initialized. So if rushd is started during this time, the bootcheck command will delay rush from starting any jobs until this condition clears.

Other checks can be added to the command line, and are executed in order, e.g.

    os=windows bootcheck_cmd "c:/rush/bin/rush -bootcheck -hostname_not localhost -pingcheck"
    
..which adds a 'ping check' after the hostname test, to verify packets sent to the machine's hostname can reach the network interface. (Sometimes this is not possible during early stages of boot, where the network interface is taking a while to start up).

For checks above and beyond the above features, write your own script in a scripting langauge like perl or python, and configure the rush.conf file's bootcheck_cmd to run your script instead.

WHY HAVE 'rush -bootcheck' INSTEAD OF A SCRIPT?

    Why is the default to use 'rush -bootcheck', and not the rush/etc/bootcheck.pl?

    Because: perl is not native on Windows, and the Rush core tools (rush + rushd) need to work without a dependency on any specific 3rd party software (such as perl or python).

    Historically Windows has never had a decent native scripting language; DOS has severe limitations.

    Recent Windows versions were found to start Rush before the machine's static IP is assigned to the network interface. Rush would often be started while the machine's network interface is still being initialized; rush would find an IP of 127.0.0.1 or 169.x.x.x instead of the proper address, even when statically assigned. And thus be unable to communicate on the network.

    So, Rush now has to come with these checks pre-enabled by default on Windows.

    Rather than hard-code the check into the daemon, it was implemented as a 'bootcheck_cmd' that invokes this new 'rush -bootcheck' flag with arguments that can be modified or disabled, and without a scripting language needed.

    Out of the box, Rush should be able to boot without having a dependency on any specific 3rd party scripting languages (e.g. perl, python..). While Rush comes with examples in perl/python/etc, they're optional. It's always been an edict that the core Rush utilities, rush + rushd, have no dependency on any specific scripting language.

    Out of the box, Rush daemons should be able to start reliably without a scripting language installed, as some users of Rush don't use a scripting language; some use their own inhouse tools written in C, or native DOS, or some other scripting language like PHP or Lua.

    Note: You can change the default rush.conf 'bootcheck_cmd' to instead run the rush/etc/bootcheck script, replacing the boilerplate 'rush -bootcheck' command. The perl script is more configurable. Or, supply your own bootcheck script in the language you prefer.

See Also

   rush -catlog  
rush -catlog [-foe] <file> [file..] [host..]

cat(1) one of the daemon's logs.

View one of the various local [remote] daemon logs; access both new and old versions of rushd.log, cpu.acct, etc.

'file' can be any one of:

    Catlog File Options
    rushd.log View today's daemon log
    Orushd.log View yesterday's daemon log (O='Old')
    cpu.acct View current rush accounting log
    Ocpu.acct View last rotated (O='Old') accounting log
    mail.log View the mail.log (if enabled via smtp.debug)

[-foe] "Fail On Error". Causes the command to stop on any hostname lookup errors (such as temporary DNS or LDAP issues), instead of just printing warnings and continuing to run to handle any other hostnames specified. (default: off) *This flag new in 103.07*

What follows are some examples of using 'rush -catlog'..

    Catlog Examples
    rush -catlog rushd.log View the local rushd.log file
    rush -catlog rushd.log geneva View geneva's daemon log
    rush -catlog Orushd.log geneva View yesterday's daemon log from host 'geneva'
    rush -catlog cpu.acct View the local accounting log
    rush -catlog cpu.acct +any | cpu-utilization.pl -asciigraph Generate a cpu utilization report from all host's accounting logs

Caveat: This command makes the older '-dcatlog' command redundant.

   rush -checkconf  
rush -checkconf [filename|-]

Check for errors in specified rush.conf file..

(Administrative) Checks a rush/etc/rush.conf file for errors.
Returns an exit code indicating success or failure; 0=ok, 1=error(s) found.
If filename is unspecified, $RUSH_DIR/etc/rush.conf is checked.
If filename is '-', contents to be checked is read from stdin.

NOTE: In Rush 102.42a9c and older: <filename|-> was REQUIRED.

Example:

    Example CSH Script Using 'rush -checkconf'
    #!/bin/csh -f
    
    ###
    ### Script to edit the rush.conf file, and rdist it out                        
    ###
    
    set TMPFILE=/usr/tmp/rush.conf.$$
    cp /usr/local/rush/etc/rush.conf $TMPFILE      # work on a COPY, in case user really messes it up
    vi $TMPFILE
    
    rush -checkconf $TMPFILE
    if ( $? ) then
        echo Config file has errors
        set err=1
    else
        foreach i ( tahoe superior erie )
            echo --- $i
            rdist -c $TMPFILE ${i}:/usr/local/rush/etc/rush.conf
        end
        set err=0
    endif
    rm -f $TMPFILE
    exit $err
            

To use rsync(1), try replacing 'rdist -c' with 'rsync -avz' in the above example.

   rush -checkhosts  
rush -checkhosts <filename|->

Checks for errors in the specified rush/etc/hosts file.

(Administrative) Checks a rush/etc/hosts file for errors.
Returns an exit code indicating success or failure; 0=ok, 1=error(s) found.
If filename is unspecified, $RUSH_DIR/etc/hosts is checked.
If filename is '-', contents to be checked is read from stdin.

NOTE: In Rush 102.42a8 and older: <filename|-> was REQUIRED.

Example:

    Example CSH Script Using 'rush -checkhosts'
    #!/bin/csh -f
    
    ###
    ### Script to edit the rush/hosts file, and rdist it out                        
    ###
    
    set TMPFILE=/usr/tmp/rush-hosts.$$
    cp /usr/local/rush/etc/hosts $TMPFILE      # work on a COPY, in case user really messes it up
    vi $TMPFILE
    
    rush -checkhosts $TMPFILE
    if ( $? ) then
        echo Rush hosts file has errors
        set err=1
    else
        foreach i ( tahoe superior erie )
            echo --- $i
            rdist -c $TMPFILE ${i}:/usr/local/rush/etc/hosts
        end
        set err=0
    endif
    rm -f $TMPFILE
    exit $err
            

To use rsync(1), try replacing 'rdist -c' with 'rsync -avz' in the above example.

   rush -command  
rush -command 'cmd [args..]' [jobid..]
rush -command [jobid..] < file

Set Render Command.

Changes the command that is executed every frame, normally set in your Submit Script by the Command. Example:

If 'cmd [args]' are not supplied, the command will be read from stdin. This is useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into one single line.

    Command Examples
    rush -command '/bin/sleep 30' Set command to sleep 30 seconds per frame
    rush -command < file Set command to contents of 'file' on stdin
    rush -command '/job/run/myrender' tahoe.21 tahoe.22 Change command for two jobs

Note that a full path to the command is normally required. Spaces are used to separate arguments, but the entire command and arguments must be quoted.

The command is usually a simple Render Script. Here are some example 3rd Party Render Scripts.

See Also

    Command -- Submit command to set the job's render command
    rush -commandflags -- Sets flags that affect how the render command is executed

   rush -commandflags  
rush -commandflags option[,option] [jobid..]

Set Command Flags.

(New in 103.00)
Sets optional flags that affect how the render command is executed.

The specified flags are logically 'OR'ed to augment existing flags. So if the current flags are 'quote,escape', and you then run 'rush -commandflags useshell,quote', the result will leave 'quote,escape,useshell' enabled.

If you want to set ONLY the flags you specify, precede your list of flags with 'off' to zero out any existing flags first. Then your flag specifications will be appended to that. (Note: flags are parsed left-to-right, so you want 'off' to be first in the list). So if the current flags are 'quote,escape', and you run 'rush -commandflags off,quote,useshell', the result will leave only 'quote,useshell' enabled.

The following options are supported, which can be separated by commas to specify several at a time:

    -commandflags Options
    quote Honor double quotes in the command.
    This can be used to protect spaces in command arguments.
    escape Honor escape characters (\) in the command.
    Use this to escape certain characters (like '$' and '"').
    Use this to protect quotes-within-quotes (if 'quote' is specified)
    and to protect '$' (if 'expandvars' is specified)
    expandvars Expand environment variables in the command
    before it's executed. Variables can be specified in the format ${VARNAME}
    across both platforms.
    useshell Run the command in a native shell.
    Under windows this means a DOS "cmd" shell.
    Under unix this means a Bourne /bin/sh.
    off Disable all command flags.
    Use this at the beginning of your list of flags

   rush -cont  
rush -cont [jobid..]
rush -cont -

Continue a Paused Job.

Continues a job that was paused with rush -pause.

If '-' is specified, jobids are read from stdin. (Useful if the list of jobids exceeds operating system limits for argument lists and environment variable sizes)

Pausing a job no longer cancels 'Waitfor'; it was decided in 102.40h that the older behavior of having 'rush -cont' cancel 'WaitFor' was a bug, not a feature. The proper way to cancel WaitFor to get a job running is to use 'rush -waitfor -'.

See Also

    WaitFor -- Submit command to defined other jobs to 'wait for'

   rush -cp  
rush -cp [jobid..] <.JobTid [..]> <@[+-]pri['kar']>
rush -cp [jobid..] <cpuspec [..]> <=#cpus>

Change Priority.

Changes priority for the cpus specified.

The cpus to be changed can either be specified as 'cpuspec' (eg. "+any=10@100", or '*' for ALL cpus in the job), or by individual 'JobTid' values to change specific host values. JobTid values ("Job Task IDs") can be found in the 'rush -lc' report.

Examples:

    rush -cp examples
    rush -cp +any=50@100 @200 Change the priority from 100 to 200
    rush -cp +any=50@100 @+10 Increase the priority by 10 (from 100 to 110)
    rush -cp "*" @+10 Increase the priority of ALL cpus in the job by 10.
    rush -cp +any=50@100 =40 Change the cpu cap from 50 down to 40
    rush -cp +any=50@100 =40@200 Change the cpu cap from 50 down to 40, and the priority up from 100 to 200

Changing Priority of Cpus

Priorities can be changed while cpus are busy. If a busy cpu's priority is changed, the busy task(s) inherit the new priority, which is used for all subsequent scheduling and priority battles.

You can speicify a new priority as a fixed, e.g. "@100" to change the priority to "100".

Or you can offset the existing priority by a +/- value, e.g. '@+5' to raise the priority by 5, '@-5' to lower the priority by 5. When raising or lowering priority with offsets, the resulting value is silently clamped within the range specified with the rush.conf file's priority.range.

Examples:

    Example of Changing Cpu Priorities
    
    % rush -lc
    CPUSPEC[HOST]        STATE       FRM  PID     JOBTID  PRI   ELAPSED 
    +any=10@100[vaio]    Busy        0007 1030    131     100   00:01:45 
    +any=10@100[vaio]    Busy        0008 1031    132     100   00:01:45 
    +any=10@100[tahoe]   Busy        0004 19829   133     100   00:00:05 
    +any=10@100[tahoe]   Busy        0005 19831   134     100   00:00:05 
    
    % rush -cp '*' @+5                        
    vaio.21 +any=10@105: priority was +any=10@100, now +any=10@105
    
             PRIORITY HAS NOW BEEN INCREASED BY 5 FOR ALL CPUS IN THE JOB vaio.21 
    
    % rush -lc
    CPUSPEC[HOST]        STATE       FRM  PID     JOBTID  PRI   ELAPSED
    +any=10@105[vaio]    Busy        0007 1030    131     105   00:02:03 
    +any=10@105[vaio]    Busy        0008 1031    132     105   00:02:03 
    +any=10@105[tahoe]   Busy        0004 19829   133     105   00:00:28 
    +any=10@105[tahoe]   Busy        0005 19831   134     105   00:00:28 
    
    % rush -cp '*' @-5
    vaio.21 +any=10@100: priority was +any=10@105, now +any=10@100
    
             PRIORITY HAS NOW BEEN DECREASED BY 5 FOR ALL CPUS IN THE JOB vaio.21 
    
    % rush -lc
    CPUSPEC[HOST]        STATE       FRM  PID     JOBTID  PRI   ELAPSED
    +any=10@100[vaio]    Busy        0007 1030    131     100   00:02:14 
    +any=10@100[vaio]    Busy        0008 1031    132     100   00:02:14 
    +any=10@100[tahoe]   Busy        0004 19829   133     100   00:00:39 
    +any=10@100[tahoe]   Busy        0005 19831   134     100   00:00:39 
    
    % rush -cp .131 .132 @350k
    vaio.21 +any=10@100: priority was 100, now 350k
    vaio.21 +any=10@100: priority was 100, now 350k
    
             PRIORITY FOR TWO CPUS IN vaio.21 (SPECIFIED BY JOBTID'S) HAVE BEEN SET TO 350k 
    
    % rush -lc
    CPUSPEC[HOST]        STATE       FRM  PID     JOBTID  PRI   ELAPSED
    +any=10@100[vaio]    Busy        0007 1030    131     350k  00:02:25 
    +any=10@100[vaio]    Busy        0008 1031    132     350k  00:02:25 
    +any=10@100[tahoe]   Busy        0004 19829   133     100   00:00:40 
    +any=10@100[tahoe]   Busy        0005 19831   134     100   00:00:40 
    
            

Caveats:

  • The CPUSPEC column will not reflect the priority change if single <.jobtid> values are modified, only the effective priority (shown in the PRI column) will show the new value. The only time the CPUSPEC will reflect a priority change is the entire cpuspec is being changed (<cpuspec>='+any=10@100') or if ALL cpus in the job are being changed (<cpuspec>='*')

  • GUIs (like IRUSH) make best use of this option, since they hide the mundane task of visually finding JOBTIDs from the -lc report, and transcribing them with dot prefixes for the -lc command.

Changing Cpu Limits For Cpus

'rush -cp' can be used to change the number of cpus in a cpuspec. For instance, 'rush -cp +any=50@100 =40' will change '+any=50@100' to '+any=40@100' without killing any running frames.

Example:

    Example of Changing Cpu Limits
    
    % setenv RUSH_JOBID vaio.36
    % rush -lc
    CPUSPEC[HOST]          STATE       FRM  PID     JOBTID  PRI   ELAPSED  NOTES
    +any=10@500[tahoe]     Busy        0012 11443   343     500   00:00:10 
    +any=10@500[tahoe]     Busy        0013 11445   344     500   00:00:10 
    +any=10@500[nt-1]      Busy        0008 293     345     500   00:00:11 
    +any=10@500[nt-1]      Busy        0009 246     346     500   00:00:11 
    +any=10@500[vaio]      Busy        0015 11489   347     500   00:00:07 
    +any=10@500[vaio]      Busy        0014 11484   348     500   00:00:09 
    +any=10@500[imac]      Busy        0011 22530   351     500   00:00:11 
    +any=10@500[imac]      Busy        0010 22528   352     500   00:00:11 
    
    % rush -cp +any=10@500 =3@500
    vaio.36 +any=3@500: priority was +any=10@500, now +any=3@500
    
             NOW SET TO 3 CPUS, BUT RENDERS STILL RUNNING, NOTHING IS KILLED..
    
    % rush -lc
    CPUSPEC[HOST]          STATE       FRM  PID     JOBTID  PRI   ELAPSED  NOTES
    +any=3@500[tahoe]      Busy        0021 11453   343     500   00:00:41 
    +any=3@500[tahoe]      Busy        0020 11451   344     500   00:00:41 
    +any=3@500[nt-1]       Busy        0016 280     345     500   00:00:43 
    +any=3@500[nt-1]       Busy        0017 71      346     500   00:00:43 
    +any=3@500[vaio]       Busy        0022 11499   347     500   00:00:39 
    +any=3@500[vaio]       Busy        0023 11501   348     500   00:00:39 
    +any=3@500[imac]       Busy        0018 22534   351     500   00:00:42 
    +any=3@500[imac]       Busy        0019 22536   352     500   00:00:42 
    
             TIME PASSES, RENDERS START FINISHING, 3 CPU LIMIT TAKES EFFECT
    % rush -lc
    CPUSPEC[HOST]          STATE       FRM  PID     JOBTID  PRI   ELAPSED  NOTES
    +any=3@500[tahoe]      Busy        0026 11459   343     500   00:00:04 
    +any=3@500[tahoe]      JobPass     -    -       344     500   00:01:01 Cpu max (3/3 busy)
    +any=3@500[nt-1]       Idle        -    -       345     500   00:01:01 Cpu max (4/3 busy)
    +any=3@500[nt-1]       Idle        -    -       346     500   00:01:01 Cpu max (4/3 busy)
    +any=3@500[vaio]       JobPass     -    -       347     500   00:01:01 Cpu max (3/3 busy)
    +any=3@500[vaio]       JobPass     -    -       348     500   00:01:01 Cpu max (3/3 busy)
    +any=3@500[imac]       Busy        0024 22540   351     500   00:00:07 
    +any=3@500[imac]       Busy        0025 22542   352     500   00:00:05 
    
            

See Also

    Cpus -- Submit command has details on the syntax of cpu specifications
    rush -rc' -- Command line flag to remove cpus from an existing job

   rush -criteria  
rush -criteria 'criteria strings' [jobid..]

Job Criteria.

Sets the criteria string for the job. Use '-' to specify no criteria.

See Also

    Criteria -- Submit command to set host criteria

   rush -d  
rush -d [flags..]

Rush client side debugging.

Enables debugging messages in the 'rush' client program only. (This does not affect the daemon).

Supports the same flags as the -dlog option; see the table of logging flags for the complete list of supported flags.

Examples:

    'rush -d [flags]' Examples
    rush -lf -d t Enable TCP debug messages (t) for the '-lf' command.
    rush -laj -d u Enable UDP debugging messages (u) for the '-laj' command.
    rush -lf -d a Enable ALL debugging messages (a) for the '-lf' command.

See Also

    rush -dlog' -- full description of all the [flags]

   rush -dcatlog  
rush -dcatlog [host..]

NOTE: This option is deprecated, in favor of rush -catlog

Daemon cat(1) Log File.

View the rushd.log daemon on the local [remote] host.

  • rush -dcatlog prints the current rushd.log (usually today's)
  • rush -dcatoldlog prints the last rotated log (usually yesterday's).

Caveat: for reasons strange and unusual, all daemon 'catlog' output has a '>' prepended to each line.

   rush -deltaskfu  
rush -deltaskfu <jobid> <JobTid>
rush -deltaskfu <jobid> <CpuTid> cpuserver

Delete Task, Force Unconditionally

(Administrative) Users should not use this command. Use 'rush -rc' to remove cpus from your job.

Deletes individual 'tasks' from a job server or cpu server. Only root and the rush adminuser can use this command.

This command has two forms; in one case it takes two arguments; a jobid and JobTid, to forcibly remove a task from a job's list of cpu reservations. The second form it takes 3 arguments; a jobid, a CpuTid, and a hostname of a cpu server, forcibly removing that task from the cpu server's tasklist (i.e. 'rush -tasklist <hostname>'). Examples:

    Delete Task Examples
    rush -deltaskfu vaio.55 117 Delete JOBTID #117 from job vaio.55
    rush -deltaskfu vaio.55 183 tahoe    Delete CPUTID #183 (from 'rush -tasklist') from host tahoe for jobid vaio.55

JOBTID values can be found in -lc, -lcf and -tasklistfull reports.
CPUTID values can be found in -lcf and -tasklistfull reports.

Caveat: This command is not intended for casual use. It is to be used by administrators in unusual situations where cpus are stuck due to communications problems between machines. Use this command carefully. Users should really use 'rush -down' instead, which is the best way to tell Rush a remote machine is 'down' (or not responding).

   rush -dependon  
rush -dependon [-|depjobid[,..]]

Set Dependency On Other Jobs.

Sets new 'dependon' jobids for the current [specific] job. See the submit script docs on DependOn for more info.

Read the examples carefully, to avoid confusing jobid(s) to affect with new jobids for the dependon command; remember to separate all 'dependon' jobids with commas instead of spaces.

Caveats:

Setting 'dependon' to a new value disables all previous values (i.e., clears them first).

Setting dependon to '-' will disable it, but you will have to un-Hold any frames manually.

All 'dependon' jobids must have the same hostname as the job being modified.

See also, rush -waitfor.

To differentiate between the new value for 'dependon', and the jobid(s) to be affected, the new value for 'dependon' must be a comma-separated list of jobids with no spaces. Examples:

    Examples of using 'rush -dependon'
    rush -dependon - Disable all dependencies
    rush -dependon tahoe.37,tahoe.38, Job now depends on 'tahoe.37' and 'tahoe.38'.
    Trailing comma required
    rush tahoe.40 -dependon tahoe.37,tahoe.38, tahoe.40 now depends on 'tahoe.37' and 'tahoe.38'
    rush tahoe.40 -dependon tahoe.37, tahoe.40 now depends on job 'tahoe.37'
    Trailing comma on tahoe.37 is required

See Also

    -dependonstate -- Rush command line flag to change the job's DependOn frame state
    DependOn -- Submit command to defined jobs this job should 'depend on'
    DependOnState -- Submit command to set the DependOn frame states

   rush -dependonstate  
rush -dependonstate <done|fail|donefail> [jobid]

Controls circumstances that cause DependOn to trigger.

(New in 103.07e)
See the submit script docs on DependOnState for more info.

Examples:

    'rush -dependonstate' Examples
    rush -dependonstate done # Causes 'dependon' to trigger when corresponding frame is Done
    rush -dependonstate fail # Causes 'dependon' to trigger when corresponding frame is Fail
    rush -dependonstate donefail # Causes 'dependon' to trigger when corresponding frame is Done or Fail

See Also

    rush -dependon -- Rush command line flag to change a job's existing dependon jobids
    DependOnState -- Submit command to set state 'DependOn' jobs wait for
    DependOn -- Submit command to set jobid of other jobs whose frames we wait for
    WaitFor -- Submit command to make a job wait for a specific time or for other jobs
    WaitForState -- Submit command to set the job state we expect to see for other jobs

   rush -dexit  
rush -dexit [-foe] [remotehost..]

Daemon Exit.

(Administrative) Tells a daemon to exit immediately via TCP. Acknowledges success. Only root and the rush adminuser can use this command.

Automatically saves job checkpoint before exiting.

[-foe] "Fail On Error". Causes the command to stop on any hostname lookup errors (such as temporary DNS or LDAP issues), instead of just printing warnings and continuing to run to handle any other hostnames specified. (default: off) *This flag new in 103.07*

   rush -dexitnow  
rush -dexitnow [-foe] [remotehost..]

Daemon Exit Now.

(Administrative) Tells daemon to exit immediately, via UDP. There is no acknowledge. Only root and the rush adminuser can use this command. Use only in case of emergency.

Automatically saves job checkpoint before exiting.

[-foe] "Fail On Error". Causes the command to stop on any hostname lookup errors (such as temporary DNS or LDAP issues), instead of just printing warnings and continuing to run to handle any other hostnames specified. (default: off) *This flag new in 103.07*

   rush -dlog  
rush -dlog <flags> [remotehost..]

Daemon Debug Logging

(Administrative) Enables certain daemon debug logging flags. Only root and the rush adminuser can use this command. Use 'rush -dlog -' to disable all debugging.

    Logging Flags
    These are flags that can be used with:

    • 'rush -dlog' to enable daemon debugging messages in the rushd.log, eg: 'rush -dlog utE'
    • 'rush -d' to enable client side debugging messages to the user's screen, eg: 'rush -d t -lfi'
    • In the rush.conf file's LogFlags

    These flags can be combined to accumulate logging verbosity.

    • All flags can be enabled by specifying 'a', e.g. 'rush -dlog a'.
    • All flags can be disabled by specifying '-', e.g. 'rush -dlog -'


    
        - - A single dash is used to disables all debugging flags
        a - all
        b - bump mechanism logging
        c - select() info
        d - log duplicate/redundant receipt of packet drops
        e - events (time oriented, async, lic unpause)
        f - fork/wait
        g - taskcleanup related (103.00+)
        h - hostname lookups
        j - Log job submissions and job state changes
        k - Log bumped/killed/usurped tasks
        l - Logical string evaluations
        m - memory calculations (RAM) during priority battles, etc
        n - network commands (udp/tcp)
        o - connect()/open()/close()/bind()/socket() (low level)
        p - parse command line arguments, submit scripts
        q - sequence id errors are printed (default:off)
        r - reboot management/transactions, and checkpoint purges of old tasks
        s - signals
        t - tcp
        u - udp
        w - 'waitfor' and 'dependon' checks
        x - show negative cache events
        y - yp lookups
        B - debugging for BadJobid transmissions
        C - class ToWords/FromWords (PROBABLY OBSOLETE)
        D - dumpsig debugging (Bluesky: 1.21)
        F - File loading line-by-line debugging
        E - Errors not normally displayed (benign, but suspect)
        J - Job Objects (win2k job objects kill mechanism msgs)
        K - Actual kill commands executed
        L - Locks on files
        M - Microsoft specific debugging
        N - Newtask and notify advertising
        O - (The letter 'O', NOT zero) program loops
        P - server: Priority mechanism, client: log 'rush -push' mechanism
        Q - monitor timeouts
        R - Reaper msgs
        S - Server/Client context switches
        T - task/taskack transactions
        U - update (scheduling, priority mechanism, idle cpu management)
        V - ram choking info (expandcpus_ramchoke)
        W - 'Wakeup' pass states; pass->unpass state changes
        X - Random UDP message dropping -- TESTING ONLY!!
            ('a' does not affect this option, it must be specified explicitly)
        Z - reserved for on the fly debugging problems
        0 - (ZERO) jobcheckpoint snapshot messages
    
        @ - PROTOCOL TEST FOR TASK_TIMEOUT -- TESTING ONLY!!
            Induces 60 second sleep before sending TASK_START.
            ('a' does not affect this option, it must be specified explicitly)
    
        + - Disables cpu intensive bump checks
    
        % - Debug Start state problem (timeouts, task state changes)
    
        ^ - Update choke messages
    
            

Examples:

    Examples: Enabling And Disabling Rushd 'Debug Logging'
    rush -dlog ut tahoe      Enable UDP (u) and TCP (t) debugging for daemon on host tahoe
    rush -dlog - tahoe Disable all debugging flags for the daemon on host tahoe

   rush -dlogstats  
rush -dlogstats [host|+hostgroup..]

Daemon Log Statistics

(Administrative) Useful for easily determining problems with machines on large networks.

Checks the local [remote] daemon log for particular kinds of errors, and logs the number of times each error appears in a table summary.

    Check Daemon's Log File Statistics
    % rush -dlogstats +any
    #                        ************ Problems ************* **Security* ************* Normal **************
    # SID   Hostname         Alert  Fail Error  Warn  Exit  Fork  Bump Secur Start   Rot  Info JobSt JobEn JobKi
    l 0     geneva               2     0     0     0     0     0     0     0     0     1     0     0     0     0
    l 0     rotwang              0     0     0     0     1     0     0     1     1     1     2     0     0     0
    l 0     ontario              0     0     0     0     0     0     0     0     1     1     2     1     0     0
        

This report shows that geneva has two 'ALERT' errors in its daemon log, which might indicate a problem worth investigating with 'rush -catlog rushd.log geneva | grep ALERT'.

    Debug Log Statistics: Field Descriptions
    Alert Possible system problem; a systems operation unexpectedly
    failed. For instance, daemon can't bind() to port, because
    some other entity is holding the port open. Or a select()
    that should either timeout or return data, actually failed.
    Or, a tcp/udp command was malformed.
    Fail Normal operations, failure condition detected and handled.
    Error An error condition (usually enabled via 'rush -dlog E')
    Warn Warn of a non-error condition (ie. use of obsolete ops)
    Exit Error so fatal, it caused daemon to exit.
    Fork Unable to fork a new process.
    Bump A job was bumped (killed) during eg. a priority battle.
    Secur(e) Attention to sysadmins: someone is trying to circumvent
    the permissions. eg. someone is changing a job they don't own
    with '-fu'.
    Start Daemon was started.
    Rot(ate) rushd.log was rotated.
    Info Normal operations information worthy of note.
    JobSt(art) A job was started.
    JobEn(d) A job ended.
    JobKi(ll) A job was dumped.

   rush -drivemaps  
rush -drivemaps

Drive Mappings Report.

Lists any drive mappings configured in the rush/etc/path_convert file. The output of this command can be used by scripts to determine which drive mappings should be in place.

    'rush -drivemaps' Example
    % rush -drivemaps
    x //server/prod1
    y //server/prod2
    z //server/prod3
    
    % grep drive_map /usr/local/rush/etc/path_convert
    os=all  drive_map       "x:"    "//server/prod1"
    os=all  drive_map       "y:"    "//server/prod2"
    os=all  drive_map       "z:"    "//server/prod3"
    
    %
    

   rush -dstats  
rush -dstats [-z] host

Daemon Statistics.

(Internal use). Show the daemon's internal statistics. Used mainly for debugging.

If -z is specified, it zeroes out the statistics.

    'rush -dstats' Example
    % rush -dstats tahoe
               Host: tahoe
    JixHits(Misses): 485 (0)    # number of successful job index lookups, value in parens indicates number of misses.
    CixHits(Misses): 491 (0)    # number of successful cpu index lookups, value in parens indicates number of misses.
    TixHits(Misses): 491 (0)    # number of successful task index lookups, value in parens indicates number of misses.
    FixHits(Misses): 286 (0)    # number of successful frame index lookups, value in parens indicates number of misses.
    SchedulerElapsed[0]: 05/18,10:37:42 -- scheduler took 0 secs, _Update() ran 1 loops, AdvertiseIdle took 0 passes  # Info about the
    SchedulerElapsed[1]: 05/18,10:37:34 -- scheduler took 0 secs, _Update() ran 1 loops, AdvertiseIdle took 0 passes  # last 10 runs
    SchedulerElapsed[2]: 05/18,10:37:26 -- scheduler took 0 secs, _Update() ran 1 loops, AdvertiseIdle took 0 passes  # of the scheduler
    SchedulerElapsed[3]: 05/18,10:37:18 -- scheduler took 0 secs, _Update() ran 1 loops, AdvertiseIdle took 0 passes
    SchedulerElapsed[4]: 05/18,10:37:10 -- scheduler took 0 secs, _Update() ran 1 loops, AdvertiseIdle took 0 passes
    SchedulerElapsed[5]: 05/18,10:37:02 -- scheduler took 0 secs, _Update() ran 1 loops, AdvertiseIdle took 0 passes
    SchedulerElapsed[6]: 05/18,10:36:54 -- scheduler took 0 secs, _Update() ran 1 loops, AdvertiseIdle took 0 passes
    SchedulerElapsed[7]: 05/18,10:36:46 -- scheduler took 0 secs, _Update() ran 1 loops, AdvertiseIdle took 0 passes
    SchedulerElapsed[8]: 05/18,10:36:38 -- scheduler took 0 secs, _Update() ran 1 loops, AdvertiseIdle took 0 passes
    SchedulerElapsed[9]: 05/18,10:36:30 -- scheduler took 0 secs, _Update() ran 1 loops, AdvertiseIdle took 0 passes
               Jobs: 62         # How many jobs are being served by this machine
              Procs: 0          # How large the G_procs[] array is
              Tasks: 0          # How many tasks are in this cpuserver's tasklist
           LogFlags: -          # Current -dlog flags enabled for the daemon
             NextFD: 7          # The next unused file descriptor (useful for detecting unclosed files)
      SecsSinceBoot: 425002     # How long the daemon has been running
     UdpTransactors: 0 (ttl=10) # UDP packet assembly debugging info
    

   rush -done  
rush -done <framerange|framestate..> [jobid..]

Done Frames.

Changes frame(s) into the Done state for current [specific] job. If frames are running, they are killed and enter the 'Done' state on completion.

The frames to affect can either be specified as a frame range (i.e., 1-100) or as a frame state, for which all matching frames will be affected. Examples:

    'rush -done' Examples
    % rush -done 1-100 'Done' frames 1 through 100
    % rush -done fail 'Done' all frames currently Fail
    % rush -done fail que    'Done' all frames currently Fail or Que  
    % rush -done all 'Done' all frames

   rush -donemail  
rush -donemail [email[,email..]] [jobid..]

Set Job Done Mail.

Changes who to send mail to when job is done. This means when the job renders its last frame, or when the job is dumped. Which means if you requeue frames after a job is done, you will receive an email each time the last frame is rendered.

To set several names, separate the names by commas. Each time you set the value, the previous value is overwritten.

See DoneMail submit command for more info and examples.

See also DumpMail submit command if you want mail sent only once, when the job is actually dumped.

    Done Mail Examples
    rush -donemail fred@foo.com,jack@foo.com Fred and Jack will receive mail
    when the current job is done.
    rush -donemail fred@foo.com tahoe.37 Fred will receive mail
    when jobid tahoe.37 is done.
    rush -donemail - tahoe.37 tahoe.38 Disable all done emails
    for jobids tahoe.37 and tahoe.38  

See Also

    DoneMail -- Submit command defines who to send email to when job is done
    DumpMail -- Submit command defines who to send email to when job is dumped
    rush -dumpmail -- Command line flag changes who to send email when job is dumped

   rush -down  
rush -down downhost[,downhost..] jobhost

A Machine Is Down.

Tells rush jobs a machine is down.

If a machine hangs or goes down, rush has no way to know it.. for all rush knows, the remote machine is just under heavy load, and is being unresponsive.

Use '-down' to requeue frames in jobs that are stuck in the Run or Start state, because the remote machine is either hung or has been turned off unexpectedly.

Normally frames will requeue when the remote is rebooted, but in cases where the machine needs to stay off for a while, the user can use '-down' to manually tell rush the remote is down, and to requeue the hung frames elsewhere.

This also helps jobs that won't dump because some frames are 'stuck' on a host that is down.

Warning: This command should only be used in extreme situations where job(s) show the frames running on a remote machine that is known to be down, or not running the frame.

    'rush -down' Examples
    rush -down huron,erie tahoe   Tell all jobs on tahoe that machines   
    'huron' and 'erie' are down
    rush -down huron,erie +any Tell all hosts that 'huron' and 'erie'
    are down

   rush -dump  
rush -dump [jobid|user|user@host..]
rush -dump -

Dump Job.

Dumps the job. Busy frames are killed right away.

If no arguments are specified, the RUSH_JOBID variable is used to determine which job to dump.

If jobid(s) are specified as arguments, those jobs will be dumped.
If '-' is specified as an argument, jobids are read from stdin. (Useful if the list of jobids exceeds operating system limits for argument lists and environment variable sizes)

If a user's name is specified as an argument, all jobs owned by that user on the local machine will be dumped (e.g., 'rush -dump fred').

If user@host is specified as an argument, all jobs owned by 'user' served by 'host' will be dumped.
i.e., 'rush -dump fred@tahoe' will dump all jobs owned by fred that were submitted to host tahoe.

    'rush -dump' Examples
    % rush -dump tahoe.37 tahoe.38    Dumps jobs tahoe.37 and tahoe.38
    % rush -dump fred Dumps all jobs owned by 'fred' served by the local machine  
    % rush -dump fred@superior Dumps all jobs owned by 'fred' served by host 'superior'

   rush -dumpmail  
rush -dumpmail [email[,email..]] [jobid..]

Set Job Dump Mail.

Changes who to send mail to when job is dumped.

To set several names, separate the names by commas. Each time you set the value, the previous value is overwritten.

See DumpMail submit command for more info and examples.

If you want mail sent whenever the job finishes rendering the last frame, then use instead -donemail.

    Dump Mail Examples
    rush -dumpmail fred@foo.com,jack@foo.com Fred and Jack will receive mail
    when the current job is dumped.
    rush -dumpmail fred@foo.com tahoe.37 Fred will receive mail
    when jobid tahoe.37 is dumped.
    rush -dumpmail - tahoe.37 tahoe.38 Disable all dump emails
    for jobids tahoe.37 and tahoe.38  

See Also

    DoneMail -- Submit command defines who to send email to when job is done
    DumpMail -- Submit command defines who to send email to when job is dumped
    rush -donemail -- Command line flag changes who to send email when job is done

   rush -end  
rush -end [jobid..]
rush -end -

End Job.

Ends a job. Similar to rush -dump, but lets running frames finish (i.e., running frames are not killed).

See Also

    rush -dump -- Dump a job and kill running frames

   rush -exitnotes  
rush -exitnotes "text message"

Exit Notes for frame.

Use this from within render scripts as an alternative to 'rush -notes' to change the NOTES field of 'rush -lf' reports when a render script finishes rendering a frame.

This command should ONLY be used from within a running render script to change the notes for the currently rendering frame just before the script is about to exit. Do NOT use this command to attempt to change frames in other jobs, or different frames within a job; use 'rush -notes' in such cases.

The command will not take effect until the process actually exits; this is due to how the notes are delivered to the jobserver as part of the process exit code delivery.

Unlike most rush commands, this command does not accept command line arguments to specify the jobid or frame number; it is assumed this command will only be run from within the render script during its execution, and it therefore uses the environment variables RUSH_JOBID and RUSH_FRAME (which are automatically in the render script's environment) to determine which job/frame to affect. For this reason, do not attempt to unset or modify these variables from within the render script, and do not attempt to force-set these variables in attempt to subvert this command to be used in other contexts.

Example Use

This command is safe to use within render scripts in both error and non-error conditions, since it is a low-bandwidth operation, so it can be used when 100's of machines are all rendering at the same time without negatively impacting the job server (as would be the case if 'rush -notes' were used)

    Example use of 'rush -exitnotes'
    #!/bin/csh -f
    ### RENDER SCRIPT
    
    # Render
    render //jobserver/jobs/inuyasha/s8a/dropshadow/${RUSH_PADFRAME}.rib   
    set err = $status
    if ( $err != 0 ) then
        rush -exitnotes "Render failed: exitcode $err"
        exit 1
    endif
    
    # Image integrity check
    imagecheck //jobserver/images/inuyasha/s8a/drop-${RUSH_PADFRAME}.sgi
    if ( $status ) then
        rush -exitnotes "Failed image check"
        exit 1
    endif
    
    rush -exitnotes "Image check OK"
    exit 0

In the above case, as each frame finishes, the 'NOTES' field in the 'rush -lf' frames report might look like:

    Frames Report Showing -exitnotes
    % rush -lf
    STAT FRAME   TRY HOSTNAME        PID     JOBID            START          ELAPSED  NOTES
    Done 0001    2   ontario         19752   ontario.12       05/18,18:51:03 00:01:17 Image OK
    Done 0002    4   ontario         19763   ontario.12       05/18,18:51:05 00:01:15 Image OK
    Fail 0003    3   erie            11112   ontario.12       05/18,18:51:03 00:00:27 Failed image check  
    Fail 0004    9   erie            11121   ontario.12       05/18,18:51:44 00:00:25 Failed image check
    Done 0005    10  cross           19779   ontario.12       05/18,18:52:23 00:00:02 Image OK
    Done 0006    3   cross           19781   ontario.12       05/18,18:52:23 00:00:02 Image OK
    Done 0007    2   cyanic          11129   ontario.12       05/18,18:52:22 00:00:25 Image OK
    

New in Rush 103.06: If no frame# is specified, RUSH_FRAME will be used to determine which frame will be affected.

Low Latency

The use of this command within render scripts is low latency, and the specified notes get piggy-backed along with the exit codes returned to the job server when the render script exits.

Before Rush 103.06, the 'rush -notes' command used TCP for communications, TCP being a higher latency protocol that could overload the job server if too many of these commands happened to run all at once across the farm.

In Rush 103.06, the 'rush -notes' command now uses the lower latency UDP protocol, lessening the need for 'rush -exitnotes'. As well, 'rush -notes' takes effect immediately, making it more flexible should one want to adjust the notes /while/ a frame is rendering (something 'rush -exitnotes' does not support).

   rush -fail  
rush -fail <framerange|framestate..> [jobid..]

Fail Frames.

Changes frame(s) into the Fail state for current [specific] job. If frames are running, they are killed and enter the 'Fail' state on completion.

The frames to affect can either be specified as a frame range (i.e., 1-100) or as a frame state, for which all matching frames will be affected. Examples:

    'rush -fail' Examples
    % rush -fail 1-100 Fail frames 1 through 100
    % rush -fail done Fail all frames currently Done
    % rush -fail done hold    Fail all frames currently Done or Hold  
    % rush -fail all Fail all frames

   rush -fifo-order  
rush -fifo-order <date|timeval> [jobid..]

Fifo Ordering.

(new in 103.06)

Changes the FIFO ordering for a job when FIFO scheduling has been enabled by the administrator.

By definition, a job's FIFO ordering value is determined by the date/time the job was submitted; jobs submitted first will be scheduled to run before jobs submitted later.

This date oriented order can be changed while jobs are running to change the way jobs are scheduled to run by redefining the FIFO order date/time value to some new value.

By default, the 'rush -fifo-order' command is only available to the rush administrators, but individual users can be added by the systems administrator; see FIFO ORDER SECURITY below.

The new FIFO date can be specified either in MM/DD/YY[YY],HH:MM:SS format (e.g. "06/16/2014,10:48:20"), or as a unix time(2) value (e.g. 1402940900). Examples:

    'rush -fifo-reorder' Examples
    % rush -fifo-reorder 06/16/2014,10:48:20 tahoe.12    Set FIFO order for jobid tahoe.12
    to a specific date/time.
    % rush -fifo-reorder 1402940900 tahoe.12 Set FIFO order for jobid tahoe.12
    to a specific time(2) value.

FIFO ORDER SECURITY

By default, the 'rush -fifo-order' command is made available to the rush administrators as defined by the rush.conf 'permit' command for the 'everything' keyword.

However, specific usernames can be enabled to use this option with the permit 'fifo-order' keyword in the rush.conf file, e.g.

    Enabling Specific Users     
    
        permit
        {
            ..
            fifo-order:
            {
                jack
                jane
                wrangler1
                wrangler2
            }
        }
            

To enable all users, '*' can be specified in place of specific usernames.

    Enabling All Users     
    
        permit
        {
            ..
            fifo-order:
            {
                *
            }
        }
            

See Also:

   rush -frameflags  
rush -frameflags <-|keepnotes..> [jobid..]

Change Frame Flags.

Changes flags that affect frames.

    Frame Flags
    rush -frameflags keepnotes Disables automatic clearing of 'NOTES" field in framelist
    when a frame starts rendering.
    rush -frameflags - No flags. (Default)
    Default behavior is for NOTES field in framelist to be cleared   
    whenever a frame starts rendering.

See Also

    FrameFlags -- Submit command to set the job's frame flags

   rush -fu  
rush [args..] -fu

Force User1

The -fu flag augments other operations. It overrides warnings pertaining to lack of permission to do an operation because the user invoking it is not the owner of the job.

For instance, if you want to control another person's job, you might get an error, e.g.:

    Using the 'rush -fu' Flag
    % rush -an vaio va-229
    rush: va-229: you're not owner!  
    Attempt to add 'vaio' as a nevercpu to someone's job   
    fails because you're not the job's owner
    % rush -an vaio va.229 -fu        
    Add nevercpu 'vaio'
    Same command with -fu to force it...
    ...now it works

Note: This feature may be disabled by the system administrator via the DisableFu flag in the Rush Configuration File.

1 This acronym is rumored to have an alternate, pejorative expansion.

See Also

    $RUSH_FU -- An environment variable that enables FU mode
    DisableFu -- A rush config flag to let the admin disable FU mode

   rush -getoff  
rush -getoff [-foe] [-msg 'text'] [remotehost|+hostgroup..]

Get Off A Machine.

Similar to 'rush -offline', but kills and requeues any currently running frames before taking the specified host/hostgroups offline. If no hosts/hostgroups are specified, the local machine's rush daemon is taken offline.

When a machine is taken offline, frames that are already rendering are allowed to complete before taking the processors offline. Once offline, no new frames will be started.

[-foe] "Fail On Error". Causes the command to stop on any hostname lookup errors (such as temporary DNS or LDAP issues), instead of just printing warnings and continuing to run to handle any other hostnames specified. (default: off) *This flag new in 103.07*

An optional remark can be specified via "-msg 'text'" which will show up in 'rush -lac' reports, eg:

    Using 'rush -getoff' with a Custom Message
    % rush -getoff -msg 'Client Demo' ontario
    ontario: Offline (killing 1 procs)
    
    % rush -lac
    HOST       OWNER    JOBID          TITLE                     FRM  PRI   PID        ELAPSED REMARKS
    ---------- -------- -------------- ------------------------- ---- ----- ------- ---------- --------
    ontario    -        -              -                         -    -     -          Offline Client Demo
    :

Examples of 'rush -getoff':

    'rush -getoff' Examples
    % rush -getoff Kills frames running on localhost (requeing them elsewhere)
    leaving the localhost offline (won't accept any more renders)
    % rush -getoff tahoe erie      Kills frames running on tahoe and erie (requeing the frames elsewhere)   
    leaving both hosts offline (won't accept any more renders)
    % rush -getoff -msg 'Bad Memory' huron    Offline 'huron' with the remark 'Bad Memory'

Changes to the online/offline state are appended to the rushd.log.
Also: changes to the online/offline state append 's' entries to the cpu.acct accounting log

   rush -hold  
rush -hold <framerange|framestate..> [jobid..]

Hold Frames.

Changes frame(s) into the Hold state for current [specific] job. If frames are running, they are killed and enter the 'Hold' state on completion.

When a frame is in the Hold state the frame will not be rendered, and a job will not autodump if there are any Hold frames in the frame list.

The frames to affect can either be specified as a frame range (i.e., 1-100) or as a frame state, for which all matching frames will be affected. Examples:

    'rush -hold' Examples
    % rush -hold 1-100 Hold frames 1 through 100
    % rush -hold fail Hold all frames currently Fail
    % rush -hold fail done    Hold all frames currently Fail or Done  
    % rush -hold all Hold all frames

   rush -imgcommand  
rush -imgcommand 'command [args..]' [jobid..]
rush -imgcommand [jobid..] < file

Set Image Display Command

Used by IRUSH to determine the command to execute whenever the user double clicks on a frame in the IRUSH 'Frames' report. Normally, the command invoked should display the rendered image for that frame.

This should be a complete command to view an image in your favorite image viewer, with %d or %04d in place of the frame number.

If 'cmd [args]' are not supplied, the command will be read from stdin. This is useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into one single line.

The image viewing command must background itself, or you must force it by using '&' (Unix), or 'start /b' (Windows). Examples:

    'rush -imgcommand' Examples
    % rush -imgcommand 'xv /job/images/foo.%04d.sgi &' (Unix) Invokes 'xv' to view images
    C:\> rush -imgcommand "start /b flip //nt/images/s1.%04d.png" (Windows) Invokes 'flip' in background   
    to view images

See Also

    ImgCommand -- Submit command sets the imgcommand used by irush

   rush -jobcheckpoint  
rush -jobcheckpoint [-foe] [host|+hostgroup..]

Save all jobs to the checkpoint file

This tells the local host (or remotehost|+hostgroup if specfied) to save a checkpoint now. Checkpoint will be loaded on reboot automatically, if the checkpoint.onboot is set to '1' in the rush.conf file (which is the default).

[-foe] "Fail On Error". Causes the command to stop on any hostname lookup errors (such as temporary DNS or LDAP issues), instead of just printing warnings and continuing to run to handle any other hostnames specified. (default: off) *This flag new in 103.07*

   rush -jobdata  
rush -jobdata [-key keyname] 'data..' [jobid..]
rush -jobdata [-key keyname] [jobid..] < data.txt
rush -jobdata '-' [jobid..]

Set Job Data.

(New in 103.00)
Appends free form strings of 'job data' to the current [specific] job. Job data is viewable in 'rush -ljf'.

This is similar to '-jobnotes'; both keep track of strings of data in the same way. '-jobnotes' is intended for notes to users, or notes about the job. '-jobdata' is more for data communications between programs, such as data payloads to be injected into databases, or data used by GUIs such as IRUSH, submit forms, etc.

If 'data..' is not specified, the data can be supplied on stdin.

If 'data..' is specified on the command line, it must be quoted if it contains spaces.

Specifing 'rush -jobdata -' will clear all previous data.

(New in 103.06)
The new "-key <keyname>" option allows one to change the contents of specific lines in the jobdata that start with the specified <keyname>, where lines are of the form 'keyname: value', example:

    % rush -ljf -fmt JobData tahoe.123
    JobData[0]: SHOW: bigshow
    JobData[1]: SHOT: some shot
    JobData[2]: TD_NOTES: Awaiting approval

    % rush -jobdata -key TD_NOTES 'Director approved.' tahoe.123
    tahoe.123: jobdata for 'TD_NOTES' is now 'Director approved.'

    % rush -ljf -fmt JobData tahoe.123
    JobData[0]: SHOW: bigshow
    JobData[1]: SHOT: some shot
    JobData[2]: TD_NOTES: Director approved.
        

Multiline data may also be modified with -key, e.g.

    % rush -ljf -fmt JobData tahoe.123
    JobData[0]: SHOW: bigshow
    JobData[1]: SHOT: some shot
    JobData[2]: TD_NOTES: This is line one
    JobData[3]: TD_NOTES: This is line two
    JobData[4]: TD_NOTES: This is last line

    % cat /var/tmp/newnotes
    The special effects supe approved on Monday
    but the director asked for some mods.

    % cat /var/tmp/newnotes | rush -jobdata -key TD_NOTES tahoe.123
    tahoe.123: jobdata for 'TD_NOTES' is now 'The special effects supe approved on Monday'
    tahoe.123: jobdata for 'TD_NOTES' is now 'but the director asked for some mods.'

    % rush -ljf -fmt JobData tahoe.123
    JobData[0]: SHOW: bigshow
    JobData[1]: SHOT: some shot
    JobData[2]: TD_NOTES: The special effects supe approved on Monday
    JobData[3]: TD_NOTES: but the director asked for some mods.

    % rush -ljf -fmt 'JobData[TD_NOTES]' tahoe.123
    JobData[TD_NOTES]: The special effects supe approved on Monday
    JobData[TD_NOTES]: but the director asked for some mods.
        

Note: when using -key, it is best to use 'key: values' throughout the jobdata for a job, to avoid having any free-form data confused for key names.

See JobData submit command for more info.

    'rush -data' Examples
    rush -data < foo.txt Appends contents of foo.txt to job's data
    rush -jobdata 'MYPROG: Line one'
    rush -jobdata 'MYPROG: Line two'
    Appends two lines of text to the job's data

You can dynamically modify the job data (e.g. insert, remove, reformat), using these steps:

    1) Use 'rush tahoe.123 -ljf -fmt JobData > tmpfile' to get the current jobdata for jobid tahoe.123 into a file called tmpfile.
    2) Rearrange/amend the data in tmpfile as needed.
    3) Use 'rush -jobdata -' to clear the previous data.
    4) Use 'rush tahoe.123 -jobdata < tmpfile' to update the job with the new data.

See Also

    JobData -- Submit command sets the job's data
    rush -ljf -- Command line 'list job full', which includes jobdata

   rush -jobnotes  
rush -jobnotes [-key keyname] 'notes..' [jobid..]
rush -jobnotes [-key keyname] [jobid..] < notes.txt
rush -jobnotes '-' [jobid..]

Set Job Notes.

Appends free form strings of 'job notes' to the current [specific] job. Job notes are viewable in 'rush -ljf'.

If 'notes..' are not specified, the notes can be supplied on stdin.

If 'notes..' are specified on the command line, they must be quoted if they contain spaces.

Specifing 'rush -jobnotes -' will clear all previous notes.

(New in 103.06)
The new "-key <keyname>" option allows one to change the contents of specific lines in the job notes that start with the specified <keyname>, where lines are of the form 'keyname: value', example:

    % rush -ljf -fmt Notes tahoe.123
    Notes[0]: SHOW: bigshow
    Notes[1]: SHOT: some shot
    Notes[2]: TD_NOTES: Awaiting approval

    % rush -jobnotes -key TD_NOTES 'Director approved.' tahoe.123
    tahoe.123: jobnotes for 'TD_NOTES' is now 'Director approved.'

    % rush -ljf -fmt Notes tahoe.123
    Notes[0]: SHOW: bigshow
    Notes[1]: SHOT: some shot
    Notes[2]: TD_NOTES: Director approved.
        

Multiline data may also be modified with -key, e.g.

    % rush -ljf -fmt Notes tahoe.123
    Notes[0]: SHOW: bigshow
    Notes[1]: SHOT: some shot
    Notes[2]: TD_NOTES: This is line one
    Notes[3]: TD_NOTES: This is line two
    Notes[4]: TD_NOTES: This is last line

    % cat /var/tmp/newnotes
    The special effects supe approved on Monday
    but the director asked for some mods.

    % cat /var/tmp/newnotes | rush -jobnotes -key TD_NOTES tahoe.123
    tahoe.123: jobnotes for 'TD_NOTES' is now 'The special effects supe approved on Monday'
    tahoe.123: jobnotes for 'TD_NOTES' is now 'but the director asked for some mods.'

    % rush -ljf -fmt Notes tahoe.123
    Notes[0]: SHOW: bigshow
    Notes[1]: SHOT: some shot
    Notes[2]: TD_NOTES: The special effects supe approved on Monday
    Notes[3]: TD_NOTES: but the director asked for some mods.

    % rush -ljf -fmt 'Notes[TD_NOTES]' tahoe.123
    Notes[TD_NOTES]: The special effects supe approved on Monday
    Notes[TD_NOTES]: but the director asked for some mods.
        

Note: when using -key, it is best to use 'key: values' throughout the job notes for a job, to avoid having any free-form data confused for key names.

See Notes submit command for more info.

    'rush -jobnotes' Examples
    rush -jobnotes < foo.txt Set job notes to contents of foo.txt
    (Previous notes are completely cleared)
    rush -jobnotes 'This is a test' Set job notes to 'This is a test'

You can dynamically modify the job notes (e.g. insert, remove, reformat), using these steps:

    1) Use 'rush tahoe.123 -ljf -fmt Notes > tmpfile' to get the current job notes for jobid tahoe.123 into a file called tmpfile.
    2) Rearrange/amend the notes in tmpfile as needed.
    3) Use 'rush -jobnotes -' to clear the previous notes.
    4) Use 'rush tahoe.123 -jobnotes < tmpfile' to update the job with the new notes.

See Also

    Notes -- Submit command sets the job's 'notes'
    rush -ljf -- Command line flag to 'list job full', which shows the job 'notes'

   rush -jobremarks  
rush -jobremarks 'message_text' [jobid..]

User Defined Job Remarks.

Includes "message_text" in the "Remarks" field of the rush -lj/-laj reports. In the report, message_text will be enclosed in [brackets] to differentiate the user defined remarks from rush's own remarks (see example below). Historical Note: This feature was actually implemented in an earlier release, but fell out of subsequent releases up thru 102.42a8 due to a source code regression, and wasn't reinstated until 102.42a9.

    Example: 'rush -jobremarks'
    % rush -lj
    STATUS JOBID    TITLE      OWNER  %DONE %FAIL BUSY    ELAPSED REMARKS
    ------ -------- ---------- ------ ----- ----- ---- ---------- --------
    Run    tahoe.1  FG_PLATE   erco      %0    %0    8   00:05:53 
    Pause  tahoe.2  BG_PLATE   erco     %10    %0    0   00:05:53 Job paused.
    Wait   tahoe.3  COMP       erco      %0    %0    0   00:05:53 Wait for other jobs
    
    % rush -jobremarks "jason: please pickup" tahoe.1 tahoe.2 tahoe.3
    tahoe.1: jobremarks is now 'jason: please pickup'
    tahoe.2: jobremarks is now 'jason: please pickup'
    tahoe.3: jobremarks is now 'jason: please pickup'
    
    % rush -lj
    STATUS JOBID    TITLE      OWNER  %DONE %FAIL BUSY    ELAPSED REMARKS
    ------ -------- ---------- ------ ----- ----- ---- ---------- --------
    Run    tahoe.1  FG_PLATE   erco      %0    %0    8   00:05:53 [jason: please pickup]
    Pause  tahoe.2  BG_PLATE   erco     %10    %0    0   00:05:53 [jason: please pickup] Job paused.
    Wait   tahoe.3  COMP       erco      %0    %0    0   00:05:53 [jason: please pickup] Wait for other jobs
    
    

See Also

    jobremarks -- Submit command to set job's remarks
    rush -lj -- Command line flag to list jobs, which includes 'job remarks'

   rush -jobrestartcommand  
rush [jobid] -jobrestartcommand 'command [args..]'
rush [jobid] -jobrestartcommand < file
rush [jobid] -jobrestartcommand '-'

Set the command executed when job restarts running.

See JobRestartCommand submit command for more info.

This allows a command to be executed if a job has completed, and someone restarts it by requeueing one or more frames.

If 'cmd [args]' are not supplied, the command will be read from stdin. This is useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into one single line.

Setting the jobrestartcommand to '-' will disable it. Examples:

    'rush -jobrestartcommand' Examples
    rush -jobrestartcommand 'perl //server/share/rush/restart.pl'   Set jobrestartcommand to run a perl script
    rush -jobrestartcommand < foo.txt Set jobrestartcommand to contents of foo.txt  
    rush -jobrestartcommand - disable the jobrestartcommand (Default)

See Also

   rush -jobrestartcommandflags  
rush -jobrestartcommandflags option[,option] [jobid..]

Set Job Restart Command Flags.

(New in 103.00)
Sets optional flags that affect how the jobrestartcommand is executed.

The specified flags are logically 'OR'ed to augment existing flags. So if the current flags are 'quote,escape', and you then run 'rush -jobrestartcommandflags useshell,quote', the result will leave 'quote,escape,useshell' enabled.

If you want to set ONLY the flags you specify, precede your list of flags with 'off' to zero out any existing flags first. Then your flag specifications will be appended to that. (Note: flags are parsed left-to-right, so you want 'off' to be first in the list). So if the current flags are 'quote,escape', and you run 'rush -jobrestartcommandflags off,quote,useshell', the result will leave only 'quote,useshell' enabled.

The following options are supported, which can be separated by commas to specify several at a time:

    -jobrestartcommandflags Options
    quote Honor double quotes in the command.
    This can be used to protect spaces in command arguments.
    escape Honor escape characters (\) in the command.
    Use this to escape certain characters (like '$' and '"').
    Use this to protect quotes-within-quotes (if 'quote' is specified)
    and to protect '$' (if 'expandvars' is specified)
    expandvars Expand environment variables in the command.
    before it's executed. Variables can be specified in the format ${VARNAME}
    across both platforms.
    useshell Run the command in a native shell.
    Under windows this means a DOS "cmd" shell.
    Under unix this means a Bourne /bin/sh.
    off Disable all command flags.
    Use this at the beginning of your list of flags

   rush -jobstartcommand  
rush [jobid] -jobstartcommand 'command [args..]'
rush [jobid] -jobstartcommand < file
rush [jobid] -jobstartcommand '-'

Set the command executed when job starts running.

See JobStartCommand submit command for more info.

A job might not start running right away, such as if the job has a 'waitfor' command defined.

If 'cmd [args]' are not supplied, the command will be read from stdin. This is useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into one single line.

Setting the jobstartcommand to '-' will disable it. Examples:

    'rush -jobstartcommand' Examples
    rush -jobstartcommand 'perl //server/share/rush/start.pl'   Set job start command to run a perl script
    rush -jobstartcommand < foo.txt Set job start command to contents of foo.txt  
    rush -jobstartcommand - disable the start command (Default)

See Also

    JobStartCommand -- Submit command to set command to run when job starts running
    rush -jobstartcommandflags -- Sets flags that affect how the jobstartcommand is executed
    WaitFor -- Submit command to make a job wait for a specific time or for other jobs

   rush -jobstartcommandflags  
rush -jobstartcommandflags option[,option] [jobid..]

Set Job Start Command Flags.

(New in 103.00)
Sets optional flags that affect how the JobStartCommand is executed.

The specified flags are logically 'OR'ed to augment existing flags. So if the current flags are 'quote,escape', and you then run 'rush -jobstartcommandflags useshell,quote', the result will leave 'quote,escape,useshell' enabled.

If you want to set ONLY the flags you specify, precede your list of flags with 'off' to zero out any existing flags first. Then your flag specifications will be appended to that. (Note: flags are parsed left-to-right, so you want 'off' to be first in the list). So if the current flags are 'quote,escape', and you run 'rush -jobstartcommandflags off,quote,useshell', the result will leave only 'quote,useshell' enabled.

The following options are supported, which can be separated by commas to specify several at a time:

    -jobstartcommandflags Options
    quote Honor double quotes in the command.
    This can be used to protect spaces in command arguments.
    escape Honor escape characters (\) in the command.
    Use this to escape certain characters (like '$' and '"').
    Use this to protect quotes-within-quotes (if 'quote' is specified)
    and to protect '$' (if 'expandvars' is specified)
    expandvars Expand environment variables in the command.
    before it's executed. Variables can be specified in the format ${VARNAME}
    across both platforms.
    useshell Run the command in a native shell.
    Under windows this means a DOS "cmd" shell.
    Under unix this means a Bourne /bin/sh.
    off Disable all command flags.
    Use this at the beginning of your list of flags

   rush -jobdonecommand  
rush [jobid] -jobdonecommand 'command [args..]'
rush [jobid] -jobdonecommand < file
rush [jobid] -jobdonecommand '-'
rush [jobid] -jobdonecommand '[options] command [args]'

Set the command executed when job done running.

See JobDoneCommand submit command for more info.

If 'command [args..]' are not supplied, the command will be read from stdin; useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into a single line.

Setting the jobdonecommand to '-' will disable it. Examples:

    'rush -jobdonecommand' Examples
    rush -jobdonecommand 'perl //server/share/rush/done.pl' Set job done command to run a perl script
    rush -jobdonecommand < foo.txt Set job done command to contents of foo.txt  
    rush -jobdonecommand - Disable the job done command (Default)

By default, the jobdonecommand is executed whenever the last running frame finishes. This behavior can be redefined by specifying [options] before the 'command [args..]':

    -jobdonecommand Options
     -nolog  Disables creation of jobdonecommand.log in the log directory.

Here are some examples with [options] specified:

    Example JobDoneCommand With Options
    rush -jobdonecommand '-nolog perl //path/mydonecommand.pl' Disables the logging to 'jobdonecommand.log',
    so cleanup avoids Windows 'file in use' errors.

See Also

   rush -jobdonecommandflags  
rush -jobdonecommandflags option[,option] [jobid..]

Set Job Done Command Flags.

(New in 103.00)
Sets optional flags that affect how the JobDoneCommand is executed.

The specified flags are logically 'OR'ed to augment existing flags. So if the current flags are 'quote,escape', and you then run 'rush -jobdonecommandflags useshell,quote', the result will leave 'quote,escape,useshell' enabled.

If you want to set ONLY the flags you specify, precede your list of flags with 'off' to zero out any existing flags first. Then your flag specifications will be appended to that. (Note: flags are parsed left-to-right, so you want 'off' to be first in the list). So if the current flags are 'quote,escape', and you run 'rush -jobdonecommandflags off,quote,useshell', the result will leave only 'quote,useshell' enabled.

This command has some other special flags the other commands do not support that are mutually exclusive: done,donefail,fail,always. (see table below) Only one of these flags can be on at a time, and 'always' is the default.

The following options are supported, which can be separated by commas to specify several at a time:

    -jobdonecommandflags Options
    quote Honor double quotes in the command.
    This can be used to protect spaces in command arguments.
    escape Honor escape characters (\) in the command.
    Use this to escape certain characters (like '$' and '"').
    Use this to protect quotes-within-quotes (if 'quote' is specified)
    and to protect '$' (if 'expandvars' is specified)
    expandvars Expand environment variables in the command.
    before it's executed. Variables can be specified in the format ${VARNAME}
    across both platforms.
    useshell Run the command in a native shell.
    Under windows this means a DOS "cmd" shell.
    Under unix this means a Bourne /bin/sh.
    off Disable all command flags.
    Use this at the beginning of your list of flags

These additional flags are mutually exclusive:

    JobDoneCommandFlags that control exexcution based on job state
    These flags are mutually exclusive.
    done Runs jobdonecommand only if all frames are "Done"
    fail Runs jobdonecommand only if *any* frames are "Fail" and rest are "Done"
    donefail Runs jobdonecommand only if all frames are either "Done" "Fail"
    always Runs jobdonecommand when job is done, regardless of the state of the frames (Default)
    (If job is dumped while frames are still 'Que', command will still run)

   rush -jobdumpcommand  
rush [jobid] -jobdumpcommand '[-nolog] command [args..]'
rush [jobid] -jobdumpcommand < file
rush [jobid] -jobdumpcommand '-'

Set the command executed when job dumps.

See JobDumpCommand submit command for more info.

If 'cmd [args]' are not supplied, the command will be read from stdin. This is useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into one single line.

Specifying the '-nolog' option in front of 'command' will disable the creation of the 'jobdumpcommand.log' file in the LogDir. This is useful to prevent 'file in use' errors on Windows if you want to remove the entire logdir as part of a cleanup operation.

Setting the jobdumpcommand to '-' will disable it. Examples:

    'rush -jobdumpcommand' Examples
    rush -jobdumpcommand 'perl //server/share/rush/dump.pl' Set job dump command
    to run a perl script
    rush -jobdumpcommand '-nolog perl //server/share/rush/dump.pl' Same as above, disables creation of
    'jobdumpcommand.log'
    rush -jobdumpcommand < foo.txt Set job dump command to
    contents of foo.txt
    rush -jobdumpcommand - Disable the dump command (Default)  

See Also

   rush -jobdumpcommandflags  
rush -jobdumpcommandflags option[,option] [jobid..]

Set Job Dump Command Flags.

(New in 103.00)
Sets optional flags that affect how the JobDumpCommand is executed.

The specified flags are logically 'OR'ed to augment existing flags. So if the current flags are 'quote,escape', and you then run 'rush -jobdumpcommandflags useshell,quote', the result will leave 'quote,escape,useshell' enabled.

If you want to set ONLY the flags you specify, precede your list of flags with 'off' to zero out any existing flags first. Then your flag specifications will be appended to that. (Note: flags are parsed left-to-right, so you want 'off' to be first in the list). So if the current flags are 'quote,escape', and you run 'rush -jobdumpcommandflags off,quote,useshell', the result will leave only 'quote,useshell' enabled.

The following options are supported, which can be separated by commas to specify several at a time:

    -jobdumpcommandflags Options
    quote Honor double quotes in the command.
    This can be used to protect spaces in command arguments.
    escape Honor escape characters (\) in the command.
    Use this to escape certain characters (like '$' and '"').
    Use this to protect quotes-within-quotes (if 'quote' is specified)
    and to protect '$' (if 'expandvars' is specified)
    expandvars Expand environment variables in the command.
    before it's executed. Variables can be specified in the format ${VARNAME}
    across both platforms.
    useshell Run the command in a native shell.
    Under windows this means a DOS "cmd" shell.
    Under unix this means a Bourne /bin/sh.
    off Disable all command flags.
    Use this at the beginning of your list of flags

   rush -lac  
rush -lac [-s secs] [-r retry] [-c count] [-foe]

List All Cpus.

Lists the status of all rush cpus on the network. Report should come up quickly, even on large networks.

This report is useful for seeing which cpus are running whose jobs.

[-r secs] is an optional argument specifying the number of times to retry contacting the remote machine, before giving up. Default is 4.

[-s secs] is an optional argument specifying the number of seconds between retries for remotes that don't respond right away. Default is 2.

[-c count] is an optional argument to specify the number of times to automatically regenerate the report (for continuous updates). A value of 0 indicates infinite updating. Default is 1. Example report:

[-foe] "Fail On Error". Causes the command to stop on any hostname lookup errors (such as temporary DNS or LDAP issues), instead of just printing warnings and continuing to run to handle any other hostnames specified. (default: off) *This flag new in 103.07*

    'rush -lac' Report
    % rush -lac
    HOST       OWNER    JOBID          TITLE                     FRM  PRI   PID        ELAPSED
    tahoe      -        -              -                         -    -     -           Online
    tahoe      -        -              -                         -    -     -           Online
    superior   erco     vaio.50        FOREST_RENDER             0034 100   4026      00:10:01
    superior   erco     vaio.50        FOREST_RENDER             0035 100   4027      00:10:02
    superior   tamu     ibm4.208       AUTO_CAD                  0101 300k  2031      02:43:31
    superior   tamu     ibm4.208       AUTO_CAD                  0102 300k  2035      02:43:31
    erie       erco     vaio.50        FOREST_RENDER             0041 900k  14560     00:10:15
    erie       erco     vaio.50        FOREST_RENDER             0042 900k  14561     00:10:15
    michigan   -        -              -                         -    -     -          Offline
    michigan   -        -              -                         -    -     -          Offline
    michigan   -        -              -                         -    -     -          Offline
    michigan   -        -              -                         -    -     -          Offline
    *** NO RESPONSE FROM:
    *** ontario huron

Note that machines that did not respond are listed in the summary at the bottom of the report; all lines of these summaries are prefixed by '***'.

All Cpus Report Format

   rush -lacf  
rush -lacf [-s secs] [-r retry] [-c count]

List All Cpus Full.

Lists the full info of all rush cpus on the network. This report may take a while to run.

This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.

[-r secs] is an optional argument specifying the number of times to retry contacting the remote machine, before giving up. Default is 4.

[-s secs] is an optional argument specifying the number of seconds between retries for remotes that don't respond right away. Default is 2.

[-c count] is an optional argument to specify the number of times to automatically regenerate the report (for continuous updates). A value of 0 indicates infinite updating. Default is 1.

   rush -lah  
rush -lah [-t timeout] [hostname..|+hostgroup..]

List All Hosts.

Lists the Ram/Criteria configuration for all hosts in the render queue, which is loaded directly from the $RUSH_DIR/etc/hosts file. Example:

    'rush -lah' Report
      % rush -lah
      IP               Hostname   Ram  Cpus MinPri Criteria
      10.100.100.1     tahoe      512  2    0      +any,irix
      10.100.100.2     superior   512  2    0      +any,irix,+dante
      10.100.100.3     erie       512  2    0      +any,irix,+dante
      10.100.100.4     ontario    512  2    0      +any,irix,+dante
      10.100.100.5     vaio       128  1    0      +any,linux,intel
      10.100.100.6     rotwang    128  1    100    +any,linux,intel    

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

If hostname is specified, i.e., 'rush -lah hostname', the output comes from the cache of the daemon running on the specified host. This is useful in determining hostname caching problems.

If a +hostgroup is specified, i.e. 'rush -lah +farm', the output comes from the cache of the daemons in that hostgroup.

A daemon's current negative hostname cache will be shown at the bottom of an -lah report, if the 'rush -lah <hostname>' syntax is used. The following example shows 3 negative cache entries ("geneeva", "thoe", and "foobar") as being cached by the daemon on host 'tahoe':

    Negative Cache
      % rush -lah tahoe
     Cache on tahoe:
     IP               Hostname        Ram  Cpus MinPri Criteria
     192.168.0.10     geneva          1024 2    0      +any,+w2k,+work
     192.168.0.3      rotwang         100  2    0      +any,+erco,+linux,linux,linux6.0,intel,dante
     192.168.0.9      ontario         100  2    0      +any,+erco,+linux,linux,linux6.0,intel,dante
     192.168.0.14     meade           512  2    0      +any,+erco,+linux,linux
     192.168.0.7      tahoe           512  2    0      +any,+erco,+linux,linux
     *** Negative Cache: geneeva     (expires 00:12:38 secs from now)
     *** Negative Cache: thoe        (expires 00:14:53 secs from now)
     *** Negative Cache: foobar      (expires 00:14:53 secs from now)

When too many hosts have been added to the rush/etc/hosts file than one is licensed for, rush simply ignores the extra hosts, and the 'rush -lah' report will warn about the extras in its report:

    Warnings for Unlicensed Hosts
    
    % rush -lah
    IP               Hostname        Ram  Cpus MinPri Criteria
    192.168.0.10     geneva          1024 2    0      +any,+w2k,+work
    192.168.0.3      rotwang         100  2    0      +any,+erco,+linux,intel,dante
    192.168.0.9      ontario         100  2    0      +any,+erco,+linux,intel,dante
    [..]
    192.168.5.132    r0129           100  1    0      +any,+sgi
    192.168.5.133    r0130           100  1    0      +any,+sgi
    *** ERROR: TOO MANY HOSTS IN RUSH HOSTS LIST [24 found, 20 max]
    *** ERROR: The following hosts are being ignored by rush..
    ***     192.168.5.134    r0131           100  1    0      +any,+sgi
    ***     192.168.5.135    r0132           100  1    0      +any,+sgi
    ***     192.168.5.136    r0133           100  1    0      +any,+sgi
    ***     192.168.5.137    r0134           100  1    0      +any,+sgi

   rush -lahf  
rush -lahf [-t timeout] [hostname..]

List All Hosts Full.

Full hosts report for all hosts on local [remote] host.

This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

   rush -laj  
rush -laj [-fmt str|env|defaults|help] [-x] [-s secs] [-r retry] [-c count] [-r retries] [-foe] [hostname..|+hostgroup..]

List All Jobs.

Queries all hosts on the network for what jobs are running. (All hosts being all hosts in the "+any" hostgroup)

[-x] is an optional 'extended report' flag that includes job progress bars; 'blue' indicates frames rendering, 'green' indicates frames done successfully, 'red' indicates failed frames. These colors will appear in irush reports and terminals that support "ANSI" colors or XTERM colors. For examples, see 'rush -lj'

[-fmt str] is an optional flag to control the formatting of the report. You can choose which fields to show from a table of possible field names, allowing you to completely customize the report.

If '-fmt env' is used, then the environment variable RUSH_LAJ_FMT will be read for the format info. Example: 'export RUSH_LAJ_FMT="Jobid Title"; rush -laj -fmt env'

    rush -laj options
    Option Description
    -x Extended report that includes job progress bars.
    -fmt str Sets a custom report formatting string.
    Same format as the 'rush -lj' report; see that for more info.

    Example: rush -laj -fmt "Jobid:-10 Status:-8 Title:-20 Progress:-20"
    The special flag "-fmt env" lets you use define the field names in the RUSH_LAJF_FMT
    environment variable, to avoid having to specify the flags on the command line each time.
    Example: 'export RUSH_LAJ_FMT="Jobid Title"; rush -laj -fmt env'
    The special flag "-fmt help" lists the format syntax and field names.
    The special flag "-fmt defaults" forces the software's built in format defaults.
    -c cnt Sets the number of times to continuously update the report.
    A value of zero ("-c 0") creates a continuous update.
    Default is 1, as defined by the rush.conf rush.laj_count command.
    -s secs Sets the floating point number of seconds between polls.
    Use with -c to poll automatically at a specified interval.
    Default is defined by the rush.conf rush.laj_secs command.
    -r secs Sets the number of times to retry contacting the remote machine
    before giving up.
    Default is defined by the rush.conf rush.laj_retry command.
    -ss secs Sets the floating point number of seconds between udp acknowledges.
    You probably don't want to mess with this unless you're getting
    weird communications behavior. Default is 0.05.
    -foe "Fail On Error". Causes the command to stop on hostname errors
    instead of continue to run for any other hostnames that resolve
    correctly. (default: off) *This flag new in 103.07*

Optional hostnames or +hostgroups can be specified to list jobs served by only those hosts. If unspecified, all hosts in the +any group is used.

    'rush -laj' Report
    % rush -laj
    STATUS JOBID          TITLE                     OWNER    %DONE %FAIL BUSY    ELAPSED
    Run    erie.167       100/340/master            tamu     %18   %0    8       01:15:28
    Run    erie.169       100/410/master            tamu     %75   %0    20      01:11:21
    Run    erie.170       100/390/master            tamu     %10   %34   4       01:10:44
    Run    erie.171       100/360/master            tamu     %10   %34   4       01:02:43
    Run    huron.454      W:090/440/all             krement  %100  %0    0       26:25:31
    Run    huron.457      W:090/425/master          krement  %100  %0    0       26:11:59
    Run    huron.476      W:090/425/master          krement  %100  %0    0       23:40:37
    Run    huron.519      Q:090/460/master          krement  %5    %0    8       00:21:27
    Pause  tahoe.76       230/250/layout_lock       erco     %2    %0    0       01:52:08
    Run    allek.790      qr2                       tje      %80   %1    4       06:58:20
    Run    allek.794      180/260/DAY               tje      %80   %1    2       06:38:48
    Run    allek.796      180/250/DAY               tje      %0    %0    0       06:34:21
    Run    allek.801      qr2                       tje      %0    %0    0       05:27:40
    Run    vaio.663       230/tdcheck/roomBC        woz      %80   %20   0       00:44:53
    
    % rush -laj -fmt "Jobid:-12 Title"
    Jobid        Title
    erie.167     100/340/master
    erie.169     100/410/master
    erie.170     100/390/master
    erie.171     100/360/master
    huron.454    W:090/440/all
    huron.457    W:090/425/master
    huron.476    W:090/425/master
    huron.519    Q:090/460/master
    tahoe.76     230/250/layout_lock
    allek.790    qr2
    allek.794    180/260/DAY
    allek.796    180/250/DAY
    allek.801    qr2
    vaio.663     230/tdcheck/roomBC
    

All Jobs Report Format

    The administrator has a certain amount of control over the output format of the 'All Jobs' report via the rush.conf file, specifically with the rush.lj.fmt and rush.lj_x.fmt commands.

Caveats

    Previously, this report used UDP exclusively. As of 102.42a9d2 / 103.03 and up, this report uses a combo of UDP and TCP; UDP is used to quickly identify machines that have jobs, then TCP is used to pull the report from those machines that responded over UDP that are managing any jobs. This is because often reports were so large, they were too large to send efficiently over UDP.

   rush -lajf  
rush -lajf [-s secs] [-c count|0] [-r retrycount] [-t timeout] [-fmt str|env|defaults|help] [hosts|+hostgroups..]

List All Jobs Full.

Full job report for all jobs on the network. This report may take a while to run.

This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.

Optional hostnames or +hostgroups can be specified to list jobs served by only those hosts. If unspecified, all hosts in the +any group is used.

[-fmt str] is an optional flag to limit the report to only show specific fields. (If unspecified, ALL fields are shown.) You can choose any of the field names from the 'rush -lajf' report separated by spaces, e.g. -fmt "Jobid Title Status WaitFor" which will limit the report to only show those four fields. This is similar to 'grep'ing for the fields you want, but more efficient because it avoids unnecessary network traffic for the unneeded data, making for a faster report.

    rush -lajf options
    Option Description
    -fmt str Sets a custom report formatting string.
    Same format as the 'rush -ljf' report; see that for more info.

    Example: rush -lajf -fmt "Jobid Title Status"
    The special flag "-fmt env" lets you use define the field names in the RUSH_LAJF_FMT
    environment variable, to avoid having to specify the flags on the command line each time.
    Example: 'export RUSH_LAJF_FMT="Jobid Title"; rush -lajf -fmt env'
    The special flag "-fmt defaults" forces the software's built in format defaults (all fields shown).
    -c cnt Sets the number of times to continuously update the report.
    Default is 1. -c 0 creates a continuous update.
    -s secs Sets the floating point number of seconds between polls.
    Use with -c to poll automatically at a specified interval.
    Default is 3.0.
    -r secs Sets the number of times to retry contacting the remote machine
    before giving up. Default is 2.
    -t timeout Sets the number of seconds to wait for response from server
    before giving up. Default is 40.

    'rush -lajf' Report
    % rush -lajf
           Jobid: zoar.1
           State: Done
        ActState: Done
        RebootId: 2076
           Title: TEST_0001                                                     
     DefPriority: 1
    [..snip..]
            Cpus: +any=1@100
            Cpus: +any=1@200
        Criteria: 
    
           Jobid: zoar.2
           State: Done
        ActState: Done
        RebootId: 28
           Title: TEST_0002
     DefPriority: 1
    [..snip..]
            Cpus: +any=10@100
        Criteria: 
    
    % rush -lajf -fmt "Jobid Title Cpus"
           Jobid: zoar.1
           Title: TEST_0001
            Cpus: +any=1@100
            Cpus: +any=1@200
    
           Jobid: zoar.2
           Title: TEST_0002
            Cpus: +any=10@100
    
    

Caveats

    Up until Rush 103.03, this report used UDP exclusively. As of 103.03 and up, this report uses a combo of UDP and TCP; UDP is used to quickly identify machines that have jobs, then TCP is used to pull the report from those machines that responded over UDP that are managing any jobs. This is because often reports were so large, they were too large to send efficiently over UDP.

   rush -lc  
rush -lc [-t timeout] [hosts..|+hostgroups|jobids..]

List Cpus.

Lists the cpus the job requested to use, and the status of those requests, i.e. busy, or if they were turned down for some reason (shown in the NOTES field).

  • If one or more space-delimited [hosts..] are specified, the cpus of all jobs being managed by those hosts are shown.

  • If one or more space-delimited [jobids..] are specified, only the cpus requested by those jobs are listed.

  • If neither [hosts..] or [jobids..] are specified, the contents of the RUSH_JOBID environment variable is checked for jobids, and the cpu requests of only those jobs are reported.

  • If none of the above are available, an error is reported

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

Example:

    'rush -lc' Report
      % rush -lc tahoe.1        # List Cpus for job tahoe.1
      CPUSPEC[HOST]           STATE       FRAME  PID     JOBTID  PRI  ELAPSED  JOBID    NOTES
      huron=4@800k            JobPass     -      -       178     1    00:00:00 tahoe.1  This is a 'neverhost'  
      huron=4@800k            JobPass     -      -       179     1    00:00:00 tahoe.1  This is a 'neverhost'
      huron=4@800k            JobPass     -      -       180     1    00:00:00 tahoe.1  This is a 'neverhost'
      huron=4@800k            JobPass     -      -       181     1    00:00:00 tahoe.1  This is a 'neverhost'
      +farm=10@100[howland]   Busy        0080   14733   165     1    00:00:01 tahoe.1   
      +farm=10@100[howland]   Busy        0078   14731   166     1    00:00:01 tahoe.1   
      +farm=10@100[howland]   Busy        0079   14732   167     1    00:00:01 tahoe.1   
      +farm=10@100[toronto]   Idle/Nak    -      -       171     1    00:00:00 tahoe.1   -
      +farm=10@100[toronto]   Idle/Nak    -      -       172     1    00:00:00 tahoe.1   -
      +farm=10@100[toronto]   Idle/Nak    -      -       173     1    00:00:00 tahoe.1   -
      +farm=10@100[vaio]      Busy        0074   4107    174     1    00:00:03 tahoe.1   
      +farm=10@100[vaio]      Busy        0071   4100    175     1    00:00:03 tahoe.1   
      +farm=10@100[vaio]      Busy        0072   4101    176     1    00:00:03 tahoe.1   
      +farm=10@100[vaio]      Busy        0073   4106    177     1    00:00:03 tahoe.1
    
    % rush -lc tahoe        # List Cpus for /all/ jobs managed by host 'tahoe'
      CPUSPEC[HOST]           STATE       FRAME  PID     JOBTID  PRI  ELAPSED  JOBID    NOTES                  _
      huron=4@800k            JobPass     -      -       178     1    00:00:04 tahoe.1  This is a 'neverhost'   |
      huron=4@800k            JobPass     -      -       179     1    00:00:04 tahoe.1  This is a 'neverhost'   |
      huron=4@800k            JobPass     -      -       180     1    00:00:04 tahoe.1  This is a 'neverhost'   |
      huron=4@800k            JobPass     -      -       181     1    00:00:04 tahoe.1  This is a 'neverhost'   |
      +farm=10@100[howland]   Busy        0080   14733   165     1    00:00:05 tahoe.1                          |
      +farm=10@100[howland]   Busy        0078   14731   166     1    00:00:05 tahoe.1                          |
      +farm=10@100[howland]   Busy        0079   14732   167     1    00:00:05 tahoe.1                          |-- job tahoe.1   
      +farm=10@100[toronto]   Idle/Nak    -      -       171     1    00:00:04 tahoe.1   -                      |
      +farm=10@100[toronto]   Idle/Nak    -      -       172     1    00:00:04 tahoe.1   -                      |
      +farm=10@100[toronto]   Idle/Nak    -      -       173     1    00:00:04 tahoe.1   -                      |
      +farm=10@100[vaio]      Busy        0074   4107    174     1    00:00:07 tahoe.1                          |
      +farm=10@100[vaio]      Busy        0071   4100    175     1    00:00:07 tahoe.1                          |
      +farm=10@100[vaio]      Busy        0072   4101    176     1    00:00:07 tahoe.1                          |
      +farm=10@100[vaio]      Busy        0073   4106    177     1    00:00:07 tahoe.1                         _|
      +farm=10@100.1[howland] Idle        -      -       195     1    00:00:00 tahoe.2                         _ 
      +farm=10@100.1[toronto] Idle        -      -       192     1    00:00:00 tahoe.2                          |-- job tahoe.2
      +farm=10@100.1[vaio]    Idle        -      -       194     1    00:00:00 tahoe.2                         _|
    
    % rush -lc +work -t 3      # List Cpus for /all/ jobs managed by hosts in the +work group
      [..large report..]
    
            

    List Cpus Headings
    Heading Description
    CPUSPEC[HOST] The cpu specification supplied by the user during submit.
    If hostgroups are specified (i.e. +any), the
    group is broken out into individual cpus, with the hostname
    of each processor shown in [brackets].
    STATE The state the cpu's 'task reservation' is in. In the following,
    'cpuserver' means the remote host serving the cpu.
    
        Idle        --  task idle, waiting for cpuserver
        Idle/Nak    --  job is still trying to reach the cpuserver
        Avail       --  cpuserver wants us to run a frame
        Start       --  found frame to start, reply to cpuserver
        Run         --  cpuserver telling us frame started ok
        Busy        --  frame is busy, reply to cpuserver
        CpuPass2    --  replies to cpuserver, remains in state
        JobPass     --  job told cpuserver we can't use its cpu (see NOTES for why)
        Unpass      --  remote reverting to idle state (cosmetic)

    FRAME The frame that is being rendered.
    PID The process id on HOST for the running render script. This is also
    the 'process group leader'.
    JOBTID The 'job task id' for this processor. Only useful as an
    argument commands like 'rush -rc'.
    PRI The priority being requested
    ELAPSED The wall clock elapsed time in hours:minutes:seconds for the
    rendering frame.
    JOBID The jobid of the job (useful when more than one jobid is being reported)
    NOTES Various notes regarding the state of the job. These notes can
    sometimes be stale; the 'STATE' information is a more accurate
    representation of the state of the cpu.

Some examples of the 'NOTES' column in the 'rush -lc' report:

    List Cpus Notes Field
    Message Description
      Cpu max (#/# busy)   Job could not run on available cpu because the job's
    cpu cap (left value) has been reached; there are already
    cpus busy (right value). Examples:

      "Cpu max (2/2 busy)" -- job has 2 cpus busy,
      cap was set to 2. Typical case where the job is already
      rendering on as many cpus as it requested.

      "Cpu max (4/2 busy)" -- job has 4 cpus busy,
      and cap was set to 2. (This can happen if the cap was
      recently lowered from 4 to 2, while 4 were already busy.

      Below MinPri (#<#)   Machine became available, but this job was turned away
    because the cpu's priority was less-than the minimum
    priority (MinPri) set by the sysadmin for this host in the
    rush/etc/hosts file.
      Job state is 'pause'  
      Job state is 'wait'  
    ..etc..
    Job can't run because the job is in the state shown.
    For instance, 'pause' indicates the job was paused
    at the time the cpu became available to run this job,
    so it was turned down.
      No more frames   Job can't run because all the frames in the job
    have already been rendered at the time the cpu
    became available.
      Maxcpus reached (#/#)   Job can't use the available cpu because the job
    already has enough cpus busy running to fulfill the
    'maxcpus' set for the job.

    The two values shown (#/#): the 'maxcpus' for the job
    followed by the number of cpus the job had busy
    at the time the cpu became available.
      This is a 'nevercpu'   Job can't use the available cpu because the job
    has this machine blacklisted in its 'nevercpus' (rush -an).
      Failed 'criteria' check   Job can't use the available cpu because the job's
    configured 'criteria' didn't meet the machine's criteria
    configured in the rush/etc/hosts file.

   rush -lcf  
rush -lcf [-t timeout] [jobid..]

List Cpus Full.

A 'full' report for all of the current [specific] job's cpus.

This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

   rush -lf  
rush -lf [-t timeout] [jobid..]

List Frames.

Lists a job's frames to see what state they're in. The default is to show all the job's frames, but optional frame range(s) can be specified to limit the report. If a jobid is not specified, the RUSH_JOBID variable will be used. If multiple jobids are specified, the frames in each are listed.

    'rush -lf' Report
    
        $ rush -lf              # List Frames for the current job   
        STATUS FRAME TRY HOSTNAME       PID     JOBID    START          ELAPSED  NOTES  
        Run    5000  1   vaio           18092   tahoe.34 02/26,02:15:40 00:00:21 -
        Run    5001  1   vaio           18093   tahoe.34 02/26,02:15:40 00:00:21 -
        Run    5002  1   vaio           18100   tahoe.34 02/26,02:15:50 00:00:11 -
        Que    5003  0   -              0       tahoe.34 00/00,00:00:00 00:00:00 -
        Que    5004  0   -              0       tahoe.34 00/00,00:00:00 00:00:00 -
        Que    5005  0   -              0       tahoe.34 00/00,00:00:00 00:00:00 -
    
        $ rush -lf 5003-5005     # show only frames 5003 thru 5005      
        STATUS FRAME TRY HOSTNAME       PID     JOBID    START          ELAPSED  NOTES  
        Que    5003  0   -              0       tahoe.34 00/00,00:00:00 00:00:00 -
        Que    5004  0   -              0       tahoe.34 00/00,00:00:00 00:00:00 -
        Que    5005  0   -              0       tahoe.34 00/00,00:00:00 00:00:00 -
            

The -x flag includes optional 'extended' information at the bottom of the report; if the job has a 'jobstartcommand', 'jobrestartcommand', 'jobdonecommand', or 'jobdumpcommand' command configured, and the logs exist, these are shown as separate entries at the bottom of the -lf report. (IRUSH makes use of this, to allow the user to double-click on these lines in the 'Frames' report to view the contents of those log files.) Example:

    'rush -lf -x' Report
        $ rush -lf -x           # include extended info   
        STATUS FRAME TRY HOSTNAME       PID     JOBID    START          ELAPSED  NOTES  
        Done   5000  1   vaio           18092   tahoe.34 02/26,02:15:40 00:00:21 -
        Done   5001  1   vaio           18093   tahoe.34 02/26,02:15:40 00:00:21 -
        Done   5002  1   vaio           18100   tahoe.34 02/26,02:15:50 00:00:11 -
        Done   5003  1   tahoe          20478   tahoe.34 02/26,02:15:58 00:00:18 -
        Done   5004  1   tahoe          20483   tahoe.34 02/26,02:15:58 00:00:18 -
        Done   5005  1   superior       1978    tahoe.34 02/26,02:15:58 00:00:18 -
        *** LOGFILE: tahoe.34 jobstartcommand.log
        *** LOGFILE: tahoe.34 jobdonecommand.log
      

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

    List Frames Examples
    Heading Description
    Stat The status of the frame. Can be any of:

    • Que - The frame is queued, ready to be run by an available cpu
    • Run - The frame is busy running
    • Done - The frame completed successfully
    • Fail - The frame failed to run successfully (exit code was 1)
    • Hold - The frame will not run until changed manually to Que

    The following are transitional states for frames, which you should only
    see for a short time:

    • Strt - Frame just started to run, no process id yet
    • Die - The frame is being killed, but hasn't received
      an exit code back yet

    Frame The frame number, padded 4 digits.
    (Or as otherwise specified by 'framepad' in the rush.conf file.)
    Try The number of times this frame has been tried. Includes
    requeues due to exit(2), but does not include priority bumps.
    Hostname The name of the host that is running (or that ran) the frame.
    PID The process id of the render script. This is also the process group
    leader for the frame's process hierarchy. (In windows, the top level
    'Job Object')
    Start The date/time the render began running.
    Elapsed The wall clock elapsed time in hours:minutes:seconds.
    Notes Free form notes text, set by the user's invocation of rush -notes,
    e.g. 'rush -notes 0012:"License Error"'

Frames Report Format

   rush -lff  
rush -lff [-t timeout] [jobid..]

List Frames Full.

A 'full' report of all frames' state information.

This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

   rush -lfi  
rush -lfi [-t timeout] [jobid..]

List Frame Information.

Lists Frame info for current [specific] job, including average render times, ETAs, percent Busy/Done/Fail, etc. Example:

    'rush -lfi' Report
      % rush -lfi vaio.139               # List Frame Info for job 'vaio.139'      
                                    Average    Total
      Jobid        State Total Perc Elapsed    Elapsed    Average ETA
      ------------ ----- ----- ---- ---------- ---------- ------------------------
      rotwang.3    Que   2     %4   -          -          -
      rotwang.3    Run   2     %4   -          -          -
      rotwang.3    Done  46    %92  00:00:11   00:08:36   Thu Feb 12 17:47:16 2004  
      rotwang.3    Fail  0     %0   -          -          -
      rotwang.3    Hold  0     %0   -          -          -
      rotwang.3    ---
      rotwang.3    Render Elapsed: 00:02:25
      rotwang.3      Render Start: 02/12,17:44:45
      rotwang.3        Render End: 02/12,17:47:10

  • Jobid
    The jobid for this job. (If multiple jobs are being listed, this column shows which info goes with which job)

  • State
    The various states the frames can be in, and to the right, the accumulated data for all the frames in each of those states.

  • Total
    The total number of frames in each 'State'.

  • Perc
    The percentage of frames in each 'State'.

  • Average Elapsed
    The average render times. This is the sum of all the Done frames' render times divided by the total Done frames. (Currently only 'Done' state frames are shown)

  • Total Elapsed
    The total elapsed time for the frames; a sum of all the 'Elapsed' times in the 'Frames' report. (Currently only 'Done' state frames are shown)

  • Average ETA
    This is the estimated time of completion for the job, based on the average render time for frames rendered successfully so far vs. the wall clock time spent running the job. e.g.:

      oldest_start   = "The oldest start time for all Done frames"
      recent_end     = "Most recent end time for all Done frames"
      time_spent     = ( recent_end - oldest_start )
      time_per_frame = time_spent / total_done_frames
      time_to_go     = time_per_frame * ( total_que + total_hold )
              
    This time will vary widely, since completion time is dependent on the cpu availability, which is always changing.

  • Render Elapsed
    The elapsed time for rendering, ie. how long the TD had to wait for the job to finish rendering. The difference between the 'Render Start' and the 'Render End' times.

  • Render Start
    The 'Start Time' for the first frame.

  • Render End
    The 'End Time' for the last frame that finished rendering.

Warning: the ETA is meant for ballpark estimates only, and is not meant to be taken literally.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

   rush -lhc  
rush -lhc [criteria]

List Host Criteria.
Shows all [specific] host criteria your sysadmin has configured in rush.

If no arguments are specified, all host criteria are listed. Otherwise, arguments can be the criteria names to limit the report to display.

Both criteria and host names are sorted alphabetically.

    'rush -lhc' Reports
      % rush -lhc       # Lists all criteria when no arguments specified
      intel: vaio
             rotwang
    
       irix: erie
             ontario
             superior
    
      linux: rotwang
             tahoe
             vaio
           
      % rush -lhc linux       # Lists only hosts matching 'linux' criteria  
      linux: rotwang
             tahoe
             vaio

   rush -lhg  
rush -lhg [+hostgroup..]

List Host Groups.

Shows all [specific] hostgroups your sysadmin has configured in rush.

If no arguments are specified, all hostgroups are listed. Otherwise, arguments can be the names of hostgroups to limit the report to display.

Both hostgroups and hostnames are sorted alphabetically.

    'rush -lhg' Reports
     % rush -lhg       # Lists all hostgroups when no arguments specified  
       +any: erie
             ontario
             rotwang
             superior
             tahoe
             vaio
    
     +dante: erie
             ontario
             superior
    
     +intel: rotwang
             tahoe
             vaio
    
      +irix: erie
             ontario
             superior
    
     +linux: rotwang
             tahoe
             vaio
            
     % rush -lhg +dante       # Lists only the '+dante' hostgroup
     +dante: erie
             ontario
             superior

   rush -lj  
rush -lj [-fmt str|env|defaults|help] [-x] [-t timeout] [hosts|+hostgroups..] [jobids..]

List Jobs.

Shows list of jobs.

If '-' is specified as an argument, jobids are read from stdin. (Useful if the list of jobids exceeds operating system limits for argument lists and environment variable sizes)

  • If one or more space-delimited [hosts..] are specified, the list of all jobs being managed by each host is shown.

  • If one or more space-delimited [jobids..] are specified, only those jobids are listed.

  • If neither [hosts..] or [jobids..] are specified, the contents of the RUSH_JOBID environment variable is checked for jobids, and all jobs managed by the servers of those jobids will be listed.

  • If none of the above are available, 'rush -lj' defaults to listing all jobs being managed by the local machine.

    rush -lj options
    Option Description
    -x Enable 'extended' fields. A shortcut method for showing some fields
    users like to see besides the default plain report. The -x report
    format is defined by the rush.lj_x.format command.
    -t timeout Controls the number of seconds rush waits to make a connection
    to each remote host before it gives up. The default timeout
    is 40 seconds, eg. '-t 40', the default defined in rush.conf
    by the tcptimeout command.

    Use this if there are many machines on your network that are
    turned off, and you don't want to wait a full 40 seconds for
    each host to respond. (eg. '-t 3' will be much quicker).
    -fmt str Sets a custom report formatting string.
    Same format as the 'rush -ljf' report; see that for more info.

    Example: rush -lajf -fmt "Jobid Title Status"
    The special flag "-fmt env" lets you use define the field names in the RUSH_LAJF_FMT
    environment variable, to avoid having to specify the flags on the command line each time.
    Example: 'export RUSH_LAJF_FMT="Jobid Title"; rush -lajf -fmt env'
    The special flag "-fmt defaults" forces the software's built in format defaults (all fields shown).
    -json Generates reports in JSON format.
    Can be mixed with -fmt to return only specific fields.
    See "JSON FORMAT" section below for more info.
    - Read jobids from stdin.
    This, as opposed to specifying them on the command line.
    Use this if the number of jobids to be specified might be more
    than your shell's command line limit enforces.

    'rush -lj' Reports
      [you@erie] % rush -lj                   # List Jobs on local workstation
      STATUS   JOBID          TITLE                     OWNER    %DONE %FAIL BUSY    ELAPSED REMARKS
      -------- -------------- ------------------------- -------- ----- ----- ---- ---------- --------
      Run      erie.167       100/340/master            tamu       %18    %0    8   01:15:28
      Run      erie.169       100/410/master            tamu       %75    %0   20   01:11:21
      Run      erie.170       100/390/master            tamu       %10   %34    4   01:10:44
      Run      erie.171       100/360/master            tamu       %10   %34    4   01:02:43
    
      [you@erie] % rush -lj tahoe             # List Jobs on remote host tahoe
      STATUS   JOBID          TITLE                     OWNER    %DONE %FAIL BUSY    ELAPSED REMARKS
      -------- -------------- ------------------------- -------- ----- ----- ---- ---------- --------
      Run      tahoe.5        100/540/master            izzie      %25    %0    4   00:30:01
      Run      tahoe.6        100/540/comp              izzie      %25    %0    4   00:30:02
      Run      tahoe.10       100/640/master            izzie      %10    %0    4   00:22:48
      Pause    tahoe.11       100/640/comp              izzie      %12    %0    4   00:22:49 Job paused.  
    
      [you@erie] % rush -lj tahoe.5 tahoe.10  # List Jobs  tahoe.5 and tahoe.10 only
      STATUS   JOBID          TITLE                     OWNER    %DONE %FAIL BUSY    ELAPSED REMARKS
      -------- -------------- ------------------------- -------- ----- ----- ---- ---------- --------
      Run      tahoe.5        100/540/master            izzie      %25    %0    4   00:30:05
      Run      tahoe.10       100/640/master            izzie      %10    %0    4   00:22:52
    

[-x] is an optional 'extended report' flag that includes job progress bars; 'blue' indicates frames rendering, 'green' indicates frames done successfully, 'red' indicates failed frames. These colors will appear in irush reports and terminals that support "ANSI" colors or XTERM colors.

[-fmt str] is an optional flag to control the formatting of the report. You can choose which fields to show from a table of possible field names, allowing you to completely customize the report. eg:

    'rush -lj -fmt' Reports
    % rush -lj -fmt "Jobid:-10 Status:-8 Title:-20 Progress:-20 Priority[Vals]:-5 Owner"
    Jobid      Status   Title                Progress             Pri   Owner
    tahoe.1    Done     SMX:DIRECTOR_V2      ____________________ 100   erco
    tahoe.5    Done     SMX:DIRECTOR_V3      ____________________ 100   root
    tahoe.7    Pause    SMX:HD_FIXUPS        ____________________ 100   root
    
    % export RUSH_LJ_FMT='Jobid:-10 Status:-8 Title"     # Another way to do the same thing..
    % rush -lj -fmt env                                  # '-fmt env' uses the RUSH_LJ_FMT variable's setting
    Jobid      Status   Title
    tahoe.1    Done     SMX:DIRECTOR_V2
    tahoe.5    Done     SMX:DIRECTOR_V3
    tahoe.7    Pause    SMX:HD_FIXUPS
    

A complete list of fieldnames can also be shown from the command line with 'rush -lj -fmt help'. You can also use the RUSH_LJ_FMT environment variable to pre-define a format string, and then use it at any time with just 'rush -lj -fmt env'. Finally, the systems administrator can change the default format for the report with the rush.lj.fmt command in the rush.conf file.

Most all unix terminals support this feature, including xterm, konsole, rvxt, OSX "Terminal", Windows "putty". As of this writing, DOS is the only popular terminal application that does not support ANSI colors. Example:

    'rush -lj -x' Report
    % rush -lj -x           # include extended info       
    Status   Jobid      Title                     Owner  %Done %Fail %Hold #Frms Busy  Priority Ram   Progress             Rendertime   Age          Remarks
    -------- ---------- ------------------------- ------ ----- ----- ----- ----- ----- -------- ----- -------------------- ------------ ------------ -------
    Pause    tahoe.3    AE_8-009_0056_v012.aep    jal       %0    %0    0  100   0     10       1000  ____________________ 00:00:00     00:03:10     Job paused.
    Done     tahoe.4    AE_8-005_0010_v010.aep    jal      %80   %20    0  100   0     10       1000  ____________________ 00:10:12     00:32:56
    Run      tahoe.5    AE_8-011_0090_v003.aep    jal      %70    %0    8  100   20    90       1000  ____________________ 00:15:35     00:16:51
    Run      tahoe.6    AE_8-011_0091_v001.aep    jal      %40    %0   12  100   40    90       1000  ____________________ 00:10:40     00:10:51
                                                                                                                   
    

The default 'rush -lj' report shows the following information:

    Job Report Headings
    Heading Description
    Status The status of the job.

    Here's a state diagram showing the various states the job status transitions through:


    Click on the image to see a larger version.

    • Wait - The job is waiting for another job or time to finish (WaitFor, -waitfor)
    • Pause - The job has been paused with -pause
    • Run - The job is available to run frames
    • Done - The job is done, all the frames rendered successfully
    • Fail - The job is done, some frames failed

    The following are transitional states for the job, which you should only
    see for a short time:

    Jobid The job's jobid.
    Title The job's title (Title, -title).
    Owner The user that submitted the job.
    %Done The percentage of frames that are in the Done state.
    %Fail The percentage of frames that are in the Fail state.
    Busy The number of frames currently in the Run state.
    Elapsed The elapsed time in HH:MM:SS since the job started running.
    Progress A 20 character-wide color progress bar (using XTERM compatible ANSI codes)
    Remarks This field may contain messages like 'Job paused.' or
    'Wait for other jobs'.

The default 'rush -lj -x' report shows the following information:

    Job Report Headings
    Heading Description
    Status The status of the job.

    Here's a state diagram showing the various states the job status transitions through:


    Click on the image to see a larger version.

    • Wait - The job is waiting for another job or time to finish (WaitFor, -waitfor)
    • Pause - The job has been paused with -pause
    • Run - The job is available to run frames
    • Done - The job is done, all the frames rendered successfully
    • Fail - The job is done, some frames failed

    The following are transitional states for the job, which you should only
    see for a short time:

    Jobid The job's jobid.
    Title The job's title (Title, -title).
    Owner The user that submitted the job.
    %Done The percentage of frames that are in the Done state.
    %Fail The percentage of frames that are in the Fail state.
    %Hold The percentage of frames in the 'Hold' state.
    #Frms The total number of frames in the job.
    Busy The number of frames currently in the Run state.
    Priority The priority for the cpus in the job.
    If cpus have different priorities, they'll be listed
    comma separated in decreasing order.
    This can be shown in the -fmt report with 'Priority[Vals]:-8'
    Ram The amount of ram the job has requested
    Progress A 20 character-wide color progress bar (using XTERM compatible ANSI codes)
    Rendertime The wallclock time the job spent rendering
    Age The wallclock time since the job was submitted.
    This is the same as the 'Elapsed' time shown in the old 'rush -lj' reports.
    Remarks This field may contain messages like 'Job paused.' or
    'Wait for other jobs'.

Jobs Report Format

    The administrator has a certain amount of control over the default output format of the 'Jobs' report via the rush.conf file's rush.lj.fmt and rush.lj_x.fmt commands.

JSON Format

The -json flag enables JSON formatted reports of the form:

    Job server field of -json report
        {
          "server1": {
            "Jobs": [ .. ]
            "Errors": [ .. ]                                   
          }
          ,"server2": {
            "Jobs": [ .. ]
            "Errors": [ .. ]
          }
          ..
        }

At the top level, a dict of job server hostnames is used to collect jobs by the server that manages them.

Each job server item is itself a dict that can contain at most two possible fields: "Jobs" or "Errors". These fields may not appear at all if there are no Jobs or Errors. A description of these fields:

  • The "Jobs" field is an array of dicts, one dict per job. Each job dict contains all the job fields requested. Unless -fmt is specified to return specific fields, all job fields are returned. If there are no jobs, the "Jobs" field will not appear. It's also possible the "Jobs" field can appear with an empty array, e.g. "Jobs": [ ]

      'Jobs' field of -json report
        "server1": {
         "Jobs": [
          {
           "Jobid": "server1.20",                             
           "Title": "TEST_A",
           ..
          }
         ],[
          {
           "Jobid": "server1.21",
           "Title": "TEST_B",
           ..

  • The "Errors" field contains an array of possible error message strings detected by the remote server, one array element per error message string.

    If there are no errors the server can detect, the "Errors" field will not appear.

    Note: client side network errors will NOT appear in the "Errors" field; they are reported on stderr normally with messages of the form "rush: <error-message>". So when checking for errors, the user application should check for both JSON "Error" keys and /any/ output to stderr, and present any results to the user.

Here's an example of the 'rush -lj -json' report, using -fmt to limit which fields are returned:

    'rush -lj -json -fmt' Reports
    % rush -lj -json -fmt "Jobid State Title Owner Frames Cpus" zoar.52
    { "zoar": {
     "Jobs": [
      {
       "Jobid": "zoar.52",
       "State": "Done",
       "Title": "B",
       "Owner": "erco",
       "Frames": "1",
       "Cpus": [
        "eagle2=1@101k",
        "zoar=1@1"
       ]
      }
     ]
    }
    }

Rush's -json output can be loaded from python using various techniques. Here's a simple example using the 'json' module that comes with most OS installations of python:

    Python example to load JSON output of 'rush -lj -json'
    #!/usr/bin/python
    import json,os,sys
    #
    # Simple example: Load rush JSON output as a python dict, print job info
    #
    s = os.popen("rush -lj -json -fmt 'Jobid State Title' zoar.51 zoar.52", 'r').read()
    d = json.loads(s)                     # Load rush's JSON output into a python dict
    # Now print the jobs loaded
    for jobserver in d.keys():            # Loop through jobservers dict
        if "Jobs" in d[jobserver]:        # "Jobs" field exist?
            jobs = d[jobserver]["Jobs"]   # Get array of jobs
            for job in jobs:              # Loop through jobs[]
                print("Server %s -- Jobid: %s\tState: %s\tTitle: %s" % (jobserver, job["Jobid"], job["State"], job["Title"]))
    

See Also

   rush -ljc  
rush -ljc [-c count] [-s secs] [-ss secs] [hosts|+hostgroups..]

List Job Changes.

Lists 'change counters' for the job. This is a low latency way to poll jobs to see if they've changed, ie. if any frames have changed state, or if the job has been changed in some way.

This is useful for programs that need to show the entire queue, but don't want to constantly snapshot the entire queue even if nothing has changed. (see 'APPLICATION EXAMPLE' below)

This command uses UDP to transact with the job server(s), and therefore is light load for even a busy queue.

    rush -ljc options
    Option Description
    -c cnt Sets the number of times to continuously update the report.
    Default is 1. -c 0 creates a continuous update.
    -s secs Sets the floating point number of seconds between polls.
    This is a minumum; time might actually be longer depending on
    retries caused by non-responsive machines.
    Use with -c to poll automatically at a specified interval.
    Default is 3.0.
    -r secs Sets the number of times to retry contacting the remote machine
    before giving up. Default is 4.
    -ss secs Sets the floating point number of seconds between udp acknowledges.
    You probably don't want to mess with this unless you're getting
    weird communications behavior. Default is 0.05.
    -foe "Fail On Error". Causes the command to stop on hostname errors
    instead of continue to run for any other hostnames that resolve
    correctly. (default: off) *This flag new in 103.08*

Application Example

    There are two python examples of how to use 'rush -ljc' shown in:

            examples/python/howto/ljc_simple_example.py   -- Prints jobids of changed jobs
            examples/python/howto/ljc_example.py          -- Prints job info of changed jobs
            

    A synopsis of how one might use it in a GUI application:

            1) Run 'rush -ljf somehost' to get a full snapshot of all
               the jobs in 'somehost's queue, update an internal cache.
    
            2) Run 'rush -ljc -c 0 -s 8 somehost' in the background to keep 
               an eye on the job counters for changes. (This command
               will keep updating every 8 seconds forever)
               
            3) Whenever one or more job counters detect a change,
               the main program can run the appropriate commands to
               get the new info, eg:
               
                   a) A job counter changed? Use 'rush -ljf ..jobids..'
                   b) A frame counter changed? Use 'rush -lf ..jobids..'
                   c) A cpu counter changed? Use 'rush -lc ..jobids..'
    
            4) Repeat step 3 as needed to keep the program up to date.
            

    Note: it is wise if such an application were 'stowed' to kill the background -ljc command until it is re-opened, so that it doesn't continue to traffic the server when no one can see the interface.

   rush -ljf  
rush -ljf [-fmt str|env|defaults|help] [-t timeout] [jobids..] [hosts|+hostgroups..]

List Jobs Full.

Lists jobs showing 'full information' about the job, where data is presented as "key:value" pairs that can be easily parsed by scripts.

  • If one or more space-delimited [hosts..] are specified, the list of all jobs being managed by each host is shown.

  • If one or more space-delimited [jobids..] are specified, only those jobids are listed.

  • If neither [hosts..] or [jobids..] are specified, the contents of the RUSH_JOBID environment variable is checked for jobids, and all jobs managed by the servers of those jobids will be listed.

  • If none of the above are available, 'rush -ljf' defaults to listing all jobs being managed by the local machine.

This report prints out more information than most people usually need. The report is more intended for use by scripts needing access to as much information as rush can provide, or for debugging.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

(New in rush 103.00)
If -fmt is specified, its argument can be a space-separated list of keywords from the report, allowing you to limit the report to only show those fields of data. So where 'rush -ljf' will show ALL the fields, adding '-fmt "Jobid Title"' will only show the Jobid and Title fields from the report. Doing this also makes the rush communications use less bandwidth; useful if you just need a few fields from a LARGE list of jobs.

    'rush -ljf' Report
    
    $ rush -ljf
                 Jobid: tahoe.2
                 State: Done
              ActState: Done
              RebootId: 27
                 Title: HONDA
           DefPriority: 1
                LogDir: //server/jobs/HONDA/SC4A/rushlogs
                LogExt: -
              LogFlags: -
            ImgCommand: xv //server/jobs/HONDA/SC4A/imgs/sc4a-%04d.tif
                   Ram: 1
                  Nice: 10
               MaxCpus: 0
               MaxTime: 00:00:00
          MaxTimeState: que
          LicPauseSecs: 00:01:00
         LicPauseTimer: 00:00:00
            FrameFlags: off
               Command: python //server/jobs/HONDA/SC4A/render-vehicle 640 480
            ChkCommand: -
        JobDoneCommand: -
       JobStartCommand: -
        JobDumpCommand: -
              AutoDump: off
          WaitForState: dump
           WaitForTime: -
                  User: erco (1000/1000)
              DoneMail: -
              DumpMail: -
             StartDate: Fri Dec  9 20:58:37 2011
               EndDate: -
    FifoSchedOrderDate: 12/09/11,20:36:47.2
        FifoSchedOrder: 1322168315195
               Elapsed: 61:07:27
                Frames: 2
             FramesQue: 0
             FramesRun: 0
            FramesDone: 2
            FramesFail: 0
            FramesHold: 0
            JobRemarks: -
           RushRemarks: 
                  Cpus: superior=2@100
                  Cpus: +farm=20@10
              Notes[0]: PROD: Please see versions 1, 2 and 4 of this shot
              Notes[1]: ADMIN: Backup and remove at end of show
              Notes[2]: WRANGLERS: If this shot runs out of memory, limit the job to the +farm_bigmem hostgroup
            JobData[0]: COMMAND: ls -la
            JobData[1]: ASSETS: LINE ONE
            JobData[2]: ASSETS: LINE TWO
            JobData[3]: NOTES: requeued job
            JobData[4]: ASSETS: LINE THREE
              Criteria: 
    
    $ rush -ljf -fmt 'Jobid Command Frames'
                 Jobid: tahoe.2
               Command: python //server/jobs/HONDA/SC4A/render-vehicle 640 480
                Frames: 2
    
    $ rush -ljf -notes 'Notes[0]'
              Notes[0]: PROD: Please see versions 1, 2 and 4 of this shot
    
    $ rush -ljf -notes 'Notes[ADMIN]'
              Notes[1]: ADMIN: Backup and remove at end of show
    
    $ rush -ljf -notes 'Notes[WRANGLERS]'
              Notes[2]: WRANGLERS: If this shot runs out of memory, limit the job to the +farm_bigmem hostgroup
    
    $ rush -ljf -fmt 'JobData'
            JobData[0]: COMMAND: ls -la
            JobData[1]: ASSETS: LINE ONE
            JobData[2]: ASSETS: LINE TWO
            JobData[3]: NOTES: requeued job
            JobData[4]: ASSETS: LINE THREE
    
    $ rush -ljf -fmt 'JobData[1]'
            JobData[1]: ASSETS: LINE ONE
    
    $ rush -ljf -fmt 'JobData[COMMAND]'
            JobData[0]: COMMAND: ls -la
    
    $ rush -ljf -fmt 'JobData[ASSETS]'
            JobData[1]: ASSETS: LINE ONE
            JobData[2]: ASSETS: LINE TWO
            JobData[4]: ASSETS: LINE THREE
           

   rush -licpause  
rush -licpause [jobid..]
rush -licpause -

Pause Job Into A License Pause State.

Causes the current job to pause for the number of seconds specified by the submit command 'LicPauseSecs' or by the most recent 'rush -licpausesecs' command.

If [jobid..] is specified, that job(s) is changed to the LicPause state.
If '-' is specified, jobids are read from stdin. (Useful if the list of jobids exceeds operating system limits for argument lists and environment variable sizes)
If neither [jobid..] or '-' is specified, the RUSH_JOBID environment variable is used to determine the jobid(s).

Normally 'rush -licpause' is executed from within a render script when it detects the renderer failed with a license error, causing the job to pause for typically a minute or two, then unpauses itself again.

For info on how to use 'rush -licpause' mechanism, see the 'How To' documentation on Render License Errors.

'rush -licpause' will be ignored if the job has already been paused with 'rush -pause'.

    'rush -licpause' Example
    
       :   (your render script)
       :
       ## YOUR RENDER COMMANDS HERE
       render /job/MYSHOW/MYSHOT/ribs/foo.rib
       set $err = $status
    
       if ( $err == $PRMAN_LIC_ERROR ) then
    
           # LICENSE ERROR OCCURRED -- PAUSE JOB, REQUEUE FRAME  
           rush -licpause
           rush -notes ${RUSH_FRAME}:'License Error'
           exit 2
    
       endif
       :

See Also

   rush -licpausesecs  
rush -licpausesecs secs [jobid..]

Set Pause Time for License Errors

Sets the amount of time a job remains in the 'Pause' state when 'rush -licpause' is invoked from a render script.

For info on how to use 'rush -licpause' mechanism, see the 'How To' documentation on Render License Errors.

See Also

    LicPauseSecs -- Submit command to set the license pause seconds
    rush -licpausesecs -- Command line flag to change #secs job pauses during -licpause
    rush -licpause -- Command line flag to induce job into a 'license pause' state

   rush -log  
rush -log [-tail] [-showpath] <framerange> [jobid..]

View Frame Error Logs.

Prints the frame logs for the specified frame range.

The '-tail' option can be specified to continuously output appended data as the log file grows.

Using the job's 'logdir' value, prints the logs found in that directory for the frames you specify, e.g.:

    'rush -log' Example
    % rush -log 1-10,2 20   View logs for frms #1-10 on 2s, and frm #20  

The '-showpath' flag causes the report to show the pathname of the log file for the frame(s) specified. eg.

    'rush -log -showpath' Example
     % rush -log -showpath 5             # show logfile pathname for frame 0005
    LOGFILE: tahoe.4 /jobs/HONDA/SC13/doors-open.mb.log/0005
            

Irush internally makes use of the -showpath flag when it wants to show the log in a text editor.

This flag will take into account any log filename extensions enabled with the rush.conf file's framelog.ext setting. So if the framelog.ext flag is set to ".txt", then you will see that extension on the pathnames, e.g.:

    'rush -log -showpath' with .txt extension
     % rush -log -showpath 5
    LOGFILE: tahoe.4 /jobs/HONDA/SC13/doors-open.mb.log/0005.txt
            

   rush -logdir  
rush -logdir pathname [jobid..]

Log Directory.

Changes the job's log directory, where frame log files are written to.

This does not affect commands that are already running or have run. i.e., old logs are not moved to the new location, or renamed on the fly.

When changing the logdir with -logdir, no checks are made to see if 'path' is a valid directory.

'%s' can be used to insert the current jobid into the logdir path. Note: when the logdir is changed from the command line (rush -logdir) and has an embedded '%s', the directory is not created for you. Auto-creation of the logdir only occurs during job submission to avoid the "catch-22" scenario where the caller can't create the directory with a jobid in it if the jobid doesn't exist yet.

    Embedding the Jobid in the Log Directory's Path
    rush -logdir //server/jobs/logs/%s    This sets the LogDir to:
    //server/jobs/logs/tahoe.34  

See LogDir for more information.

   rush -logext  
rush -logext -|.txt [jobid..]

Log Extension.

Changes the filename extension used on all the text files saved into the job's log directory.

The default is normally no extension, however the sysadmin can change the network-wide default via the framelog.ext in the rush.conf file.

Changing the log extension while a job is running will only affect the logs for frames that haven't started rendering yet; frames that have already rendered or are rendering will still have the old filename extension. It is for this reason it's usually better to either submit the job with the correct 'logext' command, or have the sysadmin change the network wide default in the rush.conf file.

    Changing Log Filename Extensions
    rush -logext - tahoe.55 Disables filename extensions for log files. Example:
    //server/some/path/logs/0001
    //server/some/path/logs/0001.old
    //server/some/path/logs/framelist
    //server/some/path/logs/jobinfo
    rush -logext .txt tahoe.55 Causes all new frame logs to have ".txt" appended. Example:
    //server/some/path/logs/0001.txt
    //server/some/path/logs/0001.old.txt
    //server/some/path/logs/framelist.txt
    //server/some/path/logs/jobinfo.txt

Caveats

  • Changing the log extension while a job is running will not affect frames already rendered, or are in progress.
  • Changing this setting while a job is running will affect irush(1): irush will use the new log extension setting to view all logs, effectively preventing it from seeing logs for frames rendered with the old setting.

It is for these reasons it's better to set the logext either as part of the job submission, or have the sysadmin set the network wide default in the rush.conf file.

See Also

    logext -- Submit command to set the log filename extension
    framelog.ext -- Rush config file command to set the network-wide default

   rush -logflags  
rush -logflags <-|keeplast|keepall> [jobid..]

Log Flags.

Changes the log flags for the job. Examples:

    'rush -logflags' Examples
    rush -logflags - Logs are overwritten (Default)
    rush -logflags keepall   Keep all logs; concatenate old to *.old (e.g. 0012.old)
    rush -logflags keeplast    Like 'keepall', but only keeps last log (don't concatenate)

See Also

    LogFlags -- Submit command to set the logging flags

   rush -maxcpus  
rush -maxcpus <#cpus> [jobid..]

Sets the maximum number of cpus a job can use.

Puts a cap on the entire job. If maxcpus is set to 0, there is no limit.

    MaxCpus
    rush -maxcpus 0 Disabled -- no limit
    rush -maxcpus 15 Limit job to no more than 15 cpus busy rendering

See Also

    MaxCpus -- Submit command to set the maximum cpus the job can use

   rush -maxtime  
rush -maxtime <hh:mm:ss> [jobid..]

Sets maximum time for rendering frames.

Value is in HH:MM:SS (hours:minutes:seconds).

The system will automatically kill frames that exceed the specified maxtime.

You can control what happens to the frame when maxtime expires using MaxTimeState.

See Also

    MaxTime -- Submit command to set the maximum time for rendering frames

   rush -maxtimestate  
rush -maxtimestate <que|fail|done|hold> [jobid..]

What happens when MaxTime expires.

By default, rush requeues (Que) a frame when its MaxTime timer expires. You can change this to any of:

    'rush MaxTimeState Values
    rush -maxtimestate que Frames become "Que" when timer expires
    rush -maxtimestate fail Frames become "Fail" when timer expires
    rush -maxtimestate done Frames become "Done" when timer expires
    rush -maxtimestate hold Frames become "Hold" when timer expires

   rush -nice  
rush -nice <niceval> [jobid..]

Set Job's Niceness.

Sets the nice value for the job; 0 runs as normal user process, higher values (10, 20) increase niceness to others.

See Also

    Nice -- Submit command to set the job's unix 'niceness'

   rush -notes  
rush -notes <framerange>:'notes..' [jobid..]

Frame Notes.

Sets Notes for frame(s) on current [specific] job. These notes will show up in the rightmost column of 'rush -lf' reports.

    New in 103.06 and up.
    This command now uses the low latency UDP protocol, and so is safer to use on a large scale than the older TCP implementation (which could 'storm' the server with high latency TCP requests if many of the commands all ran across the farm in parallel at once).

    In older releases of Rush (103.05 and older), it is suggested to use 'rush -exitnotes' instead if you need to set the notes from within a render script when a frame completes.

Examples:

    'rush -notes' Examples
    rush -notes 155:"license error"   set notes for frame 155 to 'license error'
    rush -notes 200-250:"redo" set notes for frames 200 thru 250 to "redo"

Here's an example showing how to set the notes for frames 3 through 5 for a job:

    Setting The 'Notes' For Frames
    % rush -notes 3-5:"Redo For Client"
    [ontario.4] 0003: Redo For Client
    [ontario.4] 0004: Redo For Client
    [ontario.4] 0005: Redo For Client
        
    % rush -lf on.4
    STAT FRAME TRY HOSTNAME       PID     JOBID       START          ELAPSED  NOTES
    Done 0001  1   ontario        7762    ontario.4   02/23,16:17:26 00:00:11
    Done 0002  1   ontario        7764    ontario.4   02/23,16:17:26 00:00:11
    Done 0003  1   ontario        7777    ontario.4   02/23,16:17:38 00:00:12 Redo For Client
    Done 0004  1   ontario        7779    ontario.4   02/23,16:17:38 00:00:12 Redo For Client
    Done 0005  1   ontario        7787    ontario.4   02/23,16:17:51 00:00:12 Redo For Client  

Latency Issues

Most latency issues with running 'rush -notes' were solved in the Rush 103.06 release which changed to using the lower latency UDP protocol, allowing even large networks to adjust the notes en-masse without causing ill effects to the jobserver.

In older versions of Rush (103.05 and older), 'rush -notes' used the higher latency TCP protocol, bad if for example 200 machines all ran that command at once, overburdening the jobserver daemon for that job, "storming" the jobserver's kernel with too many sustained TCP connections.

The newer UDP behavior in 103.06 (and up) avoids this issue.

Before that, the only option was the 'rush -exitnotes' command (added in Rush 102.42a9) which used UDP, but could only change the notes when the render script exited. (The notes change happened as part of the rush messages that deliver the render script's exit code back to the job server). This is no longer needed in 103.06, as 'rush -notes' has become safer to use with its newer UDP implementation.

See Also

    Notes -- Submit command to set the job's notes
    rush -exitnotes -- Rush command line 'low latency' flag for renders to add notes to frames report

   rush -offline  
rush -offline [-foe] [-msg 'text'] [remotehost|+hostgroup..]

Offline A Machine.

Offline's the rush daemon on the specified hosts/hostgroups. If hostnames/hostgroups aren't specified, the local machine's rush daemon is taken offline.

When a machine is taken offline, frames that are already rendering are allowed to complete before taking the processors offline. Once offline, no new frames will be started.

[-foe] "Fail On Error". Causes the command to stop on any hostname lookup errors (such as temporary DNS or LDAP issues), instead of just printing warnings and continuing to run to handle any other hostnames specified. (default: off) *This flag new in 103.07*

'rush -offline' is similar to 'rush -getoff', differing only in that -offline allows any currently rendering frames to complete before taking the processors offline, whereas 'rush -getoff' will bump any running renders, freeing up the processors right away. The bumped frames will be requeued.

An optional remark can be specified via "-msg 'text'" which will show up in 'rush -lac' reports, eg:

    Offline With Custom Message
      % rush -offline -msg 'Bad Memory' ontario
      ontario: Offline
    
      % rush -lac
      HOST       OWNER    JOBID          TITLE                     FRM  PRI   PID        ELAPSED REMARKS
      ---------- -------- -------------- ------------------------- ---- ----- ------- ---------- --------
      ontario    -        -              -                         -    -     -          Offline Bad Memory  
      :

Examples of using 'rush -offline':

     'rush -offline' Examples 
    rush -offline   offline all cpus on local host
    rush -offline tahoe offline all of tahoe's cpus
    rush -offline +farm   offline the entire render farm
    rush -offline -msg 'Failing drive' tahoe   offline 'huron' with the remark 'Failing drive'

To return processors to the online state, use rush -online.

Changes to the online/offline state are appended to the rushd.log.
Also: changes to the online/offline state append 's' entries to the cpu.acct accounting log

   rush -online  
rush -online [-foe] [-msg 'text'] [remotehost|+hostgroup..]

Online A Machine.

Online's the rush daemon on the specified hosts/hostgroups. If hosts/hostgroups are not specified, the local machine's rush daemon is put online.

When a machine is online, frames will immediately be allowed to start rendering on the available processors.

The '-msg' flag is not recommended for use with -online, due to the message's persistence in 'rush -lac' reports when online, but is supported for symmetry with the -getoff/-offline commands.

[-foe] "Fail On Error". Causes the command to stop on any hostname lookup errors (such as temporary DNS or LDAP issues), instead of just printing warnings and continuing to run to handle any other hostnames specified. (default: off) *This flag new in 103.07*

Examples of using 'rush -online':

    'rush -online' Examples
    rush -online   Online all cpus on local host
    rush -online tahoe Online all cpus on host tahoe
    rush -online +farm   Online all cpus in the '+farm' hostgroup   

To take hosts offline, use either rush -offline or rush -getoff.

Changes to the online/offline state are appended to the rushd.log.
Also: changes to the online/offline state append 's' entries to the cpu.acct accounting log

   rush -pathconvert  
rush -pathconvert [-list] <pathname>

Do pathname conversions

When the admin configures pathname-to-pathname conversions in the rush/etc/path_convert file, Rush will internally use these mappings for localizing e.g. 'logdir' paths. But scripts (such as the example submit scripts, or your own custom scripts) can use 'rush -pathconvert /some/path' to normalize paths via the rules setup in path_convert as well.

For instance, let's say your path_convert file is setup to change /Volumes/share/xxx into //someserver/share/xxx. Then in your custom scripts, you can invoke:

    rush -pathconvert /Volumes/share/foo
..and that would print to stdout the revised pathname:
    //someserver/sharename/foo

..based on the rules in the path_convert file. It's also dependent on which platform the command is run; depends on the platform specific rules the sysadmin configured in the path_convert file.

The following example shows how 'rush -pathconvert pathname' can be used to translate pathnames from within e.g. a csh script:

    Sample use of 'rush -pathconvert' in a script
    
        #!/bin/csh -f
        # MyRender <xres> <yres> <scenefile>
        set xres=$1
        set yres=$2
        set scenefile=`rush -pathconvert $3`                # convert the specified scenefile
        Render -r mr -s $RUSH_FRAME -e $RUSH_FRAME $scenefile
        if ( $status ) exit 1
        exit 0
    
            

New in 103.03 and up.
The optional "-list" flag can be specified to list all the path conversions the sysadmin has configured in the rush/etc/pathconvert file.

     Using 'rush -pathconvert' -list option 
    $ rush -pathconvert -list
    FROM             TO
    ---------------  --------------------
    /ourserver/data  //ourserver/data
    /Volumes/data    //ourserver/data
    z:               //ourserver/data
            

   rush -pause  
rush -pause [jobid..]
rush -pause -

Pause Job

Pauses the current [specific] job. Busy frames will be allowed to finish, no new frames will be started.

If '-' is specified, jobids are read from stdin. (Useful if the list of jobids exceeds operating system limits for argument lists and environment variable sizes)

To pause a job and kill the running frames, use:

     Pause a Job, Kill Running Frames 
    rush -pause && rush -que run

To continue a job from pause, use rush -cont.

   rush -ping  
rush -ping [-t timeout] [-foe] [remotehost|+hostgroup..]

Ping Rush Daemon.

(Administrative) Pings the local [remote] daemon to see if it's running, and what it's doing.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

[-foe] "Fail On Error". Causes the command to stop on any hostname lookup errors (such as temporary DNS or LDAP issues), instead of just printing warnings and continuing to run to handle any other hostnames specified. (default: off) *This flag new in 103.07*

    Ping Info
    
        % rush -ping +any -t 3
         ozark: RUSHD-64bit 103.00/Linux  PID=1534    Boot=02/03/11,20:08:26  Online, 0 jobs, 0 procs, 0 tasks, dlog=-, nfd=7, ssb=1402, sched=rr
            ta: RUSHD-64bit 103.00/Linux  PID=8263    Boot=01/25/11,11:25:02  Online, 2 jobs, 0 procs, 0 tasks, dlog=-, nfd=7, ssb=15419525, sched=rr
         tower: RUSHD-64bit 103.00/Mac    PID=12233   Boot=01/15/11,15:42:14 Offline, 1 jobs, 0 procs, 0 tasks, dlog=-, nfd=8, ssb=4101073, sched=rr
       crystal: RUSHD-64bit 103.00/Mac    PID=1861    Boot=01/19/11,01:27:41  Online, 2 jobs, 0 procs, 0 tasks, dlog=-, nfd=8, ssb=1427328, sched=rr
      superior: RUSHD-64bit 103.00/WinXP  PID=940     Boot=02/03/11,20:29:51  Online, 0 jobs, 0 procs, 0 tasks, dlog=-, nfd=4, ssb=36790, sched=rr
                      ----- ------ -----  ---------   ---------------------- -------  ------  -------  -------  ------  -----  ---------  --------
                       |       |     |       |                |                |        |       |         |       |       |      |         |
                       |       |     |       |                |                |        |       |         |       |       |      |         Scheduler setting in rush.conf
                       |       |     |       |                |                |        |       |         |       |       |      |         rr=round robin
                       |       |     |       |                |                |        |       |         |       |       |      |         fifo=first-in,first-out
                       |       |     |       |                |                |        |       |         |       |       |      (debugging)
                       |       |     |       |                |                |        |       |         |       |       (debugging) Number of open file descriptiors
                       |       |     |       |                |                |        |       |         |       |                   Used to detect leaks.
                       |       |     |       |                |                |        |       |         |       Daemon's current logging flags.
                       |       |     |       |                |                |        |       |         |       (Can be changed with 'rush -dlog xxx')
                       |       |     |       |                |                |        |       |         Number of tasks scheduled
                       |       |     |       |                |                |        |       Number of procs busy running renders.
                       |       |     |       |                |                |        Number of jobs being hosted by this machine.
                       |       |     |       |                |                Daemon's online/offline state.
                       |       |     |       |                |                Controlled by 'rush -online/-offline/-getoff'
                       |       |     |       |                Daemon's start time.
                       |       |     |       Daemon's PID
                       |       |     Operating system details.
                       |       Daemon's version number.
                       Daemon executable (32bit=32bit application, 64bit=64bit application)
        

   rush -push  
rush -push [-s secs] [-c tries] <file[,file..]> [host|+hostgroup]

Push File(s) To Remotes.

(Administrative) Pushes rush configuration files (such as rush.conf, hosts, or any files in the rush/etc directory) to remote servers. Use to re-distribute one machine's rush config files to all machines quickly.

  • -s secs - Specifies the number of seconds to wait for replies from remotes. Default is 5.

  • -c tries - Number of retries made if no response in the number of seconds specified (-s secs). Default is only 1 try is made.

  • [host|+hostgroup] - Optional hostname or +hostgroup. Multiple hostnames or +hostgroups can be specified if separated by spaces. If unspecified, the default is "+any" (all machines).

Files to be 'pushed' must be specified by name. Only files in the rush/etc directory can be 'pushed'. Specify filenames only, not full paths. More than one file can be specified if separated by commas.

"file" may only contain the characters "A-Z", "a-z", "0-9", "-", "_", and a single ".". This prevents abusive use of -push, since only files in $RUSH_DIR/etc are affected.

Only root and the rush adminuser can use this command.

The sysadmin can enable/disable 'rush -push' operations via a setting in the rush.conf file, using the AllowPush flag.

    'rush -push' Examples
    % rush -push rush.conf +any     Push rush.conf to only hosts
    in the +any hostgroup
    % rush -push hosts +any     Push rush hosts file to all hosts
    in the +any hostgroup
    % rush -push hosts,rush.conf,path_convert Push hosts, rush.conf and path_convert file to all hosts
    % rush -push rush.conf,license.dat     Push rush.conf and license.dat to all rush hosts  
    % rush -push rush.conf tahoe huron     Push rush.conf to hosts 'tahoe' and 'huron' only
    % rush -push rush.conf,hosts,license.dat tahoe huron     Push rush.conf, hosts, and license.dat to hosts
    'tahoe' and 'huron' only

   rush -que  
rush -que <framerange|framestate..> [jobid..]

Re-Queue Frames

Changes frame(s) into the Que state for current [specific] job. If frames are running, they are killed and enter the 'Que' state on completion.

The frames to affect can either be specified as a frame range (i.e., 1-100) or as a frame state, for which all matching frames will be affected. Examples:

    'rush -que' Examples
    % rush -que 1-100 Que frames 1 through 100
    % rush -que fail Que all frames currently Fail
    % rush -que fail done    Que all frames currently Fail or Done  
    % rush -que all Que all frames

   rush -ram  
rush -ram <ramval> [jobid..]

Set Job's Ram Use.

Sets the Ram use (in MB) for the job. See Ram submit command for more info.

See Also

    Ram -- Submit command to set the job's estimated ram use (used for scheduling processors)

   rush -ramlist  
rush -ramlist [remotehost..]

View the ram usage of rush jobs for localhost [remotehost] machine.

Shows a report of all running rush processes on the machine, including how much ram each has requested to reserve, and a running total for the available ram left over.

    'rush -ramlist' Report
      % rush -ramlist tahoe
      STATE   JOBID/TITLE                    PRI    RAMUSE NOTES  
      Busy    neverland.57,grkenttest        1         800     
      Busy    neverland.58,pflare            1         200     
                                                    ------
          Total ram on tahoe: 1024  
      Available ram on tahoe: 24 

   rush -rc  
rush -rc <cpuspec|tidspec|hostname|*> [jobid..]

Remove Cpu

Removes Cpu from cpu list of current [specific] job. If the cpus being removed are busy running frames, the frames are killed and re-submitted elsewhere.

Cpus can be removed in one of several ways:

  • Individually by the JOBTID number, e.g. rush -rc .32
  • By hostname (or host alias), e.g. rush -rc tahoe, rush -rc ta
  • By a particular cpu specification as shown in the 'rush -lc' report, e.g., rush -rc tahoe=4@200
  • The special string "*" can be used to remove all cpus

To remove a cpu via 'JOBTID' values (e.g., 'rush -rc .334') you must precede each value with a period. When you delete by JOBTID, you are deleting single cpus from the 'rush -lc' report. Note: this report includes the JOBTID values so you can see which values to delete. If the cpu you delete is part of a larger specification, (e.g., tahoe=4@12), then the cpu count for the spec will be modified and the cpu count in that spec will be decremented as well (e.g., tahoe=3@12)

If you remove all cpus (e.g., 'rush -rc "*"') then all cpus assigned to the job are removed. This would then allow you to add all new cpus.

If you remove a hostname (e.g., 'rush -rc tahoe') then all cpu specifications that have that host name (e.g., tahoe=3@100) will be removed. Also, any hostgroups that expand to include that host will have that host removed from the expansion (e.g., +any=3@100, which includes tahoe).

Hostname aliases can also be used to remove hosts. In previous versions, hostname aliases were not guaranteed to work for 'rush -rc'.

    Removing a Host
     % rush -lc
      CPUSPEC[HOST]        STATE       FRM  PID     JOBTID  PRI   ELAPSED  NOTES
      howland=2@900        JobPass     -    -       171     900   00:00:00 No more frames
      howland=2@900        JobPass     -    -       172     900   00:00:00 No more frames
      +any=1@1[howland]    JobPass     -    -       147     1     00:00:01 No more frames
      +any=1@1[howland]    JobPass     -    -       148     1     00:00:00 No more frames
      +any=1@1[nt-1]       JobPass     -    -       149     1     00:00:31 This is a 'neverhost'
      +any=1@1[nt-1]       JobPass     -    -       150     1     00:00:31 This is a 'neverhost'
      +any=1@1[rotwang]    JobPass     -    -       151     1     00:00:01 No more frames
      +any=1@1[rotwang]    JobPass     -    -       152     1     00:00:01 No more frames
    
      % rush -rc howland
      nt-1.15 'howland=2@900': Removed
      nt-1.15 '+any=1@1': 2 task(s) deleted
    
      % rush -lc
      CPUSPEC[HOST]        STATE       FRM  PID     JOBTID  PRI   ELAPSED  NOTES
      +any=1@1[nt-1]       JobPass     -    -       149     1     00:00:37 This is a 'neverhost'  
      +any=1@1[nt-1]       JobPass     -    -       150     1     00:00:37 This is a 'neverhost'
      +any=1@1[rotwang]    JobPass     -    -       151     1     00:00:08 No more frames
      +any=1@1[rotwang]    JobPass     -    -       152     1     00:00:08 No more frames

If you remove a cpu specification (e.g., 'rush -rc +any=3@100'), it must match character-for-character the entry shown in the 'rush -lc' report for the job:

    Removing a Hostgroup
      % rush -ac tahoe@100                        # Add a cpu.
      % rush -rc tahoe@100                        # Now try to remove it
      'tahoe@100' no such cpu specification       # FAILED: need to use spec shown in 'rush -lc'  
     
      % rush -lc                                  # Look at 'rush -lc' report
      CPUSPEC      STATE  FRM   PID    ELAPSED .. # More complete specification in report.
      tahoe=1@100  Run   0002  26747  00:00:11 ..    
     
      % rush -rc tahoe=1@100                      # Remove using spec shown in report
      'tahoe=1@100' removed                       # It works
            

   rush -reload  
rush -reload <[hosts|rush.conf|pathconvert]]> [host|+hostgroup..]

Reload daemon's files (rush.conf, hosts..).

Forces the local [remote] daemon to reload the specified file(s).

The first argument after -reload can be "hosts", "rush.conf", or "pathconvert" or comma separated combinations, eg. "hosts,rush.conf".

Whenever 'hosts' is specified, the rush/etc/hosts file is reloaded by the local [remote] daemon, its IP cache flushed, and the negative cache is cleared.

    'rush -reload' Examples
    rush -reload hosts Tells the local machine's rushd to reload its hosts file
    flushing its IP cache and negative cache.
    rush -reload hosts +any Tells all machines to reload their hosts files
    and flush their IP and negative caches
    rush -reload rush.conf +any Tells all machines to reload their rush.conf files
     
    rush -reload pathconvert +any Tells all machines to reload their rush/etc/pathconvert files (if any)
     
    rush -reload hosts,rush.conf +any Tells all machines to reload hosts and rush.conf
    and flush their IP and negative caches
    rush -reload hosts tahoe geneva Tells only 'tahoe' and 'geneva' to reload hosts files
    and flush their IP and negative caches

Caveats

    In DNS environments, using 'rush -reload hosts +any' should be safe for your DNS server, though it will keep it steadily busy while the command is updating each host; each host will reload the hosts file, causing all the hosts to resolve to update the IP cache.

    If your hosts file is large, this will cause a lot of requests to your DNS server from each machine while the hosts file is reloaded and re-cached. (The exception being if your machines do their own DNS caching) Fortunately, the 'rush -reload' operates serially; it hits each machine one at a time, and waits for the remote to load the hosts file completely before moving to the next machine, ensuring there's only one machine reloading the hosts file at any given moment in time.

   rush -reorder  
rush -reorder <framerange..> [jobid..]

Reorder Frames.

Reorders frames in the frame list to the new order specified. <framerange> specifies the new order of the frames in the frame list. Several frame ranges can be specified. Unspecified frames are simply left alone.

Changing the order of the framelist affects the order frames are rendered in, since frames are issued from the top of the list, down.

Example-- If you have a job with 10 frames in the frame list that are in normal 1 thru 10 order, you can use 'rush -reorder' to get different orderings, such as these:

    'rush -reorder' Examples
    rush -reorder 10-1 becomes 10 9 8 7 6 5 4 3 2 1
    rush -reorder 1-10,2 2-10,2   becomes 1 3 5 7 9   2 4 6 8 10  
    rush -reorder 5 6 7 8 9 10 1 2 3 4   becomes exactly that order: 5 6 7 8 9 10 1 2 3 4  

See Also

    Frames -- Submit command to set the job's frames (including custom frame# ordering)

   rush -reserve  
rush -reserve [-server <host>] <cpuspec> [ramval]

Reserve Cpus.

Reserves cpus by creating a 'dummy' job that holds the cpus. The job's title will be RESERVE. Reservation jobs are like any other-- to remove a reservation, dump the job. To change the reservation, you can use IRUSH or the usual rush commands like -ac/-rc to modify the job.

'cpuspec' indicates which processors are reserved, and at what priority. It is an error not to include a priority value in the cpuspec.

The optional '-server <host>' can be specified, to make 'host' the job server for the reserve job, instead of the default which is to have the job server be on the same host as the cpus being reserved.

A reservation job with a priority of '500' will prevent jobs with priorities less than 500, but yields to jobs higher than 500k. Including the 'k' flag in your reservation will also bump any renderings that are running on your machine that are lower than 500 priority. See Priority Description for more info on how rush priorities work.

As an example of reserving cpus on your workstation, let's assume you sit at a dual proc workstation called 'tahoe':

  • To reserve both processors of your dual proc workstation at a priority of '500', use:

       Reserve Both Processors On A Dual Processor Workstation 
      rush -reserve tahoe=2@500k   Reserve both cpus at 500k priority  

  • To submit a job that actually renders on your own workstation, such that one processor is used for rendering, leaving the other for your own interactive use, just submit with a cpu specification that requests your workstation at a priority higher than your reservation, with the 'kill' flag, e.g. "tahoe=1@510k".

    To get both cpus, just request both processors, e.g. "tahoe=2@510k".

  • To remove the reservation for your machine, simply dump the reservation job.

  • To prevent all jobs from rendering on your workstation-- even your own-- make your reservation at a priority of 999. Or, just use 'rush -offline' and/or 'rush -getoff'.

The optional 'ram value' allows one to reserve an amount of ram per processor. This is useful on multiproc machines where you want to prevent large jobs from rendering on the few processors left unreserved. Increasing the ram value you reserve will decrease the size of jobs rush will allow to run on the unused processors.

If 'cpuspec' doesn't include the number of processors, one processor is assumed. If 'ramval' is unspecified, a value of '1' is assumed.

    'rush -reserve' Examples
    rush -reserve tahoe@998 Reserve 1 cpu on tahoe@998
    rush -reserve tahoe=2@998 128   Reserve 2 cpus @998, 128MB of ram to each
    rush -reserve -server univac tahoe=2@998 128   Same as above, but jobserver will be 'univac'  
    rush -reserve tahoe=2@500 128 Reserve 2 cpus @500, 128MB of ram to each

   rush -rf  
rush -rf <framerange..> [jobid..]

Remove Frames.

Removes frames from the frame list of the current [specific] job.

If frames are Busy at the time they are removed, they will first be killed, then scheduled for removal when confirmation arrives from the remote machine(s).

    'rush -rf' Examples
    rush -rf 10-15 Removes 10 thru 15 from the current job
    rush -rf 108 110 800-899 Remove frames 108, 110, and 800-899
    from the current job
    rush -rf -10--5 Removes negative frame range: -10 thru -5
    rush -rf .1-5,.1 Removes floating point frames .1 through 5   
    stepping by .1 (.1, .2, .3 .. 4.9, 5.0)

   rush -rn  
rush -rn <hostname|+hostgroup|*..> [jobid..]

Remove Nevercpus.

Removes Nevercpus for current [specific] job.

    'rush -rn' Examples
    rush -rn tahoe meade eagle.123    Removes 'tahoe' and 'meade' from job eagle.123's nevercpus list
    rush -rn +farm eagle.123 Removes all hosts in the +farm group from eagle.123's nevercpus list
    rush -rn +any eagle.123 Removes all hosts from job eagle.123's nevercpus list
    rush -rn "*" eagle.123 Removes all nevercpus from job eagle.123

   rush -rotate  
rush -rotate [rushd.log|cpu.acct] [remotehost|+hostgroup..]

Rotate Rush Log Files

(Administrative) Rotates the rushd.log file by default on local [remote] host.

The cpu.acct log can also be rotated if the optional argument 'cpu.acct' is specified.

Can be used in crontab scripts to rotate logs on local or remote machines.

The rushd.log is normally automatically rotated with the LogRotateHour command in the rush.conf file, however that can be disabled so that you can configure your own weekly or monthly log rotations via a crontab using 'rush -rotate'.

    'rush -status' Examples
    rush -rotate rushd.log Rotates the rushd.log on local machine
    rush -rotate rushd.log tahoe ontario    Rotates rushd.log on remote hosts 'tahoe' and 'ontario'
    rush -rotate rushd.log +any Rotates rushd.log on all machines
    rush -rotate cpu.acct Rotates cpu.acct log on local machine
    rush -rotate cpu.acct tahoe ontario    Rotates cpu.acct log on remotes 'tahoe' and 'ontario'
    rush -rotate cpu.acct +any Rotates cpu.acct on all machines

Only root and the rush administrators can use this command.

   rush -showconf  
rush -showconf [remotehost|+hostgroup..]

Show Daemon's rush.conf Settings

Views a remote daemon's current loaded rush.conf settings, so one can see what settings the daemon is actually using.

This does not necessarily reflect the contents of the remote machine's rush.conf file, but rather, the config the daemon currently has loaded into memory.

So for instance, if the rush.conf file was recently changed, 'rush -showconf' won't show the change until the daemon notices the changed rush.conf file and reloads it. (Usually within 30 seconds).

Use this to verify changes to the rush.conf file were interpreted by the daemon as you expected.

   rush -status  
rush -status [-jobs] [-cpus] [-usage] [-s secs] [-c count] [-foe] [remotehost|+hostgroup..]

Status of Host.

(Administrative) 'rush -status' quickly reports the status of jobs and cpus on either the local machine, or the specified remote [host].

Commonly used as 'rush -status' to get the status of the local host, or 'rush -status +any' for the status of all hosts.

If none of the options '-jobs', '-cpus' or '-usage' are specified, all are assumed. ie. 'rush -status' is the same as 'rush -status -jobs -cpus -usage'.

Output is intended for other programs to parse and regenerate.

    'rush -status' Options
    -jobs Shows job info ('j' entries).
    -cpus Shows cpu info ('p' entries).
    -usage Shows usage of cpu, memory and swap ('C', 'M' and 'S' entries).
    -s <secs>   Sets the amount of time to wait for responses from remotes.  
    Default is 3.
    -c <count> Sets the number of times to continuously update the report.
    Default is 1. -c 0 creates a continuous update.
    -foe "Fail On Error". Causes the command to stop on hostname errors
    instead of continue to run for any other hostnames that resolve
    correctly. (default: off) *This flag new in 103.07*

The output contains several records of information for each host, one record per line. Host records start with an 'h' record, and terminate with a line of '--'.

8 different types of data records are possible. Data record types are defined by the first character in each record line, and can be one of:

    Line Prefixes
    Prefix Description
    h
    Hostname header. Leads off records for the specified host.
    d
    Daemon information. Info about the running daemon.
    j
    Job information. One line per job. (-jobs)
    p
    Processor status. One line per processor. (-cpus)
    C
    Cpu status. (-usage)
    M
    Memory status. (-usage)
    N
    Network I/O (-usage)
    S
    Swap status. (-usage)
    X
    Error message (eg. "X PdhGetFormatted[usr0]: query not validated")

Each record has its own fields:


    h <hostname>
    d <sequence-id> <daemon> <version> <PID=pid> <online state> <jobs> <busy procs> <total procs>
    r <boot-time> <idle-time> <total-render-time>
    j <owner> <jobid> <job title> <job state> <elapsed> <percent done> <percent fail> <# frms busy> <"job remarks">
    p <owner> <jobid> <job title> <frame> <priority> <pid> <elapsed>
    C <ncpus> <usr%> <sys%> [<usr%> <sys%> ..]
    M <total mb> <used%>
    S <total mb> <used%> <swapin> <swapout> <'bad'swapping>
    N <timeofday (in floating secs)> <in(kbytes)> <out(kbytes)> <ifaces total errs> <ifaces total spd (Mbits/sec)>
    X <Error message text..>
    W <Warning message text..>
    
CAVEATS:
  'r' (render stats):

      o 'boot-time' (seconds) is the time(2) the rush daemon started.

      o 'idle-time' (HH:MM:SS) is how long since the machine last rendered
        a job.  The value is in hours:minutes:seconds; 00:00:00 indicates
        it's actively rendering. The timer starts counting when all
        processors on the host are idle, and resets when a new render
        picks up.  Use this to determine if a farm machine should be
        powered down due to being idle for too long.

      o 'total render time' (HH:MM:SS) is a total of the frame elapsed
        times for all renders that completed running (done or fail or
	even interrupted) so far since the rushd daemon started.
	Does not include times for renders that are in progress.

  'S' (Swap) entries:
      o swapin and swapout and 'bad swapping' values are in 'pages'

  'X' (Error) entries:
       o Error lines are not present unless there were fatal errors
       o There may be more than one 'X' line, a separate error message in each
       o 'X' lines should be shown to the user along with the hostname
         from the 'h' line
 
  'W' (Warning) entries:
       o Warning lines are not present if there are no warnings
       o 'W' lines should be shown to the user along with the hostname
         from the 'h' line

  'N' (Network) entries:
       o It is advised you graph these values as follows:
         {
           // Determine floating point #secs between last sample and current
           static double peak = 0;
           // Make sure none of the values 'wrapped'
           // If they did, skip calculations and just save the current as last
           // so that the /next/ sample will be with unwrapped values..
           if ( curr_secs > last_secs &&
                curr_in_kb > last_in_kb &&
                curr_out_kb > last_out_kb ) {
             double secs_diff = curr_secs - last_secs;
             if ( secs_diff > 0 ) {
               in_diff_kbps  = (curr_in_kb  - last_in_kb)  / secs_diff;  // determine input kb per sec
               out_diff_kbps = (curr_out_kb - last_out_kb) / secs_diff;  // determine output kb per sec
               io_diff_kbps  = indiff_kbps + outdiff_kbps;               // determine total kb per sec
               if ( io_diff_kbps > peak ) peak = io_diff_kbps;           // determine peak kb per sec
               if ( peak_kbs != 0 ) {
                 in_percent  = (in_diff_kbps  / peak_kbps) * 100;        // percent of peak for in
                 out_percent = (out_diff_kbps / peak_kbps) * 100;        // percent of peak for out
               }
               ..graph 'in_percent' and 'out_percent'..
             }
           }
           // Save current samples as 'last'
           last_secs   = curr_secs;
           last_in_kb  = curr_in_kb;
           last_out_kb = curr_out_kb;
        }
       o Any of the sample values (including the time) may wrap around.
         If so, it's suggested you save all the values in this sample as
         'last', and skip graphing this sample.
       o 'timeofday' is the sample time as a floating point number of seconds
         It's used by the client to calculate peak kbs/sec between samples.
       o 'iface speed' is considered 'optional data', as it's not always
         provided by the OS. It's therefore recommended you not try to use
         the iface speed value at all, and instead calculate 'peak use'
         and graph against that.
       o To calculate peak use in kbps (kilobytes per sec), take diff
         between last+current in/out kbytes sample, and divide
         by the diff between last+current timevals.
       o in/out totals are totals of /all/ interfaces found.
         This may include 'logical' interfaces (such as filters
         and bridges) which may mirror a physical interface's
         usage. We try to ignore logical interfaces, but sometimes
         the OS doesn't make this evident, and in such cases
         we just sum them all. Thus, kb totals may be artificially
         scaled by 2x or 3x.
    

    'rush -status' Examples
    rush -status Shows the status of the local host
    rush -status +any Shows the status of all hosts
    rush -status -usage +any Shows only the cpu usage of all hosts
    rush -status -jobs +any Shows only the job info for all hosts
    rush -status -jobs -cpus +any Shows only the job and cpu info for all hosts
    rush -status -jobs -cpus -usage +any   Same as 'rush -status +any'

   rush -submit  
rush -submit [remotehost] [-sh|-csh|-dos|-python|-perl|-jobid]

Submit Job

Submits a job to be managed by the local [remote] job server. Reads stdin for Rush Submit Commands which define the job submission.

Optional [remotehost] can be specified to specify some other machine to be the server for the submitted job. By default, the machine that 'rush -submit' runs on becomes the machine that manages this job. This is usually the user's workstation. This ensures there's no "single point of failure" for the render queue, and distributes the overall load of managing jobs to multiple machines.

Optional shell flags [-sh|-csh|-dos|-python|-perl|-jobid] control how the jobid of a successfully submitted job is printed on stdout. For instance, '-csh' allows a csh script to directly 'eval' the output of the 'rush -submit' command to automatically set the RUSH_JOBID environment variable on a successful submit, eg:

     CSH using 'eval' to set the RUSH_JOBID 
    #!/bin/csh -f
    eval `rush -submit -csh` << EOF
        frames  1-10
        command sleep 50
        cpus    +any=5
    EOF
    if ( "$RUSH_JOBID" == "" ) then
        echo Job failed to submit
        exit 1
    else
        echo Job submitted: $RUSH_JOBID
        rush -lj    # RUSH_JOBID set by 'eval' command above
        rush -lf    # RUSH_JOBID set by 'eval' command above
    endif
            

    Optional Submit Shell Flags
    New in 103.00
    rush -submit -sh Bourne Shell syntax: RUSH_JOBID=xxx.123; export RUSH_JOBID
    rush -submit -csh CSH syntax: setenv RUSH_JOBID xxx.123
    rush -submit -dos DOS syntax: set RUSH_JOBID=xxx.123
    rush -submit -python    Python syntax: os.environ["RUSH_JOBID"]="xxx.123"
    rush -submit -perl Perl syntax: $ENV{"RUSH_JOBID"}="xxx.123";
    rush -submit -jobid Just print the jobid: xxx.123

For examples of how to make custom scripts that use 'rush -submit', see submit scripts in various languages and the rush command line tutorial which shows how to submit and manage jobs from the command line.

When this command completes successfully, it returns an exit code of 0, and the jobid of the submitted job is printed on stdout, which can be used to monitor the job. Warnings may also be printed.

If the job failed to submit, a non-zero exit code is returned, and the reason for the failure will be printed on stderr.

If you're writing a GUI application that submits jobs, you should keep stdout AND stderr of the 'rush -submit' command, and present the output to the user in case of error or warnings. Here's a python example:

    Python Example: How to parse errors from 'rush -submit'
    import os,re,string
    
    # Temp file for output
    tmpfile = "/tmp/.rush-submit-" + str(os.getpid())
    
    # Submit the job, save stdout+err to 'tmpfile'
    submit = os.popen("rush -submit > " + tmpfile + " 2>&1", 'w')
    submit.write("title    MYJOB\n" +
                 "frames   1-10\n" +
                 "command  ls\n" +
                 "cpus     +any=5@1\n")
    submit.close()
    
    # Read all lines of stdout/err from 'rush -submit'
    f = open(tmpfile, 'r')
    submitout = string.join(f.readlines())
    f.close()
    os.remove(tmpfile)              # remove tmp file
    
    # Parse jobid (if any)
    try:
        jobid = re.search("RUSH_JOBID.(\S+)",submitout).groups()[0]       
    except:
        jobid = ""
    
    # If no jobid found, submit failed
    if ( jobid == "" ):
        ShowDialogToUser("SUBMIT FAILED\n" + submitout)
        sys.exit(1)
    else:
        ShowDialogToUser("SUBMIT WORKED: JOBID IS " + jobid)
    
        

   rush -tail  
rush -tail [-f] [-#lines] file [file..]

Tail A File.

New in rush 103.00 and up.

This allows mainly Windows users to run tails on files (such as the daemon log), since windows does not have a built-in 'tail' feature. This option attempts to duplicate the operation of the unix tail(1) command.

New in 103.08 and up To see the entire file, set #lines to -1.
Since specifying a number of lines must be preceded with a dash, you must use "--1" to specify a -1 for lines.

     Various Examples of 'rush -tail' 
    rush -tail /usr/local/rush/var/rushd.log View last 10 lines of rushd.log
    rush -tail -100 /usr/local/rush/var/rushd.log View last 100 lines of rushd.log
    rush -tail -f /usr/local/rush/var/rushd.log View last 10 lines of rushd.log, and follow all new output
    rush -tail -f --1 /usr/local/rush/var/rushd.log Show entire file and follow all new output

   rush -tasklist  
rush -tasklist [-desched] [host..|+hostgroup]

Task List.

Lists scheduled tasks on the specified remote [host..|+hostgroup]. If no host or +hostgroup is specified, the local machine's scheduled tasks are shown.

This report shows the scheduler's sorted list of tasks wanting to run on the machine queried.

Tasks are sorted in the order they will be executed; the topmost task is next to be rendered.

Tasks that are already busy running are sorted to the bottom. Tasks that were denied by either the job, or by cpu criteria, are also sorted to the bottom.

Tasks are sorted into 'groups' of the same priority, and will schedueled within those groups in either round robin or FIFO order, depending on the value of the rush.conf sched setting.

The -desched option enables showing which tasks are 'descheduled', ie. not being considered for future scheduling, because the job decided against it (job is Done, etc).

For a 'full' version of a tasklist report, use 'rush -tasklistfull'.

   rush -tasklistfull  
rush -tasklistfull [host|+hostgroup]

Task List Full Report.

A 'full report' version of -tasklist, where data is presented as "key:value" pairs that are easily parsed by scripts.

For more info, see -tasklist.

   rush -title  
rush -title <text> [jobid..]

Job Title

Sets title for current [specific] job. For more information, see Title.

   rush -trs  
rush -trs [csh|perl]

Template Render Script

Prints the template render script on stdout.

This can be redirected to a file, to quickly create a working script, which should then be customized using a text editor.

You can choose which scripting language your prefer; optional arguments 'csh' or 'perl' can be specified. If none are specified, 'perl' is the default.

     Create a Render Script 
    % rush -trs > render_me       # Save template as 'render_me'   
    % chmod +x render_me          # Make executable (unix)
    % vi render_me                # Edit the file to customize..

You will at least need to setup the render commands (the '### YOUR RENDER COMMAND(S) HERE' section) at minimum.

Administration Notes

    The systems administrator can customize the template render scripts by modifying the $RUSH_DIR/etc/templates file, which 'rush -trs' simply prints out.

   rush -try  
rush -try <newcount> <framerange..> [jobid..]

Set Try Count

Changes the 'Try' count for the frames specified.

A 'try count' is maintained for each frame in a job's frame list. This value is shown in the 'Try' column of 'rush -lf' reports, and is passed to render scripts via the $RUSH_TRY environment variable.

     Example: Changing 'Try' Counts 
    % rush -lf                     # List Frames to see try counts
    STAT FRAME TRY HOSTNAME       PID     START          ELAPSED  NOTES        
    Done 0001  4   superior       4144    10/04,03:29:10 00:00:18 
    Done 0002  4   tahoe          4160    10/04,03:29:30 00:00:16 
    Done 0003  4   placid         4163    10/04,03:29:47 00:00:16 
    Done 0004  6   huron          4168    10/04,03:30:04 00:00:16 
    Done 0005  4   finger         4171    10/04,03:30:21 00:00:16 
    
    % rush -try 0 1-5              # Reset Try count to zero
    0001: tries was 4, now 0
    0002: tries was 4, now 0
    0003: tries was 4, now 0
    0004: tries was 6, now 0
    0005: tries was 4, now 0
    
    % rush -lf                     # List Frames to see changes
    STAT FRAME TRY HOSTNAME       PID     START          ELAPSED  NOTES
    Done 0001  0   superior       4144    10/04,03:29:10 00:00:18 
    Done 0002  0   tahoe          4160    10/04,03:29:30 00:00:16 
    Done 0003  0   placid         4163    10/04,03:29:47 00:00:16 
    Done 0004  0   huron          4168    10/04,03:30:04 00:00:16 
    Done 0005  0   finger         4171    10/04,03:30:21 00:00:16
            

   rush -tss  
rush -tss [csh|perl]

Template Submit Script.

Prints the template submit script on stdout.

This can be redirected to a file, to quickly create a working script, which should then be customized using a text editor.

You can choose which scripting language your prefer; optional arguments 'csh' or 'perl' can be specified. If none are specified, 'perl' is the default.

     Create a Submit Script 
    % rush -tss > submit_me       # Save template as 'submit_me'   
    % chmod +x submit_me          # Make executable (unix)
    % vi submit_me                # Edit the file to customize..

You will, for instance, need to customize the command, title, cpus, and ram values at minimum.

Administration Notes

    The systems administrator can customize the template render scripts by modifying the $RUSH_DIR/etc/templates file, which 'rush -tss' simply prints out.

   rush -uping  
rush -uping [-c count] [-s secs] [remotehost..]

UDP Ping.

(Administrative) UDP Pings the local [remote] daemon. Useful for checking for UDP packet dropping. If [remotehost] is unspecified, the local host is assumed.

Options:

  • -c <count>

    Sets the number of udp transmissions sent to the remote, using the number of seconds (-s secs) between each transmission. If -c is unspecified, the default is 1.

    Zero can be specified for <count> to send udp transmissions indefinitely, or until you hit ^C.

  • -s <secs>

    Sets the number of seconds between transmissions. If -s is unspecified, the default is 5.

  • -b <bytes>

    Sets the number of bytes sent in each transmission. Use this to test the limits of UDP packet sizes. If -b is unspecified, the default is 32.

What follows are some example reports.

     'rush -uping' Example Report 
    
    % rush -uping -c 5 -s 1 tahoe
    >> UDP Packet Burst Response Test
    >> Hit ^C when done.
    
           tahoe: uping 1 236020307 (0.0026 secs roundtrip)  
           tahoe: uping 2 237020120 (0.0022 secs roundtrip)
           tahoe: uping 3 238020080 (0.0022 secs roundtrip)
           tahoe: uping 4 239020077 (0.0023 secs roundtrip)
           tahoe: uping 5 240019993 (0.0022 secs roundtrip)
            

The operation completes as soon as all hosts have responded. Machines that don't respond will show on stderr at the end of the report following a '*** NO RESPONSE FROM:' header, summarizing which machines dropped packets, and how many packets each dropped. Example:

     'rush -uping' Packet Drop Report 
    % rush -uping -c 3 -s 1 tahoe tower ontario ozark shasta
    >> UDP Packet Burst Response Test
    >> Hit ^C when done.
    
           tahoe: uping 1 4048620039 (0.0002 secs roundtrip, 32 bytes)  
           tower: uping 1 4048623461 (0.0005 secs roundtrip, 32 bytes)
           ozark: uping 1 4048631458 (0.0005 secs roundtrip, 32 bytes)
           tahoe: uping 2 4049643461 (0.0002 secs roundtrip, 32 bytes)
           tower: uping 2 4049647462 (0.0005 secs roundtrip, 32 bytes)
           ozark: uping 2 4049655459 (0.0004 secs roundtrip, 32 bytes)
          shasta: uping 2 4049655459 (0.0002 secs roundtrip, 32 bytes)
    
    *** NO RESPONSE FROM:
    ***          ontario: 2 sent, 0 received, 2 dropped.
    ***           shasta: 2 sent, 1 received, 1 dropped.
    
            

   rush -waitfor  
rush -waitfor [-|waitjobid|timeval,[..,..]] [jobid]

Wait For Other Jobs or Times

Sets the 'waitfor' jobids or times for the current [specified] job. See the submit script docs on WaitFor for more info.

'timeval' specifies a time to wait for. 'timeval' can be any of:

    WaitFor Time Values
    +8h 8 hours from now
    +60m 60 minutes from now
    +08:30:00 8 hours 30 minutes from now
    17:00 at 5pm
    5:30p at 5:30pm
    5:30pm at 5:30pm
    17:45,07/22/2002 at 5:45pm on 07/22/2002
    12:30p,07/22/2002 at 12:30pm on 07/22/2002

The 'WaitFor' time value will show up in 'rush -ljf' reports in two places; once in the 'WaitForTime:' field as an absolute time in MM/DD/YY,HH:MM:SS format, the other in the 'WaitFor:' field as a relative time in +HH:MM:SS format if the time has not yet expired.

For a tutorial on how to have one job wait for another, see Chaining Jobs.

Read the examples carefully to avoid confusing the [jobid] with the [waitjobids]. Remember to specify commas to separate the [waitjobids] instead of spaces. If only one [waitjobid] is being specified, add a trailing comma.

CAVEATS

  • Using 'rush -waitfor' clears all previous waitfor jobids and times

  • Setting 'waitfor' to multiple jobids AND times means ALL conditions must be met before job will start.

  • Specifying 'waitfor -' will disable the job from waiting for other jobs and times.

  • The jobs being waited for must be served by the same machine as the job doing the waiting. (i.e. the waitfor jobids and the job doing waiting must all be on the same jobserver)

  • Note the strange syntax where commas are mandatory when specifying the first argument after '-waitfor'. This prevents the parser from being confused as to which is the 'waitjobid' and which is the jobid to affect.

To differentiate between the new value of 'waitfor' and the jobid(s) to be affected, the new value of 'waitfor' must be a comma-separated list of jobids or times with no spaces. Examples:

    'rush -waitfor' Examples
    rush -waitfor - # Disable all waitfor's
    rush -waitfor tahoe.37,tahoe.38, # Job waits for 'tahoe.37' and 'tahoe.38' to complete ***
    rush tahoe.40 -waitfor +8h, # tahoe.40 won't start running until 8 hours from now
    rush tahoe.40 -waitfor tahoe.37,tahoe.38, # tahoe.40 waits for 'tahoe.37' & 'tahoe.38' to complete ***
    rush tahoe.40 -waitfor tahoe.37, # tahoe.40 now waits for 'tahoe.37' to complete ***
    # Trailing comma on tahoe.37 is important!
    rush tahoe.40 -waitfor 5:30p,tahoe.37,tahoe.38, # tahoe.40 won't start until 5:30pm and
    # until 'tahoe.37' and 'tahoe.38' complete ***

*** Depends on setting of 'rush -waitforstate'.

See Also

    WaitFor -- Submit command to make a job wait for a specific time or for other jobs
    WaitForState -- Submit command to set what job state we expect to see for other jobs
    DependOn -- Submit command to set jobid of other jobs whose frames we wait for
    rush -dependon -- Rush command line flag to change a job's existing dependon jobids

   rush -waitforstate  
rush -waitforstate <done|donefail|fail|dump> [jobid]

Controls circumstances that cause WaitFor to trigger.

See the submit script docs on WaitForState for more info.

Examples:

    'rush -waitforstate' Examples
    rush -waitforstate dump # Causes 'waitfor' to wait on other job(s) to Dump themselves (Default)
    rush -waitforstate fail # Causes 'waitfor' to wait on other job(s) to be Fail
    rush -waitforstate donefail # Causes 'waitfor' to wait on other job(s) to either be Fail or Done
    rush -waitforstate done # Causes 'waitfor' to wait on other job(s) to be Done

See Also

    WaitFor -- Submit command to make a job wait for a specific time or for other jobs
    WaitForState -- Submit command to set what job state we expect to see for other jobs
    DependOn -- Submit command to set jobid of other jobs whose frames we wait for
    rush -dependon -- Rush command line flag to change a job's existing dependon jobids

   Frame State Table  

This is the list of states a frame can be changed to, or searched for, eg. 'rush -done fail' changes all the Fail frames to 'Done'.

    Frame States
      que  
    Frame is in the Que state, ready to be rendered
    done
    Frame is in the Done state, it completed successfully.
    fail
    Frame is in the Fail state, it failed with an error.
    hold
    Frame is in the Hold state, it won't be rendered
    until it is changed to a different state.
    run
    Valid only for searching only.
    Matches all run states for frames.
    all
    Valid only for searching only.
    Can be used to match 'all' frames, eg: 'rush -done all'