From: Antoine Durr <antoine@(email surpressed)>
Subject: semi dynamic hostgroups/criteria hosts file
   Date: Tue, 29 May 2007 15:13:00 -0400

Hi,

Right now, I have:

#Host           Cpus    Ram     MinPri  Criteria/Hostgroups
#-------------  -----   -----   ------  -------------------
rushmaster      0       1024    0       +any,+linux,+server

# workstations
f1              4       8192    0       +any,+linux,+desktop
f2              4       4096    0       +any,+linux,+desktop
f3              4       4096    0       +any,+linux,+desktop

# servers
fgs2            2       2048    0       +any,+linux,+server

-- Antoine

Floq FX Inc.
10839 Washington Blvd.
Culver City, CA 90232
310/430-2473

Antoine Durr wrote:
> Is there a way to have the set of hostgroups to which a host belongs 
> be configured from a program, rather than entered statically into the 
> Rush hosts file?

	No, rush only gets its hostgroup info from the rush/etc/hosts file.

	Rush parses and caches this file, and only reloads it when it
	sees the datestamp change, or when 'rush -reload hosts' is used
	to force a reload immediately.

	There are no hooks where rush can 'run a program' to get this
	info. That could maybe be arranged, just for the hostgroup info,
	but I didn't want to go there, because future modifications might
	involve accessing the hostgroup information fairly often, if it
	were to ever be changed to being dynamic.

	So it will probably always be a file that is 'cached', updated
	only when changed or 'poked' with a reload, to keep things from
	bottlenecking.

	In a very old render queue design, I tried to implement scheduling
	and host grouping as a script, and I very quickly found that
	the script was too much overhead (too slow) to manage the high
	turnaround needed in high volume environments, so I stopped
	researching that idea pretty quickly.

> I have a standalone program (called 'hostgroups', appropriately 
> enought!) that if I give it 'fgs2' as a parameter, returns a list that 
> looks similar to '+any,+linux,+server'.  How do I wedge my program into 
> this list here or onto what Rush things are the host groups?

	If you're trying to make jobs be able to change memberships
	on the fly, I would attack it differently, where you have
	a 'watcher script' that changes jobs on the fly, bumping
	around the cpu cap values.

	eg. if you know your job will need to be in the +a, +b and +c
	groups, submit the job with:

cpus +a=0 +b=0 +c=0

	..and then let the 'watcher script' bump the =0 values (cpu caps)
	up and down with the 'rush -cp' command, eg:

rush -cp +a=0 +a=3 somejob.123

	Rush is designed so you can do that on the fly without causing
	too much effort on the scheduler. It also allows you to decrease
	cpu caps without killing rendering frames, as opposed to actually
	REMOVING cpus (with 'rush -rc') which /will/ kill frames. I would
	not advise using -rc/-ac on the fly either; it creates a lot of
	effort to the daemon on large networks. -cp is better for batting
	priorities and cpu limits around.

Greg Ercolano wrote:
> 	..and then let the 'watcher script' bump the =0 values (cpu caps)
> 	up and down with the 'rush -cp' command, eg:
> 
> rush -cp +a=0 +a=3 somejob.123

	Oops, that should have been:

		rush -cp +a=0@1 =3@1 somejob.123
                         ^^^^^^ ^^^^
                         Old    New

	..the priorities need to be specified too, because there can be multiple
	priority groups. (Your polling script would first get the 'old' cpu/pri
	info from the 'rush -lc' report)

	For more info on the -cp (change priority and cpu cap) command, see:
	http://www.seriss.com/rush-current/rush/rush-command-line-options.html#-cp

-- 
Greg Ercolano, erco@(email surpressed)
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

On 2007-05-30 11:35:22 -0700, Greg Ercolano <erco@(email surpressed)> said:

Antoine Durr wrote:

Is there a way to have the set of hostgroups to which a host belongs
be configured from a program, rather than entered statically into the
Rush hosts file?

	No, rush only gets its hostgroup info from the rush/etc/hosts file.

	Rush parses and caches this file, and only reloads it when it
	sees the datestamp change, or when 'rush -reload hosts' is used
	to force a reload immediately.

	There are no hooks where rush can 'run a program' to get this
	info. That could maybe be arranged, just for the hostgroup info,
	but I didn't want to go there, because future modifications might
	involve accessing the hostgroup information fairly often, if it
	were to ever be changed to being dynamic.

	So it will probably always be a file that is 'cached', updated
	only when changed or 'poked' with a reload, to keep things from
	bottlenecking.

	In a very old render queue design, I tried to implement scheduling
	and host grouping as a script, and I very quickly found that
	the script was too much overhead (too slow) to manage the high
	turnaround needed in high volume environments, so I stopped
	researching that idea pretty quickly.

I have a standalone program (called 'hostgroups', appropriately
enought!) that if I give it 'fgs2' as a parameter, returns a list that
looks similar to '+any,+linux,+server'.  How do I wedge my program into
this list here or onto what Rush things are the host groups?

-- Antoine

Floq FX Inc.
10839 Washington Blvd.
Culver City, CA 90232
310/430-2473

Antoine Durr wrote:
> I don't need fast dynamic updating.  I just wanted to avoid having the 
> same lists (or similar, but eventually diverging lists) in two 
> different places.  I think the solution is to have a cron job 
> regenerate the rush hosts list once per day..

	Oh, I see.. yes, you can do that with a cron job,
	rdisting out the change to the rest of the machines.

	Although not intended for 'unattended' use, you can even
	use 'rush -push hosts +any'; just be sure to watch stderr
	for 'NO RESPONSE FROM' errors, in case some remotes are too
	busy to respond, and to retry. (Rdist'ing is better)

	You won't need to use rush -reload; the daemons will pick
	up the change automatically within 30 secs. (The delay
	is to prevent hitting DNS all at once, esp for large sites)

	rdist is safer than rcp, because there's no chance of rush
	reading a partial file while it is in transit, due to how
	rdist copies to a temp file, then renames it into place
	as an atomic operation.

> How frequently does Rush check the file for a datestamp change?  On 
> every queue submit?

	Rush checks every 30 secs for date stamp changes on the
	rush/etc/rush.conf and rush/etc/hosts file, and reloads
	only when it changes.

	So even if you rdist the files around with no rush commands
	involved, it will get the change within 30 secs.

-- 
Greg Ercolano, erco@(email surpressed)
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

On 2007-05-30 13:23:02 -0700, Greg Ercolano <erco@(email surpressed)> said:

Antoine Durr wrote:

I don't need fast dynamic updating.  I just wanted to avoid having the
same lists (or similar, but eventually diverging lists) in two
different places.  I think the solution is to have a cron job
regenerate the rush hosts list once per day..

	Oh, I see.. yes, you can do that with a cron job,
	rdisting out the change to the rest of the machines.

	Although not intended for 'unattended' use, you can even
	use 'rush -push hosts +any'; just be sure to watch stderr
	for 'NO RESPONSE FROM' errors, in case some remotes are too
	busy to respond, and to retry. (Rdist'ing is better)

	You won't need to use rush -reload; the daemons will pick
	up the change automatically within 30 secs. (The delay
	is to prevent hitting DNS all at once, esp for large sites)

	rdist is safer than rcp, because there's no chance of rush
	reading a partial file while it is in transit, due to how
	rdist copies to a temp file, then renames it into place
	as an atomic operation.

-- Antoine

Floq FX Inc.
10839 Washington Blvd.
Culver City, CA 90232
310/430-2473

Antoine Durr wrote:
> Hmmm, I hadn't thought of the problem that I need to rdist the file.  I 
> optimistically thought that I'd only need to do it in one place.  Do I 
> need to have *all* the machines have the same info at the same time, or 
> is it sufficient for the submitting machine to have the latest hosts 
> file?  

	I wouldn't recommend having a mix across the network;
	the rush/etc/* files should all be in sync.

	The 'rush -push' command makes this easy; you can update
	eg. the rush/etc/hosts file to the entire network with
	a single command: 'rush -push hosts +any', but its mainly
	meant as an 'interactive use' command.

> Maybe I could have the submit script rewrite it on the fly 
> (which is really the only time I need it updated).  But that wouldn't 
> work if all hosts need to be synchronized.

	Definitely keep them synchronized; the daemons will get confused
	if each machine has different hosts file data.

	That said, there are some exceptions where it can be desireable
	to have different contents, but that's rare. As a general rule,
	keep all files in the rush/etc/* directory in sync; whenever you
	change it on one box, copy it out to the rest with either 'rush -push',
	or rdist, or whatever you like (as long as the copy is done atomically)

-- 
Greg Ercolano, erco@(email surpressed)
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)