From: "Schneider, Abraham" <aschneider@(email surpressed)>
Subject: weighted groups of machines?
   Date: Wed, 06 Nov 2013 11:14:40 -0500

Hi there!

I'm wondering if there is any similar feature in Rush like there is in other render management tools, where you can have 'weighted groups' of machines instead of the normal grouping?

For example: in Deadline you have normal groups like the hostgroups in Rush. In addition, you have 'pools'. You can assign a host to several pools like you can with groups, but the order in which you assign them matters. So you can assign some machines to pool1 as first priority and pool2 as second, and other machines vice versa. Now if you submit a job to pool1, all machines of pool1 will render this job. pool2 will render 'pool2' jobs, but if there are no more pool2 jobs, it will also render pool1 jobs. And vice versa. Really helpful to have 1 large renderfarm and split it between 2D and 3D or so, without loosing the ability to render on the whole farm when there are no jobs of a specific pool.

Qube has something similar called 'cluster' I believe.

How would I do something like this in Rush? I'd really like to combine our 2D and 3D farm, so the 2D farm would run 2D jobs during day time and helps rendering 3D jobs during the night (we don't have 2D jobs that are running all night long). And also be able to force urgent 2D jobs to use the 2D AND the 3D farm.

Thanks, Abraham


Abraham Schneider
Head of VFX pipeline / VFX Supervisor
 

Türkenstr. 89, 80799 München / 
Phone (Tel# suppressed) 

EMail aschneider@(email surpressed)

  Visit us on Facebook!________________________________


ARRI Film & TV Services GmbH
Sitz: München Registergericht: Amtsgericht München
Handelsregisternummer: HRB 69396
Geschäftsführer: Helge Jürgens, Josef Reidinger

On 11/06/13 08:14, Schneider, Abraham wrote:
> Hi there!
> 
> I'm wondering if there is any similar feature in Rush like there is in =
> other render management tools, where you can have 'weighted groups' of =
> machines instead of the normal grouping?
> 
> For example: in Deadline you have normal groups like the hostgroups in =
> Rush. In addition, you have 'pools'. You can assign a host to several =
> pools like you can with groups, but the order in which you assign them =
> matters. So you can assign some machines to pool1 as first priority and =
> pool2 as second, and other machines vice versa. Now if you submit a job =
> to pool1, all machines of pool1 will render this job. pool2 will render =
> 'pool2' jobs, but if there are no more pool2 jobs, it will also render =
> pool1 jobs. And vice versa. Really helpful to have 1 large renderfarm =
> and split it between 2D and 3D or so, without loosing the ability to =
> render on the whole farm when there are no jobs of a specific pool.
> 
> Qube has something similar called 'cluster' I believe.
> 
> How would I do something like this in Rush? 

Hi Abraham,

	I'd suggest using a combo of cpu priorities and the MinPri
	("Minumum Priority") field in the rush/etc/hosts file.

	Let's say you have two hostgroups named +2d and +3d.

	The 2d guys submit jobs: asking for +2d cpus at >=500 pri, +3d cpus at <100 pri.
	The 3d guys submit jobs: asking for +3d cpus at >=500 pri, +2d cpus at <100 pri.

	So, a literal example of a 2d job: Cpus: +2d=10@500 +3d=10@90
	and a literal example of a 3d job: Cpus: +3d=10@500 +2d=10@90

	This would give you simple priority weighting, but it wouldn't
	actually /block/ non-team renders from running during the day.

	Now to block the low pri renders (<100) from running at particular hours,
	set up a cron job that changes the MinPri field of the rush/etc/hosts file around:

		In the morning, set the MinPri to 100
		In the evening, set the MinPri to 0

	This way from morning till evening, 2d jobs can't run at all on 3d machines,
	and vice versa.

	As an example, on the license server you could make two copies of the
	rush/etc/hosts file:

		hosts.day which has MinPri set to 100
		hosts.night which has MinPri set to 0

	The license server's morning cron job might look something like:

		( cd /usr/local/rush/etc; cp hosts.day hosts.new; mv hosts.new hosts; rush -push hosts +any )

	..and the evening cron job might do something like:

		( cd /usr/local/rush/etc; cp hosts.night hosts.new; mv hosts.new hosts; rush -push hosts +any )

	The extra step in there of doing a cp/mv is to prevent the daemon from seeing
	a partial hosts file during the cp(1) command -- the mv(1) operation is atomic.

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/

Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

On 11/07/13 21:00, Greg Ercolano wrote:
> On 11/07/13 09:06, Greg Ercolano wrote:
>>          In the morning, set the MinPri to 100
>>          In the evening, set the MinPri to 0

	Sounds like I should add a way to specify timed hostgroups,
	both user and admin defined.

	I could see the hosts file defining things like:

+3dnight
{
    active	mon-fri@9p-8a, sat-sun=all		# times the group is active
}

+2dnight
{
    inactive	mon-fri@8:00a-9:00p			# another way of saying the above
}

	Perhaps too, a user might want to say they want to use their
	own machine, but only at certain hours:

		rush -ac localhost=2@100{!mon-fri@8a-6p}	# not during hours of 8am - 6pm on mon-fri

	Suggs welcome.