From: Dylan Penhale <dylan@(email surpressed)>
Subject: Time Based Host Groups
   Date: Sun, 23 Jan 2011 20:49:42 -0500

We currently split our render groups and criteria according to machine specs, software and department. We would like to be able to reserve a pool selection from 9-00 till 22-00 and then open that pool to the rest of the farm out of those hours. 

e.g. Comp renders are fast, we don't want 3D renders on those machines during the day. However at night comp isn't (always) there so those machines could be used by 3D. Currently the only way I can think to do this is to:

1) Play about with priorities - this has proved to be unsatisfactory because 3D renders can take so long.
2) Push out the hosts files as part of a cron - I've not tried this but imagine it wouldn't be robust.
3) Employ a render wrangler. 

Any suggestions?

Dylan Penhale
CTO
Fuel VFX
Sydney

The trick is to put host group assignment into the jobstartcommand script.


On Jan 23, 2011, at 5:49 PM, Dylan Penhale wrote:

> [posted to rush.general]
> 
> We currently split our render groups and criteria according to machine spec=
> s, software and department. We would like to be able to reserve a pool sele=
> ction from 9-00 till 22-00 and then open that pool to the rest of the farm =
> out of those hours.=20
> 
> e.g. Comp renders are fast, we don't want 3D renders on those machines duri=
> ng the day. However at night comp isn't (always) there so those machines co=
> uld be used by 3D. Currently the only way I can think to do this is to:
> 
> 1) Play about with priorities - this has proved to be unsatisfactory becaus=
> e 3D renders can take so long.
> 2) Push out the hosts files as part of a cron - I've not tried this but ima=
> gine it wouldn't be robust.
> 3) Employ a render wrangler.=20
> 
> Any suggestions?
> 
> Dylan Penhale
> CTO
> Fuel VFX
> Sydney
> 
> 
> 
> 
> 
> 

We have been using option #2 for a while now, We started with a cron pushing a updated Day or Night host file, but now we do it though puppet. We generally run the change from day to night around an hour or so before the normal end of the day, so lighters can submit jobs before the end of the day and they will get the changes. Then the change from night to day about an hour before people come in so any jobs that are running have some time to finnish. Its not fullproof, but in most cases its worked for us. 

At some point I need to investigate how to add the machines that are dedicated during the day to the greater pool at night. I just haven't had the time to look into it.

BTW, congrats Greg on the Academy award!

-Brent

On Jan 23, 2011, at 5:49 PM, Dylan Penhale wrote:

> [posted to rush.general]
> 
> We currently split our render groups and criteria according to machine spec=
> s, software and department. We would like to be able to reserve a pool sele=
> ction from 9-00 till 22-00 and then open that pool to the rest of the farm =
> out of those hours.=20
> 
> e.g. Comp renders are fast, we don't want 3D renders on those machines duri=
> ng the day. However at night comp isn't (always) there so those machines co=
> uld be used by 3D. Currently the only way I can think to do this is to:
> 
> 1) Play about with priorities - this has proved to be unsatisfactory becaus=
> e 3D renders can take so long.
> 2) Push out the hosts files as part of a cron - I've not tried this but ima=
> gine it wouldn't be robust.
> 3) Employ a render wrangler.=20
> 
> Any suggestions?
> 
> Dylan Penhale
> CTO
> Fuel VFX
> Sydney
> 
> 
> 
> 
> 
> 

Dylan Penhale wrote:
> We currently split our render groups and criteria according to machine spec=
> s, software and department. We would like to be able to reserve a pool sele=
> ction from 9-00 till 22-00 and then open that pool to the rest of the farm =
> out of those hours.=20
> 
> e.g. Comp renders are fast, we don't want 3D renders on those machines duri=
> ng the day. However at night comp isn't (always) there so those machines co=
> uld be used by 3D. Currently the only way I can think to do this is to:

	I see, so you have 'day' machines that should only
	run comps during the day, but open up to all jobs at night?

	And since the 3d jobs are long, you probably want them to
	be killed at 9am so that they don't continue to run for
	what could be hours.

	So doing something like online/offline'ing the processors
	at 10pm/9am via cron wouldn't be good, because it would affect
	comps AND 3d jobs, eg:

# min hour  dom	mon dow       command
0     9     *   *   1,2,3,4,5 /usr/local/rush/bin/rush -getoff +resv_day
0     22    *   *   1,2,3,4,5 /usr/local/rush/bin/rush -online +resv_day

	So instead of that..

	I'd say the easy thing to do would be to leave the rush/etc/hosts
	file alone, and just write a script that runs on a single machine
	at 9am and 10pm that alternately:

		1) Adds the "+day" group to all the 3d jobs at 10pm
		2) Removes the "+day" group from the 3d jobs at 9am

	By removing them, that will kill them immediately.

	The jobs likely have enough info in them that the script
	can use to determine 3d jobs from comps, eg. the job title,
	job notes, or if nothing else, each job's 'command' (by
	checking for either 'submit-maya' or 'submit-nuke', etc)

	For instance, if it's maya jobs you're worried about,
	you can walk all the jobs looking at the 'Command:'
	(from the 'rush -ljf +any -t 5' report) checking for
	the word 'submit-maya, and for all that are found:

		1) At 9am use 'rush -rc +day <JOBIDS>' to remove them,
		   (first saving the +day cpu values into the job's
		   notes field, so they can later be added back at 10pm)

		2) At 10pm use 'rush -ac <SPEC> <JOBIDS>' to add them back.
		   (<SPEC> would be info saved in the job notes at 9am)

	I think something like that would work well, and would take
	immediate effect.

> 2) Push out the hosts files as part of a cron - I've not tried this but ima=
> gine it wouldn't be robust.

	Probably not a good solution, since hostgroup memberships are
	expanded at the time the job is submitted, so changing them
	back and forth are unlikely to be useful, unless the jobs
	render in less time than a full shift (ie. a job running 24 hours).

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Thanks Guys

Greg I like that sound of that script, it certainly sounds like the
simplest method, I didn't think of doing it that way. I'll report back
when I've had a chance to play.

Thanks again.
Dylan



On 24 January 2011 15:44, Greg Ercolano <erco@(email surpressed)> wrote:
> [posted to rush.general]
>
> Dylan Penhale wrote:
>> We currently split our render groups and criteria according to machine spec=
>> s, software and department. We would like to be able to reserve a pool sele=
>> ction from 9-00 till 22-00 and then open that pool to the rest of the farm =
>> out of those hours.=20
>>
>> e.g. Comp renders are fast, we don't want 3D renders on those machines duri=
>> ng the day. However at night comp isn't (always) there so those machines co=
>> uld be used by 3D. Currently the only way I can think to do this is to:
>
>        I see, so you have 'day' machines that should only
>        run comps during the day, but open up to all jobs at night?
>
>        And since the 3d jobs are long, you probably want them to
>        be killed at 9am so that they don't continue to run for
>        what could be hours.
>
>        So doing something like online/offline'ing the processors
>        at 10pm/9am via cron wouldn't be good, because it would affect
>        comps AND 3d jobs, eg:
>
> # min hour  dom mon dow       command
> 0     9     *   *   1,2,3,4,5 /usr/local/rush/bin/rush -getoff +resv_day
> 0     22    *   *   1,2,3,4,5 /usr/local/rush/bin/rush -online +resv_day
>
>        So instead of that..
>
>        I'd say the easy thing to do would be to leave the rush/etc/hosts
>        file alone, and just write a script that runs on a single machine
>        at 9am and 10pm that alternately:
>
>                1) Adds the "+day" group to all the 3d jobs at 10pm
>                2) Removes the "+day" group from the 3d jobs at 9am
>
>        By removing them, that will kill them immediately.
>
>        The jobs likely have enough info in them that the script
>        can use to determine 3d jobs from comps, eg. the job title,
>        job notes, or if nothing else, each job's 'command' (by
>        checking for either 'submit-maya' or 'submit-nuke', etc)
>
>        For instance, if it's maya jobs you're worried about,
>        you can walk all the jobs looking at the 'Command:'
>        (from the 'rush -ljf +any -t 5' report) checking for
>        the word 'submit-maya, and for all that are found:
>
>                1) At 9am use 'rush -rc +day <JOBIDS>' to remove them,
>                   (first saving the +day cpu values into the job's
>                   notes field, so they can later be added back at 10pm)
>
>                2) At 10pm use 'rush -ac <SPEC> <JOBIDS>' to add them back.
>                   (<SPEC> would be info saved in the job notes at 9am)
>
>        I think something like that would work well, and would take
>        immediate effect.
>
>> 2) Push out the hosts files as part of a cron - I've not tried this but ima=
>> gine it wouldn't be robust.
>
>        Probably not a good solution, since hostgroup memberships are
>        expanded at the time the job is submitted, so changing them
>        back and forth are unlikely to be useful, unless the jobs
>        render in less time than a full shift (ie. a job running 24 hours).
>
> --
> Greg Ercolano, erco@(email surpressed)
> Seriss Corporation
> Rush Render Queue, http://seriss.com/rush/
> Tel: +1 626-576-0010 ext.23
> Fax: +1 626-576-0020
> Cel: +1 310-266-8906
>



-- 
Dylan Penhale