From: Greg Ercolano <erco@(email surpressed)>
Subject: [Q+A] Nuke cross platform pathnames
   Date: Tue, 22 Jun 2010 19:39:33 -0400

> When I submit a Nuke scene file created on a Mac, the artist
> has to change the path within nuke from /myserver/share to //myserver/share
> in order for the render to be successful on both Mac and Windows machines.
> We are using the FixPath() function in the rushscripts/.common.pl
> but that doesn't seem to affect pathnames within the Nuke scripts.
>
> Is there a way to configure Nuke to make these changes automatically?

	Yes; short answer is to see Nuke's own 'customizing' documentation
	on handling file paths across platforms.

	I'm almost sure the nuke documenation is wrong with respect to those
	backslashes, ie: "p:\" probably needs to be e.g. "p:\\".

	But you can probably use front slashes and avoid the whole 'backslash
	as an escape character' problem, which is typical in just about all languages
	including python, perl, C, Java, and all the shells.

Greg Ercolano wrote:
> 	Yes; short answer is to see Nuke's own 'customizing' documentation
> 	on handling file paths across platforms.

More specifically you will need to define a python function that takes a
filename as an argument a returns a new filename.  You then need to install it
as a FilenameFilter callback (nuke.addFilenameFilter) in your sites init.py file.

Robert Minsk wrote:
> [posted to rush.general]
> 
> Greg Ercolano wrote:
>> 	Yes; short answer is to see Nuke's own 'customizing' documentation
>> 	on handling file paths across platforms.
> 
> More specifically you will need to define a python function that takes a
> filename as an argument a returns a new filename.  You then need to install it
> as a FilenameFilter callback (nuke.addFilenameFilter) in your sites init.py file.

	Right; I believe the "Nuke-5.1v6-customizing.pdf" documentation cites
	this exact example [sic]:

----
def filenameFix(filename):
    if platform.system() in ("Windows", "Microsoft"):
        return filename.replace("/SharedDisk/", "p:\")
    else:
        return filename.replace("p:\", "/SharedDisk/")
    return filename
----

	..which appears to show how to replace e.g. "/SharedDisk/foo" <-> "p:\foo"
	depending on the platform it's running on.

	However, there's a few things wrong with that example I think; that backslash
	after the "p:" needs to be protected, otherwise python will interpret it as an
	escape character. Also, that final 'return filename' appears to be extraneous,
	as there's no way that line will ever execute.

	So I would suggest the following instead:

----
def filenameFix(filename):
    if platform.system() in ("Windows", "Microsoft"):
        return filename.replace("/SharedDisk/", "p:\\")
    else:
        return filename.replace("p:\\", "/SharedDisk/")
----

	Another way to protect the backslashes would be to use:

		r"p:\"			# same as "p:\\"

	This is pythons little way of protecting all characters between the
	double quotes, the leading 'r' in front of the leading double quote
	means "raw string", protecting the backslash from being mistaken
	for an escape sequence.

	An example that worked for a recent client who wanted to make sure
	UNC style pathnames were used on windows, (eg. "/bart/foo" would become
	"//bart/foo" on windows machines), I suggested they use:

----
import re
import platform
def filenameFix(filename):
    if platform.system() in ("Windows", "Microsoft"):
        return re.sub("^/*bart","//bart",filename)
    else:
        return re.sub("^/*bart","/bart",filename)
----

	..the special regex ensuring that e.g. //bart doesn't become ///bart


-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Greg Ercolano wrote:
> 	Another way to protect the backslashes would be to use:
> 
> 		r"p:\"			# same as "p:\\"

	Meh, that 'r' technique only works *if* the backslash is not at the end
	of the string. Since that *is* the case here, r"p:\" won't work..
	you'd *have* to use "p:\\".

	Of course, using front slashes bypasses all this.

	Front slashes work fine in this context, as the Windows kernel internally
	understands fronts (/) and backs (\) equally well in pathnames.

	It's only DOS commands, and some of Microsoft's browsers that have trouble
	with fronts, but that's due to the design of just those Microsoft applications,
	and not the OS itself.


-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Using "import platform" was preventing our Nukes from launching.

:)

I think it's deprecated now.

This code below worked for us (windows / mac).

<snip>

import os



def filenameFix(filename):

	if os.name == "nt":

		return filename.replace( "/Volumes/Filer/", "f:\\" )

	return filename.replace( "f:\\", "/Volumes/Filer/" )



<snip>

Mat X wrote:
> Using "import platform" was preventing our Nukes from launching. :)
> I think it's deprecated now.

	Oh nice.

	I suppose you could inspect the os.environment["OS"] variable;
	that's been reliable enough (so far), and doesn't require an import.

	The OS environment variable is set to "Windows_NT" on all versions
	of windows since NT (NT through Win7).

	Apparently we've never actually gotten away from "NT Technology" ;)

> This code below worked for us (windows / mac).
> 
> <snip>
> import os
> def filenameFix(filename):
> 	if os.name == "nt":
> 		return filename.replace( "/Volumes/Filer/", "f:\\" )
> 	return filename.replace( "f:\\", "/Volumes/Filer/" )
> <snip>

	Great -- so to summarize, I think; modify the nuke init.py
	to contain e.g.:

----
import os
# Local filename modifications
def filenameFix(filename):
    if os.name == "nt":
	return filename.replace( "/Volumes/Filer/", "f:\\" )
    else:
	return filename.replace( "f:\\", "/Volumes/Filer/" )

nuke.addFilenameFilter(filenameFix)
----

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Does anyone know of a way to restrict the maximum number of cpus that a user may have running jobs at any given time?

----------
Dan "Doc" Browne
System Administrator
Evil Eye Pictures

dbrowne@(email surpressed)
Office: (415) 777-0666 x105

Daniel Browne wrote:
> [posted to rush.general]
> 
> Does anyone know of a way to restrict the maximum number of cpus that a use=
> r may have running jobs at any given time?

	Do you mean 'rush cpus' (ie. how many instances of rush renders),
	or 'physical cpus'?

	Rush cpus can be controlled in several ways; the 'CPUS' column in the
	rush/etc/hosts file, the Ram: value when a user submits, 'rush -reserve'.

	Physical cpus can be limited on windows machines using "processor affinity"
	in the rush/etc/hosts file, eg:
	http://seriss.com/cgi-bin/rush/newsgroup-threaded.cgi?-view+1899+1898+1899+1900+1905
	The 102.42a9c version of rush supports this for windows only,
	the next release of rush will also support it for linux.
	AFAIK, OSX's kernel does not yet support processor affinity.


-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Rush CPU's, yes; I'd like make it so that no matter how many jobs an artist has sitting in the queue, he or she can utilize only a maximum of say 10 render machines at any given time.


On Feb 14, 2011, at 7:41 PM, Greg Ercolano wrote:

[posted to rush.general]

Daniel Browne wrote:
> [posted to rush.general]
> 
> Does anyone know of a way to restrict the maximum number of cpus that a use=
> r may have running jobs at any given time?

	Do you mean 'rush cpus' (ie. how many instances of rush renders),
	or 'physical cpus'?

	Rush cpus can be controlled in several ways; the 'CPUS' column in the
	rush/etc/hosts file, the Ram: value when a user submits, 'rush -reserve'.

	Physical cpus can be limited on windows machines using "processor affinity"
	in the rush/etc/hosts file, eg:
	http://seriss.com/cgi-bin/rush/newsgroup-threaded.cgi?-view+1899+1898+1899+1900+1905
	The 102.42a9c version of rush supports this for windows only,
	the next release of rush will also support it for linux.
	AFAIK, OSX's kernel does not yet support processor affinity.


-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

----------
Dan "Doc" Browne
System Administrator
Evil Eye Pictures

dbrowne@(email surpressed)
Office: (415) 777-0666 x105

Daniel Browne wrote:
> [posted to rush.general]
> 
> Rush CPU's, yes; I'd like make it so that no matter how many jobs an artist=
>  has sitting in the queue, he or she can utilize only a maximum of say 10 r=
> ender machines at any given time.

	There's a few ways;

	1) When they submit the job, their "Cpus:" setting controls
	how many cpus the job gets. So in the simple case:

		+any=10@100

	..they'll only get 10 frames rendering in parallel max.

	But if they make several cpu specifications, like:

		+any=10@100 +any=5@900

	..then in that case one can get a total of 15.

	If you want to cap the /entire job/ at 10, then..


	2) Set a 'maxcpus' for the job to be 10. This will cap the entire job
	   so that even if they keep adding cpus, no more than 10 will run at a time.

	   Either the user can set this themselves at the "Submit Options:" prompt
	   (under the "Rush" tab), eg:

			Submit Options: maxcpus 10

	   ..and that would cap the job's cpus to no more than 10.
	   In the "Cpus" report of irush, you'll see messages like "Maxcpus reached 10/10"
	   when cpus get turned away due to the cap.

	   If you want to 'invisibly' force this, you can jam that command
	   into (for instance) the submit-maya.pl script by finding this section
	   of the script:

-------------------------------------------------------------------------------
    my $submit = <<"EOF";
title           $in{JobTitle}
frames          $newframespec
command         perl $G::self -render $batchframes $batchstart $batchend ...
$submitoptions
EOF
-------------------------------------------------------------------------------

	..and adding a line "maxcpus 10", eg:

-------------------------------------------------------------------------------
    my $submit = <<"EOF";
title           $in{JobTitle}
frames          $newframespec
command         perl $G::self -render $batchframes $batchstart $batchend ...
$submitoptions
maxcpus         10			# <-- ADD THIS LINE
EOF
-------------------------------------------------------------------------------

	..that will force the job to have a maxcpus of 10,
	regardless of what the user specifies.

	Note that this can be changed after the job has been submitted
	inside irush using Job Edit -> Job Info -> MaxCpus:

	I advise making it a 'policy' not to change this, and let
	peer pressure/producer pressure take it from there.

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

No no, Greg; I'm trying to restrict per user, not per job.


On Feb 15, 2011, at 2:07 PM, Greg Ercolano wrote:

[posted to rush.general]

Daniel Browne wrote:
> [posted to rush.general]
> 
> Rush CPU's, yes; I'd like make it so that no matter how many jobs an artist=
> has sitting in the queue, he or she can utilize only a maximum of say 10 r=
> ender machines at any given time.

	There's a few ways;

	1) When they submit the job, their "Cpus:" setting controls
	how many cpus the job gets. So in the simple case:

		+any=10@100

	..they'll only get 10 frames rendering in parallel max.

	But if they make several cpu specifications, like:

		+any=10@100 +any=5@900

	..then in that case one can get a total of 15.

	If you want to cap the /entire job/ at 10, then..


	2) Set a 'maxcpus' for the job to be 10. This will cap the entire job
	   so that even if they keep adding cpus, no more than 10 will run at a time.

	   Either the user can set this themselves at the "Submit Options:" prompt
	   (under the "Rush" tab), eg:

			Submit Options: maxcpus 10

	   ..and that would cap the job's cpus to no more than 10.
	   In the "Cpus" report of irush, you'll see messages like "Maxcpus reached 10/10"
	   when cpus get turned away due to the cap.

	   If you want to 'invisibly' force this, you can jam that command
	   into (for instance) the submit-maya.pl script by finding this section
	   of the script:

-------------------------------------------------------------------------------
   my $submit = <<"EOF";
title           $in{JobTitle}
frames          $newframespec
command         perl $G::self -render $batchframes $batchstart $batchend ..

Daniel Browne wrote:
> No no, Greg; I'm trying to restrict per user, not per job.

	Ah, per user across the farm? No, it's not a built in feature,
	but can be done with a 'watcher' script that polls the farm
	and plays with user's 'maxcpus' settings to get the behavior
	you're looking for.

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)ext.23
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)