From: Greg Ercolano <erco@(email surpressed)>
Subject: [Q+A] Handling of quoting in render commands
   Date: Wed, 11 May 2011 12:38:36 -0400
Msg# 2099
View Complete Thread (2 articles) | All Threads
Last Next
> What's the syntax of quoting for the render command?
> (in other words, the "command" passed to 'rush -submit')
>
> We'd like to invoke commands whose arguments contain spaces.
> We're on windows.

    There are a few ways.

    IMHO, the most portable is to convert all spaces in arguments
    into the ascii DEL character before submit, and then have the
    render script convert them back.

    So for instance on submit, say we want to run:

	perl /path/to/myscript.pl -render "one two" "three four five"

    ..on the farm. I'd suggest doing it this way:

---- snip
    $arg1 = "one two";
    $arg2 = "three four five";

    # Convert arg's spaces into DEL
    $arg1 =~ s/ /\x7f/g;
    $arg2 =~ s/ /\x7f/g;

    # Build command
    $command = "perl /path/to/myscript.pl -render $arg1 $arg2";

    open(SUBMIT, "|rush -submit");
    print SUBMIT << "EOF";
        :
        command $command
        :
---- snip

    On submit, the spaces in the arguments will be the DEL character,
    and therefore won't be mistaken as argument delimiters.

    Then, when the render script runs on the farm, it first converts
    all the DEL's in the arguments back into spaces before using them, eg:

        for ( $t=0; $t<=$#ARGV; $t++ ) {
            $ARGV[$t] =~ s/\x7f/ /g;
        }

    This will get the arguments back the way they were, with spaces intact.



    There is another way, but it's Microsoft specific, and is something
    that Microsoft has not well documented, so I can't tell you much
    other than what my empirical tests show.

    Under Windows specifically, you can use double quotes (") to protect
    strings that contain spaces in the render commands, and the escape
    character (\) can be used to protect itself and quotes.

    So for example, if you were to submit the following:

title     QUOTE_TEST
command   "//some/path with spaces/myrender.exe" "one two" "three four"
frames    1-10
cpus      +any=5
logdir    //some/emptydir

    ..then it would run on the farm as you'd expect, and with
    argv[1] set to "one two",
    argv[2] set to "three four"

    If you find you need to embed backslashes or quotes,
    you can escape them with the backslash character, eg:

command "//some/path/myrender.exe" "This is a quote(\")" "This is a backslash(\\)"

    This behavior is handled by the WIN32 call CreateProcess(),
    which is documented here (*):
    http://msdn.microsoft.com/en-us/library/ms682425%28v=vs.85%29.aspx

    As of this writing (May 2011), IMHO the above docs don't go far enough
    into enough detail on quoting rules. For instance, what characters does
    the backslash escape? Certainly not all, since backslashes can appear
    in pathnames and not get confused. Also, UNC paths start with double
    backslashes, and in that case \\ doesn't devolve into \, so what's the
    rule there?

    Anyway, my empirical tests show the above works, which is more or less
    consistent with unix shell handling in cases where the character after
    the backslash is a quote or another backslash. But obviously there are
    differences, since in unix backslash escapes ALL characters that follow it,
    whereas in Windows, it doesn't always.

    This is why the rush scripts prefer the 'convert everything to DEL'
    and to use frontslashes instead of backslashes to avoid running into
    weird cross platform parsing differences.

(*) That link may go stale; Microsoft likes to move stuff around.
    If it does, just search MSDN for CreateProcess().

   From: Greg Ercolano <erco@(email surpressed)>
Subject: Re: [Q+A] Handling of quoting in render commands
   Date: Wed, 11 May 2011 13:32:19 -0400
Msg# 2100
View Complete Thread (2 articles) | All Threads
Last Next
Greg Ercolano wrote:
> command   "//some/path with spaces/myrender.exe" "one two" "three four"
> [..]
>     ..then it would run on the farm as you'd expect, and with
>     argv[1] set to "one two",
>     argv[2] set to "three four"
> 
>     If you find you need to embed backslashes or quotes,
>     you can escape them with the backslash character, eg:
> 
> command "//some/path/myrender.exe" "This is a quote(\")" "This is a backslash(\\)"
> 
>     This behavior is handled by the WIN32 call CreateProcess(),
>     which is documented here (*):
>     http://msdn.microsoft.com/en-us/library/ms682425%28v=vs.85%29.aspx
  
   Actually, it seems to be more complicated than that.
   This complexity is not my fault! ;) It's how Microsoft works.

   CreateProcess() handles quotes around the command only,
   and the Microsoft C library handles quotes around arguments
   with its own separate rules for quoting and escaping.

   So in other words:

      "//some/path/myrender.exe" "This is a quote(\")" "This is a backslash(\\)"
      \________________________/ \_____________________________________________/
         CreateProcess()                     Microsoft C Library
         quoting rules.                        quoting rules.

   The CreateProcess() rules for quoting are relatively simple (*):
   http://msdn.microsoft.com/en-us/library/ms682425%28v=vs.85%29.aspx

   The Microsoft C Library quoting rules for quoting are more complex (**):
   http://msdn.microsoft.com/en-us/library/a1y7w461.aspx

   The CreateProcess() docs simply say double quotes can be used to protect
   spaces in the command, nothing about backslash quoting, or escaping quotes.

   The Microsoft C Library handles argument quoting at the time the
   command is invoked, and since just about everything you run is a
   C/C++ program (perl, python, cmd, maya, nuke, etc), those rules are
   followed for argument parsing.

   I'm quoting their docs verbatim here, as this directly affects
   how the Rush render command's arguments are interpreted on windows machines:

--- quote

    * Arguments are delimited by white space, which is either a space or a tab.

    * A string surrounded by double quotation marks is interpreted as a single
      argument, regardless of white space contained within. A quoted string can
      be embedded in an argument. Note that the caret (^) is not recognized as an
      escape character or delimiter.

    * A double quotation mark preceded by a backslash, \", is interpreted as a
      literal double quotation mark (").

    * Backslashes are interpreted literally, unless they immediately precede
      a double quotation mark.

    * If an even number of backslashes is followed by a double quotation mark,
      then one backslash (\) is placed in the argv array for every pair of
      backslashes (\\), and the double quotation mark (") is interpreted as a
      string delimiter.

    * If an odd number of backslashes is followed by a double quotation mark,
      then one backslash (\) is placed in the argv array for every pair of
      backslashes (\\) and the double quotation mark is interpreted as an
      escape sequence by the remaining backslash, causing a literal double
      quotation mark (") to be placed in argv.

--- snip

	My brain imploded somewhere on that fifth or sixth paragraph,
        but hopefully some of you can benefit from this. YMMV.

	If you can't get the above working, your best bet is to just
	empirically try escaping variations until it works.

        Or, /avoid/ the problem completely and don't specify complex info
	on the command line; throw your arguments into a file, and pass the
	file as the only argument to your render script, so it can load the
	file and get the data unmolested by the above parsing rules.


(*)  If the "CreateProcess" link goes stale, search MSDN for CreateProcess.

(**) If the "Microsoft C Library Parsing" link goes stale, search MSDN
     for "Parsing C Command-Line Arguments".