From: Antoine Durr <antoine@(email surpressed)> Subject: redirects and semicolons in jobdumpcommand Date: Sat, 01 Dec 2007 05:31:39 -0500 |
Msg# 1654 View Complete Thread (5 articles) | All Threads Last Next |
Hi, I tried the following in a jobdumpcommand, to no avail: printenv > /home/antoine/foo Is there a reason that does not work? I also tried: printenv ; sleep 60and while the printenv did list into the default .log file correctly, the sleep command was skipped. Is this intentional? Thanks, -- Antoine Floq FX Inc. 10659 Cranks Rd. Culver City, CA 90230 310/430-2473 |
From: Greg Ercolano <erco@(email surpressed)> Subject: Re: redirects and semicolons in jobdumpcommand Date: Sat, 01 Dec 2007 06:18:38 -0500 |
Msg# 1655 View Complete Thread (5 articles) | All Threads Last Next |
Antoine Durr wrote: > [posted to rush.general] > > Hi, > I tried the following in a jobdumpcommand, to no avail: > > printenv > /home/antoine/foo > > Is there a reason that does not work? I also tried: > > printenv ; sleep 60 The command is invoked by exec(), so no shell is involved. (';' and '>' being shell characters) To use shell specific syntax, it's best if you put your commands into a shell script and invoke that. -- Greg Ercolano, erco@(email surpressed) Rush Render Queue, http://seriss.com/rush/ Tel: (Tel# suppressed) Fax: (Tel# suppressed) Cel: (Tel# suppressed) |
From: Antoine Durr <antoine@(email surpressed)> Subject: Re: redirects and semicolons in jobdumpcommand Date: Sat, 01 Dec 2007 13:13:52 -0500 |
Msg# 1656 View Complete Thread (5 articles) | All Threads Last Next |
On 2007-12-01 03:18:38 -0800, Greg Ercolano <erco@(email surpressed)> said: Antoine Durr wrote:[posted to rush.general] Hi, I tried the following in a jobdumpcommand, to no avail: printenv > /home/antoine/foo Is there a reason that does not work? I also tried: printenv ; sleep 60The command is invoked by exec(), so no shell is involved. (';' and '>' being shell characters) To use shell specific syntax, it's best if you put your commands into a shell script and invoke that. Ahh, ok. Out of curiosity, why not just do a system() call, with shell and all? From the user's standpoint, it's not obvious that we cannot put arbitrary unix commands into those fields. Thanks, -- Antoine Floq FX Inc. 10659 Cranks Rd. Culver City, CA 90230 310/430-2473 |
From: Greg Ercolano <erco@(email surpressed)> Subject: Re: redirects and semicolons in jobdumpcommand Date: Sat, 01 Dec 2007 14:31:13 -0500 |
Msg# 1657 View Complete Thread (5 articles) | All Threads Last Next |
Antoine Durr wrote: >> The command is invoked by exec(), so no shell is involved. >> (';' and '>' being shell characters) >> >> To use shell specific syntax, it's best if you put your commands >> into a shell script and invoke that. > > Ahh, ok. Out of curiosity, why not just do a system() call, with shell > and all? From the user's standpoint, it's not obvious that we cannot > put arbitrary unix commands into those fields. Mmm, gotta know, do ya? ;) I actually used to use system(3), but for many mundane reasons it caused trouble in odd cases, and had to switch to exec(2). system(2) is actually a complex call that does a LOT of weird stuff, depending on the platform. Several times it caused critical problems in high load environments. One is system(2) uses fork(), which causes trouble when the daemon has a large memory footprint when managing large numbers of large jobs. Two customers had unix job servers swapping to near death because a user dumped a large number of jobs all at once, which caused the jobdonecommand to fork the daemon numerous times within a short period, paging their servers for long periods, making them unresponsive. I had thought copy-on-write would prevent that from being a problem, but unfortunately when destructors are called, silly things like ptr = NULL; can trigger copy-on-write for an entire page, and enough of these paging operations pretty much defeats copy-on-write. Rush now uses vfork(2) to prevent this overhead, but the vfork docs are explicit on using exec(2) immediately after vfork, since vfork is often implemented with threads, where code like system(2) creates havoc. Also, system(2) is implemented with wait, which for many reasons can be undesirable for rush's own job control, where rushd needs to do the actual wait()ing on the child process. sh's also like to play around with signal handling, exit codes, and get inbetween rush and the user's process for managing signal deliveries. The code in system(2) is non-trivial, and affects many weird things like signals, which caused trouble with rushd's careful management of things like sigpipe. Within the system() call, there was a window of opportunity for the daemon to be susceptible to sigpipe, which could be triggered by high network traffic happening at the same time the system() call was running. There were a few other reasons, but memory isn't serving me. Suffice it to say there are 'really good reasons'. You might ask why don't I stick with exec, but use 'sh -c' to ensure the commands run in a shell. Again, there are technical reasons that range from cross platform space/quote handling to resource management calls and issues where the intermediate shell can cause trouble with certain customer configs and apps. Surprisingly, I ran into similar issues on Windows with CreateProcess(), which is even more info than the above. Gotta love systems programming. I'll see if I can put a small note in the docs that commands are run with exec(), and anything more complex than running a single command is best handled by creating a script. -- Greg Ercolano, erco@(email surpressed) Rush Render Queue, http://seriss.com/rush/ Tel: (Tel# suppressed) Fax: (Tel# suppressed) Cel: (Tel# suppressed) |
From: Antoine Durr <antoine@(email surpressed)> Subject: Re: redirects and semicolons in jobdumpcommand Date: Sat, 01 Dec 2007 14:53:55 -0500 |
Msg# 1658 View Complete Thread (5 articles) | All Threads Last Next |
On 2007-12-01 11:31:13 -0800, Greg Ercolano <erco@(email surpressed)> said: Antoine Durr wrote:The command is invoked by exec(), so no shell is involved. (';' and '>' being shell characters) To use shell specific syntax, it's best if you put your commands into a shell script and invoke that.Ahh, ok. Out of curiosity, why not just do a system() call, with shell and all? From the user's standpoint, it's not obvious that we cannot put arbitrary unix commands into those fields.Mmm, gotta know, do ya? ;) Wow, thanks for the great answer! Yeah, just a note in the docs would be plenty -- I looked there first, but nothing was written, nor was the field described with anything like 'single command' in the text. It really seems like some kind of lightweight system() call is needed. Now that you mention it, I ran into similar problems with a utility I wrote at R&H that did a system() call, but would fail if the running ..hip file's current memory footprint was substantial, e.g. 2+ gigs. Thanks, -- Antoine I actually used to use system(3), but for many mundane reasons it caused trouble in odd cases, and had to switch to exec(2). system(2) is actually a complex call that does a LOT of weird stuff, depending on the platform. Several times it caused critical problems in high load environments. One is system(2) uses fork(), which causes trouble when the daemon has a large memory footprint when managing large numbers of large jobs. Two customers had unix job servers swapping to near death because a user dumped a large number of jobs all at once, which caused the jobdonecommand to fork the daemon numerous times within a short period, paging their servers for long periods, making them unresponsive. I had thought copy-on-write would prevent that from being a problem, but unfortunately when destructors are called, silly things like ptr = NULL; can trigger copy-on-write for an entire page, and enough of these paging operations pretty much defeats copy-on-write. Rush now uses vfork(2) to prevent this overhead, but the vfork docs are explicit on using exec(2) immediately after vfork, since vfork is often implemented with threads, where code like system(2) creates havoc. Also, system(2) is implemented with wait, which for many reasons can be undesirable for rush's own job control, where rushd needs to do the actual wait()ing on the child process. sh's also like to play around with signal handling, exit codes, and get inbetween rush and the user's process for managing signal deliveries. The code in system(2) is non-trivial, and affects many weird things like signals, which caused trouble with rushd's careful management of things like sigpipe. Within the system() call, there was a window of opportunity for the daemon to be susceptible to sigpipe, which could be triggered by high network traffic happening at the same time the system() call was running. There were a few other reasons, but memory isn't serving me. Suffice it to say there are 'really good reasons'. You might ask why don't I stick with exec, but use 'sh -c' to ensure the commands run in a shell. Again, there are technical reasons that range from cross platform space/quote handling to resource management calls and issues where the intermediate shell can cause trouble with certain customer configs and apps. Surprisingly, I ran into similar issues on Windows with CreateProcess(), which is even more info than the above. Gotta love systems programming. I'll see if I can put a small note in the docs that commands are run with exec(), and anything more complex than running a single command is best handled by creating a script. -- Floq FX Inc. 10659 Cranks Rd. Culver City, CA 90230 310/430-2473 |