source: python/trunk/Lib/subprocess.py@ 1254

Last change on this file since 1254 was 1254, checked in by dmik, 8 years ago

python: Use spawn instead of fork in subprocess.

This significantly improves performance when running many external programs.
See #275 for more details.

Note that this commit also makes os.pipe use pipe() instead of socketpair() and
this in turn fixes #267. Note that socketpair() is still available via socket.socketpair()
in Python.
  • Property svn:eol-style set to native
File size: 63.5 KB
Line 
1# subprocess - Subprocesses with accessible I/O streams
2#
3# For more information about this module, see PEP 324.
4#
5# Copyright (c) 2003-2005 by Peter Astrand <astrand@lysator.liu.se>
6#
7# Licensed to PSF under a Contributor Agreement.
8# See http://www.python.org/2.4/license for licensing details.
9
10r"""subprocess - Subprocesses with accessible I/O streams
11
12This module allows you to spawn processes, connect to their
13input/output/error pipes, and obtain their return codes. This module
14intends to replace several other, older modules and functions, like:
15
16os.system
17os.spawn*
18os.popen*
19popen2.*
20commands.*
21
22Information about how the subprocess module can be used to replace these
23modules and functions can be found below.
24
25
26
27Using the subprocess module
28===========================
29This module defines one class called Popen:
30
31class Popen(args, bufsize=0, executable=None,
32 stdin=None, stdout=None, stderr=None,
33 preexec_fn=None, close_fds=False, shell=False,
34 cwd=None, env=None, universal_newlines=False,
35 startupinfo=None, creationflags=0):
36
37
38Arguments are:
39
40args should be a string, or a sequence of program arguments. The
41program to execute is normally the first item in the args sequence or
42string, but can be explicitly set by using the executable argument.
43
44On UNIX, with shell=False (default): In this case, the Popen class
45uses os.execvp() to execute the child program. args should normally
46be a sequence. A string will be treated as a sequence with the string
47as the only item (the program to execute).
48
49On UNIX, with shell=True: If args is a string, it specifies the
50command string to execute through the shell. If args is a sequence,
51the first item specifies the command string, and any additional items
52will be treated as additional shell arguments.
53
54On Windows: the Popen class uses CreateProcess() to execute the child
55program, which operates on strings. If args is a sequence, it will be
56converted to a string using the list2cmdline method. Please note that
57not all MS Windows applications interpret the command line the same
58way: The list2cmdline is designed for applications using the same
59rules as the MS C runtime.
60
61bufsize, if given, has the same meaning as the corresponding argument
62to the built-in open() function: 0 means unbuffered, 1 means line
63buffered, any other positive value means use a buffer of
64(approximately) that size. A negative bufsize means to use the system
65default, which usually means fully buffered. The default value for
66bufsize is 0 (unbuffered).
67
68stdin, stdout and stderr specify the executed programs' standard
69input, standard output and standard error file handles, respectively.
70Valid values are PIPE, an existing file descriptor (a positive
71integer), an existing file object, and None. PIPE indicates that a
72new pipe to the child should be created. With None, no redirection
73will occur; the child's file handles will be inherited from the
74parent. Additionally, stderr can be STDOUT, which indicates that the
75stderr data from the applications should be captured into the same
76file handle as for stdout.
77
78If preexec_fn is set to a callable object, this object will be called
79in the child process just before the child is executed.
80
81If close_fds is true, all file descriptors except 0, 1 and 2 will be
82closed before the child process is executed.
83
84if shell is true, the specified command will be executed through the
85shell.
86
87If cwd is not None, the current directory will be changed to cwd
88before the child is executed.
89
90If env is not None, it defines the environment variables for the new
91process.
92
93If universal_newlines is true, the file objects stdout and stderr are
94opened as a text files, but lines may be terminated by any of '\n',
95the Unix end-of-line convention, '\r', the Macintosh convention or
96'\r\n', the Windows convention. All of these external representations
97are seen as '\n' by the Python program. Note: This feature is only
98available if Python is built with universal newline support (the
99default). Also, the newlines attribute of the file objects stdout,
100stdin and stderr are not updated by the communicate() method.
101
102The startupinfo and creationflags, if given, will be passed to the
103underlying CreateProcess() function. They can specify things such as
104appearance of the main window and priority for the new process.
105(Windows only)
106
107
108This module also defines some shortcut functions:
109
110call(*popenargs, **kwargs):
111 Run command with arguments. Wait for command to complete, then
112 return the returncode attribute.
113
114 The arguments are the same as for the Popen constructor. Example:
115
116 retcode = call(["ls", "-l"])
117
118check_call(*popenargs, **kwargs):
119 Run command with arguments. Wait for command to complete. If the
120 exit code was zero then return, otherwise raise
121 CalledProcessError. The CalledProcessError object will have the
122 return code in the returncode attribute.
123
124 The arguments are the same as for the Popen constructor. Example:
125
126 check_call(["ls", "-l"])
127
128check_output(*popenargs, **kwargs):
129 Run command with arguments and return its output as a byte string.
130
131 If the exit code was non-zero it raises a CalledProcessError. The
132 CalledProcessError object will have the return code in the returncode
133 attribute and output in the output attribute.
134
135 The arguments are the same as for the Popen constructor. Example:
136
137 output = check_output(["ls", "-l", "/dev/null"])
138
139
140Exceptions
141----------
142Exceptions raised in the child process, before the new program has
143started to execute, will be re-raised in the parent. Additionally,
144the exception object will have one extra attribute called
145'child_traceback', which is a string containing traceback information
146from the child's point of view.
147
148The most common exception raised is OSError. This occurs, for
149example, when trying to execute a non-existent file. Applications
150should prepare for OSErrors.
151
152A ValueError will be raised if Popen is called with invalid arguments.
153
154check_call() and check_output() will raise CalledProcessError, if the
155called process returns a non-zero return code.
156
157
158Security
159--------
160Unlike some other popen functions, this implementation will never call
161/bin/sh implicitly. This means that all characters, including shell
162metacharacters, can safely be passed to child processes.
163
164
165Popen objects
166=============
167Instances of the Popen class have the following methods:
168
169poll()
170 Check if child process has terminated. Returns returncode
171 attribute.
172
173wait()
174 Wait for child process to terminate. Returns returncode attribute.
175
176communicate(input=None)
177 Interact with process: Send data to stdin. Read data from stdout
178 and stderr, until end-of-file is reached. Wait for process to
179 terminate. The optional input argument should be a string to be
180 sent to the child process, or None, if no data should be sent to
181 the child.
182
183 communicate() returns a tuple (stdout, stderr).
184
185 Note: The data read is buffered in memory, so do not use this
186 method if the data size is large or unlimited.
187
188The following attributes are also available:
189
190stdin
191 If the stdin argument is PIPE, this attribute is a file object
192 that provides input to the child process. Otherwise, it is None.
193
194stdout
195 If the stdout argument is PIPE, this attribute is a file object
196 that provides output from the child process. Otherwise, it is
197 None.
198
199stderr
200 If the stderr argument is PIPE, this attribute is file object that
201 provides error output from the child process. Otherwise, it is
202 None.
203
204pid
205 The process ID of the child process.
206
207returncode
208 The child return code. A None value indicates that the process
209 hasn't terminated yet. A negative value -N indicates that the
210 child was terminated by signal N (UNIX only).
211
212
213Replacing older functions with the subprocess module
214====================================================
215In this section, "a ==> b" means that b can be used as a replacement
216for a.
217
218Note: All functions in this section fail (more or less) silently if
219the executed program cannot be found; this module raises an OSError
220exception.
221
222In the following examples, we assume that the subprocess module is
223imported with "from subprocess import *".
224
225
226Replacing /bin/sh shell backquote
227---------------------------------
228output=`mycmd myarg`
229==>
230output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]
231
232
233Replacing shell pipe line
234-------------------------
235output=`dmesg | grep hda`
236==>
237p1 = Popen(["dmesg"], stdout=PIPE)
238p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
239output = p2.communicate()[0]
240
241
242Replacing os.system()
243---------------------
244sts = os.system("mycmd" + " myarg")
245==>
246p = Popen("mycmd" + " myarg", shell=True)
247pid, sts = os.waitpid(p.pid, 0)
248
249Note:
250
251* Calling the program through the shell is usually not required.
252
253* It's easier to look at the returncode attribute than the
254 exitstatus.
255
256A more real-world example would look like this:
257
258try:
259 retcode = call("mycmd" + " myarg", shell=True)
260 if retcode < 0:
261 print >>sys.stderr, "Child was terminated by signal", -retcode
262 else:
263 print >>sys.stderr, "Child returned", retcode
264except OSError, e:
265 print >>sys.stderr, "Execution failed:", e
266
267
268Replacing os.spawn*
269-------------------
270P_NOWAIT example:
271
272pid = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg")
273==>
274pid = Popen(["/bin/mycmd", "myarg"]).pid
275
276
277P_WAIT example:
278
279retcode = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg")
280==>
281retcode = call(["/bin/mycmd", "myarg"])
282
283
284Vector example:
285
286os.spawnvp(os.P_NOWAIT, path, args)
287==>
288Popen([path] + args[1:])
289
290
291Environment example:
292
293os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)
294==>
295Popen(["/bin/mycmd", "myarg"], env={"PATH": "/usr/bin"})
296
297
298Replacing os.popen*
299-------------------
300pipe = os.popen("cmd", mode='r', bufsize)
301==>
302pipe = Popen("cmd", shell=True, bufsize=bufsize, stdout=PIPE).stdout
303
304pipe = os.popen("cmd", mode='w', bufsize)
305==>
306pipe = Popen("cmd", shell=True, bufsize=bufsize, stdin=PIPE).stdin
307
308
309(child_stdin, child_stdout) = os.popen2("cmd", mode, bufsize)
310==>
311p = Popen("cmd", shell=True, bufsize=bufsize,
312 stdin=PIPE, stdout=PIPE, close_fds=True)
313(child_stdin, child_stdout) = (p.stdin, p.stdout)
314
315
316(child_stdin,
317 child_stdout,
318 child_stderr) = os.popen3("cmd", mode, bufsize)
319==>
320p = Popen("cmd", shell=True, bufsize=bufsize,
321 stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True)
322(child_stdin,
323 child_stdout,
324 child_stderr) = (p.stdin, p.stdout, p.stderr)
325
326
327(child_stdin, child_stdout_and_stderr) = os.popen4("cmd", mode,
328 bufsize)
329==>
330p = Popen("cmd", shell=True, bufsize=bufsize,
331 stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
332(child_stdin, child_stdout_and_stderr) = (p.stdin, p.stdout)
333
334On Unix, os.popen2, os.popen3 and os.popen4 also accept a sequence as
335the command to execute, in which case arguments will be passed
336directly to the program without shell intervention. This usage can be
337replaced as follows:
338
339(child_stdin, child_stdout) = os.popen2(["/bin/ls", "-l"], mode,
340 bufsize)
341==>
342p = Popen(["/bin/ls", "-l"], bufsize=bufsize, stdin=PIPE, stdout=PIPE)
343(child_stdin, child_stdout) = (p.stdin, p.stdout)
344
345Return code handling translates as follows:
346
347pipe = os.popen("cmd", 'w')
348...
349rc = pipe.close()
350if rc is not None and rc % 256:
351 print "There were some errors"
352==>
353process = Popen("cmd", 'w', shell=True, stdin=PIPE)
354...
355process.stdin.close()
356if process.wait() != 0:
357 print "There were some errors"
358
359
360Replacing popen2.*
361------------------
362(child_stdout, child_stdin) = popen2.popen2("somestring", bufsize, mode)
363==>
364p = Popen(["somestring"], shell=True, bufsize=bufsize
365 stdin=PIPE, stdout=PIPE, close_fds=True)
366(child_stdout, child_stdin) = (p.stdout, p.stdin)
367
368On Unix, popen2 also accepts a sequence as the command to execute, in
369which case arguments will be passed directly to the program without
370shell intervention. This usage can be replaced as follows:
371
372(child_stdout, child_stdin) = popen2.popen2(["mycmd", "myarg"], bufsize,
373 mode)
374==>
375p = Popen(["mycmd", "myarg"], bufsize=bufsize,
376 stdin=PIPE, stdout=PIPE, close_fds=True)
377(child_stdout, child_stdin) = (p.stdout, p.stdin)
378
379The popen2.Popen3 and popen2.Popen4 basically works as subprocess.Popen,
380except that:
381
382* subprocess.Popen raises an exception if the execution fails
383* the capturestderr argument is replaced with the stderr argument.
384* stdin=PIPE and stdout=PIPE must be specified.
385* popen2 closes all filedescriptors by default, but you have to specify
386 close_fds=True with subprocess.Popen.
387"""
388
389import sys
390mswindows = (sys.platform == "win32")
391
392import os
393import types
394import traceback
395import gc
396import signal
397import errno
398
399os2 = (os.name == "os2")
400
401import sysconfig
402SHELL = sysconfig.get_config_var('SHELL') or '/bin/sh'
403
404# Exception classes used by this module.
405class CalledProcessError(Exception):
406 """This exception is raised when a process run by check_call() or
407 check_output() returns a non-zero exit status.
408 The exit status will be stored in the returncode attribute;
409 check_output() will also store the output in the output attribute.
410 """
411 def __init__(self, returncode, cmd, output=None):
412 self.returncode = returncode
413 self.cmd = cmd
414 self.output = output
415 def __str__(self):
416 return "Command '%s' returned non-zero exit status %d" % (self.cmd, self.returncode)
417
418
419if mswindows:
420 import threading
421 import msvcrt
422 import _subprocess
423 class STARTUPINFO:
424 dwFlags = 0
425 hStdInput = None
426 hStdOutput = None
427 hStdError = None
428 wShowWindow = 0
429 class pywintypes:
430 error = IOError
431elif os2:
432 import threading
433 import fcntl
434 import time
435else:
436 import select
437 _has_poll = hasattr(select, 'poll')
438 import fcntl
439 import pickle
440
441 # When select or poll has indicated that the file is writable,
442 # we can write up to _PIPE_BUF bytes without risk of blocking.
443 # POSIX defines PIPE_BUF as >= 512.
444 _PIPE_BUF = getattr(select, 'PIPE_BUF', 512)
445
446
447__all__ = ["Popen", "PIPE", "STDOUT", "call", "check_call",
448 "check_output", "CalledProcessError"]
449
450if mswindows:
451 from _subprocess import (CREATE_NEW_CONSOLE, CREATE_NEW_PROCESS_GROUP,
452 STD_INPUT_HANDLE, STD_OUTPUT_HANDLE,
453 STD_ERROR_HANDLE, SW_HIDE,
454 STARTF_USESTDHANDLES, STARTF_USESHOWWINDOW)
455
456 __all__.extend(["CREATE_NEW_CONSOLE", "CREATE_NEW_PROCESS_GROUP",
457 "STD_INPUT_HANDLE", "STD_OUTPUT_HANDLE",
458 "STD_ERROR_HANDLE", "SW_HIDE",
459 "STARTF_USESTDHANDLES", "STARTF_USESHOWWINDOW"])
460try:
461 MAXFD = os.sysconf("SC_OPEN_MAX")
462except:
463 MAXFD = 256
464
465_active = []
466
467def _cleanup():
468 for inst in _active[:]:
469 res = inst._internal_poll(_deadstate=sys.maxint)
470 if res is not None:
471 try:
472 _active.remove(inst)
473 except ValueError:
474 # This can happen if two threads create a new Popen instance.
475 # It's harmless that it was already removed, so ignore.
476 pass
477
478PIPE = -1
479STDOUT = -2
480
481
482def _eintr_retry_call(func, *args):
483 while True:
484 try:
485 return func(*args)
486 except (OSError, IOError) as e:
487 if e.errno == errno.EINTR:
488 continue
489 raise
490
491
492# XXX This function is only used by multiprocessing and the test suite,
493# but it's here so that it can be imported when Python is compiled without
494# threads.
495
496def _args_from_interpreter_flags():
497 """Return a list of command-line arguments reproducing the current
498 settings in sys.flags and sys.warnoptions."""
499 flag_opt_map = {
500 'debug': 'd',
501 # 'inspect': 'i',
502 # 'interactive': 'i',
503 'optimize': 'O',
504 'dont_write_bytecode': 'B',
505 'no_user_site': 's',
506 'no_site': 'S',
507 'ignore_environment': 'E',
508 'verbose': 'v',
509 'bytes_warning': 'b',
510 'hash_randomization': 'R',
511 'py3k_warning': '3',
512 }
513 args = []
514 for flag, opt in flag_opt_map.items():
515 v = getattr(sys.flags, flag)
516 if v > 0:
517 args.append('-' + opt * v)
518 for opt in sys.warnoptions:
519 args.append('-W' + opt)
520 return args
521
522
523def call(*popenargs, **kwargs):
524 """Run command with arguments. Wait for command to complete, then
525 return the returncode attribute.
526
527 The arguments are the same as for the Popen constructor. Example:
528
529 retcode = call(["ls", "-l"])
530 """
531 return Popen(*popenargs, **kwargs).wait()
532
533
534def check_call(*popenargs, **kwargs):
535 """Run command with arguments. Wait for command to complete. If
536 the exit code was zero then return, otherwise raise
537 CalledProcessError. The CalledProcessError object will have the
538 return code in the returncode attribute.
539
540 The arguments are the same as for the Popen constructor. Example:
541
542 check_call(["ls", "-l"])
543 """
544 retcode = call(*popenargs, **kwargs)
545 if retcode:
546 cmd = kwargs.get("args")
547 if cmd is None:
548 cmd = popenargs[0]
549 raise CalledProcessError(retcode, cmd)
550 return 0
551
552
553def check_output(*popenargs, **kwargs):
554 r"""Run command with arguments and return its output as a byte string.
555
556 If the exit code was non-zero it raises a CalledProcessError. The
557 CalledProcessError object will have the return code in the returncode
558 attribute and output in the output attribute.
559
560 The arguments are the same as for the Popen constructor. Example:
561
562 >>> check_output(["ls", "-l", "/dev/null"])
563 'crw-rw-rw- 1 root root 1, 3 Oct 18 2007 /dev/null\n'
564
565 The stdout argument is not allowed as it is used internally.
566 To capture standard error in the result, use stderr=STDOUT.
567
568 >>> check_output(["/bin/sh", "-c",
569 ... "ls -l non_existent_file ; exit 0"],
570 ... stderr=STDOUT)
571 'ls: non_existent_file: No such file or directory\n'
572 """
573 if 'stdout' in kwargs:
574 raise ValueError('stdout argument not allowed, it will be overridden.')
575 process = Popen(stdout=PIPE, *popenargs, **kwargs)
576 output, unused_err = process.communicate()
577 retcode = process.poll()
578 if retcode:
579 cmd = kwargs.get("args")
580 if cmd is None:
581 cmd = popenargs[0]
582 raise CalledProcessError(retcode, cmd, output=output)
583 return output
584
585
586def list2cmdline(seq):
587 """
588 Translate a sequence of arguments into a command line
589 string, using the same rules as the MS C runtime:
590
591 1) Arguments are delimited by white space, which is either a
592 space or a tab.
593
594 2) A string surrounded by double quotation marks is
595 interpreted as a single argument, regardless of white space
596 contained within. A quoted string can be embedded in an
597 argument.
598
599 3) A double quotation mark preceded by a backslash is
600 interpreted as a literal double quotation mark.
601
602 4) Backslashes are interpreted literally, unless they
603 immediately precede a double quotation mark.
604
605 5) If backslashes immediately precede a double quotation mark,
606 every pair of backslashes is interpreted as a literal
607 backslash. If the number of backslashes is odd, the last
608 backslash escapes the next double quotation mark as
609 described in rule 3.
610 """
611
612 # See
613 # http://msdn.microsoft.com/en-us/library/17w5ykft.aspx
614 # or search http://msdn.microsoft.com for
615 # "Parsing C++ Command-Line Arguments"
616 result = []
617 needquote = False
618 for arg in seq:
619 bs_buf = []
620
621 # Add a space to separate this argument from the others
622 if result:
623 result.append(' ')
624
625 needquote = (" " in arg) or ("\t" in arg) or not arg
626 if needquote:
627 result.append('"')
628
629 for c in arg:
630 if c == '\\':
631 # Don't know if we need to double yet.
632 bs_buf.append(c)
633 elif c == '"':
634 # Double backslashes.
635 result.append('\\' * len(bs_buf)*2)
636 bs_buf = []
637 result.append('\\"')
638 else:
639 # Normal char
640 if bs_buf:
641 result.extend(bs_buf)
642 bs_buf = []
643 result.append(c)
644
645 # Add remaining backslashes, if any.
646 if bs_buf:
647 result.extend(bs_buf)
648
649 if needquote:
650 result.extend(bs_buf)
651 result.append('"')
652
653 return ''.join(result)
654
655
656class Popen(object):
657 def __init__(self, args, bufsize=0, executable=None,
658 stdin=None, stdout=None, stderr=None,
659 preexec_fn=None, close_fds=False, shell=False,
660 cwd=None, env=None, universal_newlines=False,
661 startupinfo=None, creationflags=0):
662 """Create new Popen instance."""
663 _cleanup()
664
665 self._child_created = False
666 if not isinstance(bufsize, (int, long)):
667 raise TypeError("bufsize must be an integer")
668
669 if mswindows or os2:
670 if preexec_fn is not None:
671 raise ValueError("preexec_fn is not supported on Windows "
672 "and OS/2 platforms")
673 if not os2 and close_fds and (stdin is not None or stdout is not None or
674 stderr is not None):
675 raise ValueError("close_fds is not supported on Windows "
676 "platforms if you redirect stdin/stdout/stderr")
677 if not mswindows:
678 # POSIX
679 if startupinfo is not None:
680 raise ValueError("startupinfo is only supported on Windows "
681 "platforms")
682 if creationflags != 0:
683 raise ValueError("creationflags is only supported on Windows "
684 "platforms")
685
686 self.stdin = None
687 self.stdout = None
688 self.stderr = None
689 self.pid = None
690 self.returncode = None
691 self.universal_newlines = universal_newlines
692
693 # Input and output objects. The general principle is like
694 # this:
695 #
696 # Parent Child
697 # ------ -----
698 # p2cwrite ---stdin---> p2cread
699 # c2pread <--stdout--- c2pwrite
700 # errread <--stderr--- errwrite
701 #
702 # On POSIX, the child objects are file descriptors. On
703 # Windows, these are Windows file handles. The parent objects
704 # are file descriptors on both platforms. The parent objects
705 # are None when not using PIPEs. The child objects are None
706 # when not redirecting.
707
708 (p2cread, p2cwrite,
709 c2pread, c2pwrite,
710 errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
711
712 try:
713 self._execute_child(args, executable, preexec_fn, close_fds,
714 cwd, env, universal_newlines,
715 startupinfo, creationflags, shell, to_close,
716 p2cread, p2cwrite,
717 c2pread, c2pwrite,
718 errread, errwrite)
719 except Exception:
720 # Preserve original exception in case os.close raises.
721 exc_type, exc_value, exc_trace = sys.exc_info()
722
723 for fd in to_close:
724 try:
725 if mswindows:
726 fd.Close()
727 else:
728 os.close(fd)
729 except EnvironmentError:
730 pass
731
732 raise exc_type, exc_value, exc_trace
733
734 if mswindows:
735 if p2cwrite is not None:
736 p2cwrite = msvcrt.open_osfhandle(p2cwrite.Detach(), 0)
737 if c2pread is not None:
738 c2pread = msvcrt.open_osfhandle(c2pread.Detach(), 0)
739 if errread is not None:
740 errread = msvcrt.open_osfhandle(errread.Detach(), 0)
741
742 if p2cwrite is not None:
743 self.stdin = os.fdopen(p2cwrite, 'wb', bufsize)
744 if c2pread is not None:
745 if universal_newlines:
746 self.stdout = os.fdopen(c2pread, 'rU', bufsize)
747 else:
748 self.stdout = os.fdopen(c2pread, 'rb', bufsize)
749 if errread is not None:
750 if universal_newlines:
751 self.stderr = os.fdopen(errread, 'rU', bufsize)
752 else:
753 self.stderr = os.fdopen(errread, 'rb', bufsize)
754
755
756 def _translate_newlines(self, data):
757 data = data.replace("\r\n", "\n")
758 data = data.replace("\r", "\n")
759 return data
760
761
762 def __del__(self, _maxint=sys.maxint, _active=_active):
763 # If __init__ hasn't had a chance to execute (e.g. if it
764 # was passed an undeclared keyword argument), we don't
765 # have a _child_created attribute at all.
766 if not getattr(self, '_child_created', False):
767 # We didn't get to successfully create a child process.
768 return
769 # In case the child hasn't been waited on, check if it's done.
770 self._internal_poll(_deadstate=_maxint)
771 if self.returncode is None and _active is not None:
772 # Child is still running, keep us alive until we can wait on it.
773 _active.append(self)
774
775
776 def communicate(self, input=None):
777 """Interact with process: Send data to stdin. Read data from
778 stdout and stderr, until end-of-file is reached. Wait for
779 process to terminate. The optional input argument should be a
780 string to be sent to the child process, or None, if no data
781 should be sent to the child.
782
783 communicate() returns a tuple (stdout, stderr)."""
784
785 # Optimization: If we are only using one pipe, or no pipe at
786 # all, using select() or threads is unnecessary.
787 if [self.stdin, self.stdout, self.stderr].count(None) >= 2:
788 stdout = None
789 stderr = None
790 if self.stdin:
791 if input:
792 try:
793 self.stdin.write(input)
794 except IOError as e:
795 if e.errno != errno.EPIPE and e.errno != errno.EINVAL:
796 raise
797 self.stdin.close()
798 elif self.stdout:
799 stdout = _eintr_retry_call(self.stdout.read)
800 self.stdout.close()
801 elif self.stderr:
802 stderr = _eintr_retry_call(self.stderr.read)
803 self.stderr.close()
804 self.wait()
805 return (stdout, stderr)
806
807 return self._communicate(input)
808
809
810 def poll(self):
811 return self._internal_poll()
812
813
814 if mswindows:
815 #
816 # Windows methods
817 #
818 def _get_handles(self, stdin, stdout, stderr):
819 """Construct and return tuple with IO objects:
820 p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite
821 """
822 to_close = set()
823 if stdin is None and stdout is None and stderr is None:
824 return (None, None, None, None, None, None), to_close
825
826 p2cread, p2cwrite = None, None
827 c2pread, c2pwrite = None, None
828 errread, errwrite = None, None
829
830 if stdin is None:
831 p2cread = _subprocess.GetStdHandle(_subprocess.STD_INPUT_HANDLE)
832 if p2cread is None:
833 p2cread, _ = _subprocess.CreatePipe(None, 0)
834 elif stdin == PIPE:
835 p2cread, p2cwrite = _subprocess.CreatePipe(None, 0)
836 elif isinstance(stdin, int):
837 p2cread = msvcrt.get_osfhandle(stdin)
838 else:
839 # Assuming file-like object
840 p2cread = msvcrt.get_osfhandle(stdin.fileno())
841 p2cread = self._make_inheritable(p2cread)
842 # We just duplicated the handle, it has to be closed at the end
843 to_close.add(p2cread)
844 if stdin == PIPE:
845 to_close.add(p2cwrite)
846
847 if stdout is None:
848 c2pwrite = _subprocess.GetStdHandle(_subprocess.STD_OUTPUT_HANDLE)
849 if c2pwrite is None:
850 _, c2pwrite = _subprocess.CreatePipe(None, 0)
851 elif stdout == PIPE:
852 c2pread, c2pwrite = _subprocess.CreatePipe(None, 0)
853 elif isinstance(stdout, int):
854 c2pwrite = msvcrt.get_osfhandle(stdout)
855 else:
856 # Assuming file-like object
857 c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
858 c2pwrite = self._make_inheritable(c2pwrite)
859 # We just duplicated the handle, it has to be closed at the end
860 to_close.add(c2pwrite)
861 if stdout == PIPE:
862 to_close.add(c2pread)
863
864 if stderr is None:
865 errwrite = _subprocess.GetStdHandle(_subprocess.STD_ERROR_HANDLE)
866 if errwrite is None:
867 _, errwrite = _subprocess.CreatePipe(None, 0)
868 elif stderr == PIPE:
869 errread, errwrite = _subprocess.CreatePipe(None, 0)
870 elif stderr == STDOUT:
871 errwrite = c2pwrite
872 elif isinstance(stderr, int):
873 errwrite = msvcrt.get_osfhandle(stderr)
874 else:
875 # Assuming file-like object
876 errwrite = msvcrt.get_osfhandle(stderr.fileno())
877 errwrite = self._make_inheritable(errwrite)
878 # We just duplicated the handle, it has to be closed at the end
879 to_close.add(errwrite)
880 if stderr == PIPE:
881 to_close.add(errread)
882
883 return (p2cread, p2cwrite,
884 c2pread, c2pwrite,
885 errread, errwrite), to_close
886
887
888 def _make_inheritable(self, handle):
889 """Return a duplicate of handle, which is inheritable"""
890 return _subprocess.DuplicateHandle(_subprocess.GetCurrentProcess(),
891 handle, _subprocess.GetCurrentProcess(), 0, 1,
892 _subprocess.DUPLICATE_SAME_ACCESS)
893
894
895 def _find_w9xpopen(self):
896 """Find and return absolut path to w9xpopen.exe"""
897 w9xpopen = os.path.join(
898 os.path.dirname(_subprocess.GetModuleFileName(0)),
899 "w9xpopen.exe")
900 if not os.path.exists(w9xpopen):
901 # Eeek - file-not-found - possibly an embedding
902 # situation - see if we can locate it in sys.exec_prefix
903 w9xpopen = os.path.join(os.path.dirname(sys.exec_prefix),
904 "w9xpopen.exe")
905 if not os.path.exists(w9xpopen):
906 raise RuntimeError("Cannot locate w9xpopen.exe, which is "
907 "needed for Popen to work with your "
908 "shell or platform.")
909 return w9xpopen
910
911
912 def _execute_child(self, args, executable, preexec_fn, close_fds,
913 cwd, env, universal_newlines,
914 startupinfo, creationflags, shell, to_close,
915 p2cread, p2cwrite,
916 c2pread, c2pwrite,
917 errread, errwrite):
918 """Execute program (MS Windows version)"""
919
920 if not isinstance(args, types.StringTypes):
921 args = list2cmdline(args)
922
923 # Process startup details
924 if startupinfo is None:
925 startupinfo = STARTUPINFO()
926 if None not in (p2cread, c2pwrite, errwrite):
927 startupinfo.dwFlags |= _subprocess.STARTF_USESTDHANDLES
928 startupinfo.hStdInput = p2cread
929 startupinfo.hStdOutput = c2pwrite
930 startupinfo.hStdError = errwrite
931
932 if shell:
933 startupinfo.dwFlags |= _subprocess.STARTF_USESHOWWINDOW
934 startupinfo.wShowWindow = _subprocess.SW_HIDE
935 comspec = os.environ.get("COMSPEC", "cmd.exe")
936 args = '{} /c "{}"'.format (comspec, args)
937 if (_subprocess.GetVersion() >= 0x80000000 or
938 os.path.basename(comspec).lower() == "command.com"):
939 # Win9x, or using command.com on NT. We need to
940 # use the w9xpopen intermediate program. For more
941 # information, see KB Q150956
942 # (http://web.archive.org/web/20011105084002/http://support.microsoft.com/support/kb/articles/Q150/9/56.asp)
943 w9xpopen = self._find_w9xpopen()
944 args = '"%s" %s' % (w9xpopen, args)
945 # Not passing CREATE_NEW_CONSOLE has been known to
946 # cause random failures on win9x. Specifically a
947 # dialog: "Your program accessed mem currently in
948 # use at xxx" and a hopeful warning about the
949 # stability of your system. Cost is Ctrl+C wont
950 # kill children.
951 creationflags |= _subprocess.CREATE_NEW_CONSOLE
952
953 def _close_in_parent(fd):
954 fd.Close()
955 to_close.remove(fd)
956
957 # Start the process
958 try:
959 hp, ht, pid, tid = _subprocess.CreateProcess(executable, args,
960 # no special security
961 None, None,
962 int(not close_fds),
963 creationflags,
964 env,
965 cwd,
966 startupinfo)
967 except pywintypes.error, e:
968 # Translate pywintypes.error to WindowsError, which is
969 # a subclass of OSError. FIXME: We should really
970 # translate errno using _sys_errlist (or similar), but
971 # how can this be done from Python?
972 raise WindowsError(*e.args)
973 finally:
974 # Child is launched. Close the parent's copy of those pipe
975 # handles that only the child should have open. You need
976 # to make sure that no handles to the write end of the
977 # output pipe are maintained in this process or else the
978 # pipe will not close when the child process exits and the
979 # ReadFile will hang.
980 if p2cread is not None:
981 _close_in_parent(p2cread)
982 if c2pwrite is not None:
983 _close_in_parent(c2pwrite)
984 if errwrite is not None:
985 _close_in_parent(errwrite)
986
987 # Retain the process handle, but close the thread handle
988 self._child_created = True
989 self._handle = hp
990 self.pid = pid
991 ht.Close()
992
993 def _internal_poll(self, _deadstate=None,
994 _WaitForSingleObject=_subprocess.WaitForSingleObject,
995 _WAIT_OBJECT_0=_subprocess.WAIT_OBJECT_0,
996 _GetExitCodeProcess=_subprocess.GetExitCodeProcess):
997 """Check if child process has terminated. Returns returncode
998 attribute.
999
1000 This method is called by __del__, so it can only refer to objects
1001 in its local scope.
1002
1003 """
1004 if self.returncode is None:
1005 if _WaitForSingleObject(self._handle, 0) == _WAIT_OBJECT_0:
1006 self.returncode = _GetExitCodeProcess(self._handle)
1007 return self.returncode
1008
1009
1010 def wait(self):
1011 """Wait for child process to terminate. Returns returncode
1012 attribute."""
1013 if self.returncode is None:
1014 _subprocess.WaitForSingleObject(self._handle,
1015 _subprocess.INFINITE)
1016 self.returncode = _subprocess.GetExitCodeProcess(self._handle)
1017 return self.returncode
1018
1019
1020 def _readerthread(self, fh, buffer):
1021 buffer.append(fh.read())
1022
1023
1024 def _communicate(self, input):
1025 stdout = None # Return
1026 stderr = None # Return
1027
1028 if self.stdout:
1029 stdout = []
1030 stdout_thread = threading.Thread(target=self._readerthread,
1031 args=(self.stdout, stdout))
1032 stdout_thread.setDaemon(True)
1033 stdout_thread.start()
1034 if self.stderr:
1035 stderr = []
1036 stderr_thread = threading.Thread(target=self._readerthread,
1037 args=(self.stderr, stderr))
1038 stderr_thread.setDaemon(True)
1039 stderr_thread.start()
1040
1041 if self.stdin:
1042 if input is not None:
1043 try:
1044 self.stdin.write(input)
1045 except IOError as e:
1046 if e.errno != errno.EPIPE:
1047 raise
1048 self.stdin.close()
1049
1050 if self.stdout:
1051 stdout_thread.join()
1052 if self.stderr:
1053 stderr_thread.join()
1054
1055 # All data exchanged. Translate lists into strings.
1056 if stdout is not None:
1057 stdout = stdout[0]
1058 if stderr is not None:
1059 stderr = stderr[0]
1060
1061 # Translate newlines, if requested. We cannot let the file
1062 # object do the translation: It is based on stdio, which is
1063 # impossible to combine with select (unless forcing no
1064 # buffering).
1065 if self.universal_newlines and hasattr(file, 'newlines'):
1066 if stdout:
1067 stdout = self._translate_newlines(stdout)
1068 if stderr:
1069 stderr = self._translate_newlines(stderr)
1070
1071 self.wait()
1072 return (stdout, stderr)
1073
1074 def send_signal(self, sig):
1075 """Send a signal to the process
1076 """
1077 if sig == signal.SIGTERM:
1078 self.terminate()
1079 elif sig == signal.CTRL_C_EVENT:
1080 os.kill(self.pid, signal.CTRL_C_EVENT)
1081 elif sig == signal.CTRL_BREAK_EVENT:
1082 os.kill(self.pid, signal.CTRL_BREAK_EVENT)
1083 else:
1084 raise ValueError("Unsupported signal: {}".format(sig))
1085
1086 def terminate(self):
1087 """Terminates the process
1088 """
1089 try:
1090 _subprocess.TerminateProcess(self._handle, 1)
1091 except OSError as e:
1092 # ERROR_ACCESS_DENIED (winerror 5) is received when the
1093 # process already died.
1094 if e.winerror != 5:
1095 raise
1096 rc = _subprocess.GetExitCodeProcess(self._handle)
1097 if rc == _subprocess.STILL_ACTIVE:
1098 raise
1099 self.returncode = rc
1100
1101 kill = terminate
1102
1103 else:
1104 #
1105 # POSIX methods
1106 #
1107 def _get_handles(self, stdin, stdout, stderr):
1108 """Construct and return tuple with IO objects:
1109 p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite
1110 """
1111 to_close = set()
1112 p2cread, p2cwrite = None, None
1113 c2pread, c2pwrite = None, None
1114 errread, errwrite = None, None
1115
1116 if stdin is None:
1117 pass
1118 elif stdin == PIPE:
1119 p2cread, p2cwrite = self.pipe_cloexec()
1120 to_close.update((p2cread, p2cwrite))
1121 elif isinstance(stdin, int):
1122 p2cread = stdin
1123 else:
1124 # Assuming file-like object
1125 p2cread = stdin.fileno()
1126
1127 if stdout is None:
1128 pass
1129 elif stdout == PIPE:
1130 c2pread, c2pwrite = self.pipe_cloexec()
1131 to_close.update((c2pread, c2pwrite))
1132 elif isinstance(stdout, int):
1133 c2pwrite = stdout
1134 else:
1135 # Assuming file-like object
1136 c2pwrite = stdout.fileno()
1137
1138 if stderr is None:
1139 pass
1140 elif stderr == PIPE:
1141 errread, errwrite = self.pipe_cloexec()
1142 to_close.update((errread, errwrite))
1143 elif stderr == STDOUT:
1144 errwrite = c2pwrite
1145 elif isinstance(stderr, int):
1146 errwrite = stderr
1147 else:
1148 # Assuming file-like object
1149 errwrite = stderr.fileno()
1150
1151 return (p2cread, p2cwrite,
1152 c2pread, c2pwrite,
1153 errread, errwrite), to_close
1154
1155
1156 def _set_cloexec_flag(self, fd, cloexec=True):
1157 try:
1158 cloexec_flag = fcntl.FD_CLOEXEC
1159 except AttributeError:
1160 cloexec_flag = 1
1161
1162 old = fcntl.fcntl(fd, fcntl.F_GETFD)
1163 if cloexec:
1164 fcntl.fcntl(fd, fcntl.F_SETFD, old | cloexec_flag)
1165 else:
1166 fcntl.fcntl(fd, fcntl.F_SETFD, old & ~cloexec_flag)
1167
1168
1169 def pipe_cloexec(self):
1170 """Create a pipe with FDs set CLOEXEC."""
1171 # Pipes' FDs are set CLOEXEC by default because we don't want them
1172 # to be inherited by other subprocesses: the CLOEXEC flag is removed
1173 # from the child's FDs by _dup2(), between fork() and exec().
1174 # This is not atomic: we would need the pipe2() syscall for that.
1175 r, w = os.pipe()
1176 self._set_cloexec_flag(r)
1177 self._set_cloexec_flag(w)
1178 return r, w
1179
1180
1181 def _close_fds(self, but):
1182 if hasattr(os, 'closerange'):
1183 os.closerange(3, but)
1184 os.closerange(but + 1, MAXFD)
1185 else:
1186 for i in xrange(3, MAXFD):
1187 if i == but:
1188 continue
1189 try:
1190 os.close(i)
1191 except:
1192 pass
1193
1194
1195 def _execute_child(self, args, executable, preexec_fn, close_fds,
1196 cwd, env, universal_newlines,
1197 startupinfo, creationflags, shell, to_close,
1198 p2cread, p2cwrite,
1199 c2pread, c2pwrite,
1200 errread, errwrite):
1201 """Execute program (POSIX version)"""
1202
1203 if isinstance(args, types.StringTypes):
1204 args = [args]
1205 else:
1206 args = list(args)
1207
1208 if shell:
1209 args = [SHELL, "-c"] + args
1210 if executable:
1211 args[0] = executable
1212
1213 if executable is None:
1214 executable = args[0]
1215
1216 def _close_in_parent(fd):
1217 os.close(fd)
1218 to_close.remove(fd)
1219
1220 if os2:
1221 # We use spawn on OS/2 instead of fork because fork is very
1222 # inefficient there (mostly due to the lack of COW page support
1223 # in the kernel).
1224 mode = os.P_NOWAIT
1225 oldcwd = None
1226 dups = [ None, None, None ]
1227 cloexec_fds = set()
1228
1229 try:
1230 # Change the directory if asked
1231 if cwd is not None:
1232 oldcwd = os.getcwd()
1233 os.chdir(cwd)
1234
1235 # Duplicate stdio if needed
1236 if p2cread is not None:
1237 dups[0] = os.dup(0)
1238 os.dup2(p2cread, 0)
1239 self._set_cloexec_flag(dups[0])
1240 if c2pwrite is not None:
1241 dups[1] = os.dup(1)
1242 os.dup2(c2pwrite, 1)
1243 self._set_cloexec_flag(dups[1])
1244 if errwrite is not None:
1245 dups[2] = os.dup(2)
1246 os.dup2(errwrite, 2)
1247 self._set_cloexec_flag(dups[2])
1248
1249 # Disable inheritance
1250 if close_fds:
1251 for i in xrange(3, MAXFD):
1252 try:
1253 f = fcntl.fcntl(i, fcntl.F_GETFD)
1254 if not (f & fcntl.FD_CLOEXEC):
1255 cloexec_fds.add(i)
1256 self._set_cloexec_flag(i)
1257 except:
1258 pass
1259
1260 if env is None:
1261 pid = os.spawnvp(mode, executable, args)
1262 else:
1263 pid = os.spawnvpe(mode, executable, args, env)
1264 finally:
1265 # Restore inheritance
1266 for i in cloexec_fds:
1267 self._set_cloexec_flag(i, False)
1268
1269 # Restore the parent stdio
1270 for i, fd in enumerate(dups):
1271 if fd is not None:
1272 os.dup2(fd, i)
1273 os.close(fd)
1274
1275 # Restore the current directory
1276 if oldcwd is not None:
1277 os.chdir(oldcwd)
1278
1279 # Child is launched. Close the parent's copy of those pipe
1280 # handles that only the child should have open. You need
1281 # to make sure that no handles to the write end of the
1282 # output pipe are maintained in this process or else the
1283 # pipe will not close when the child process exits and the
1284 # ReadFile will hang.
1285 if p2cread is not None and p2cwrite is not None:
1286 _close_in_parent(p2cread)
1287 if c2pwrite is not None and c2pread is not None:
1288 _close_in_parent(c2pwrite)
1289 if errwrite is not None and errread is not None:
1290 _close_in_parent(errwrite)
1291
1292 self._child_created = True
1293 self.pid = pid
1294
1295 else:
1296 # For transferring possible exec failure from child to parent
1297 # The first char specifies the exception type: 0 means
1298 # OSError, 1 means some other error.
1299 errpipe_read, errpipe_write = self.pipe_cloexec()
1300 try:
1301 try:
1302 gc_was_enabled = gc.isenabled()
1303 # Disable gc to avoid bug where gc -> file_dealloc ->
1304 # write to stderr -> hang. http://bugs.python.org/issue1336
1305 gc.disable()
1306 try:
1307 self.pid = os.fork()
1308 except:
1309 if gc_was_enabled:
1310 gc.enable()
1311 raise
1312 self._child_created = True
1313 if self.pid == 0:
1314 # Child
1315 try:
1316 # Close parent's pipe ends
1317 if p2cwrite is not None:
1318 os.close(p2cwrite)
1319 if c2pread is not None:
1320 os.close(c2pread)
1321 if errread is not None:
1322 os.close(errread)
1323 os.close(errpipe_read)
1324
1325 # When duping fds, if there arises a situation
1326 # where one of the fds is either 0, 1 or 2, it
1327 # is possible that it is overwritten (#12607).
1328 if c2pwrite == 0:
1329 c2pwrite = os.dup(c2pwrite)
1330 if errwrite == 0 or errwrite == 1:
1331 errwrite = os.dup(errwrite)
1332
1333 # Dup fds for child
1334 def _dup2(a, b):
1335 # dup2() removes the CLOEXEC flag but
1336 # we must do it ourselves if dup2()
1337 # would be a no-op (issue #10806).
1338 if a == b:
1339 self._set_cloexec_flag(a, False)
1340 elif a is not None:
1341 os.dup2(a, b)
1342 _dup2(p2cread, 0)
1343 _dup2(c2pwrite, 1)
1344 _dup2(errwrite, 2)
1345
1346 # Close pipe fds. Make sure we don't close the
1347 # same fd more than once, or standard fds.
1348 closed = { None }
1349 for fd in [p2cread, c2pwrite, errwrite]:
1350 if fd not in closed and fd > 2:
1351 os.close(fd)
1352 closed.add(fd)
1353
1354 if cwd is not None:
1355 os.chdir(cwd)
1356
1357 if preexec_fn:
1358 preexec_fn()
1359
1360 # Close all other fds, if asked for - after
1361 # preexec_fn(), which may open FDs.
1362 if close_fds:
1363 self._close_fds(but=errpipe_write)
1364
1365 if env is None:
1366 os.execvp(executable, args)
1367 else:
1368 os.execvpe(executable, args, env)
1369
1370 except:
1371 exc_type, exc_value, tb = sys.exc_info()
1372 # Save the traceback and attach it to the exception object
1373 exc_lines = traceback.format_exception(exc_type,
1374 exc_value,
1375 tb)
1376 exc_value.child_traceback = ''.join(exc_lines)
1377 os.write(errpipe_write, pickle.dumps(exc_value))
1378
1379 # This exitcode won't be reported to applications, so it
1380 # really doesn't matter what we return.
1381 os._exit(255)
1382
1383 # Parent
1384 if gc_was_enabled:
1385 gc.enable()
1386 finally:
1387 # be sure the FD is closed no matter what
1388 os.close(errpipe_write)
1389
1390 # Wait for exec to fail or succeed; possibly raising exception
1391 # Exception limited to 1M
1392 data = _eintr_retry_call(os.read, errpipe_read, 1048576)
1393 finally:
1394 if p2cread is not None and p2cwrite is not None:
1395 _close_in_parent(p2cread)
1396 if c2pwrite is not None and c2pread is not None:
1397 _close_in_parent(c2pwrite)
1398 if errwrite is not None and errread is not None:
1399 _close_in_parent(errwrite)
1400
1401 # be sure the FD is closed no matter what
1402 os.close(errpipe_read)
1403
1404 if data != "":
1405 try:
1406 _eintr_retry_call(os.waitpid, self.pid, 0)
1407 except OSError as e:
1408 if e.errno != errno.ECHILD:
1409 raise
1410 child_exception = pickle.loads(data)
1411 raise child_exception
1412
1413
1414 def _handle_exitstatus(self, sts, _WIFSIGNALED=os.WIFSIGNALED,
1415 _WTERMSIG=os.WTERMSIG, _WIFEXITED=os.WIFEXITED,
1416 _WEXITSTATUS=os.WEXITSTATUS):
1417 # This method is called (indirectly) by __del__, so it cannot
1418 # refer to anything outside of its local scope."""
1419 if _WIFSIGNALED(sts):
1420 self.returncode = -_WTERMSIG(sts)
1421 elif _WIFEXITED(sts):
1422 self.returncode = _WEXITSTATUS(sts)
1423 else:
1424 # Should never happen
1425 raise RuntimeError("Unknown child exit status!")
1426
1427
1428 def _internal_poll(self, _deadstate=None, _waitpid=os.waitpid,
1429 _WNOHANG=os.WNOHANG, _os_error=os.error, _ECHILD=errno.ECHILD):
1430 """Check if child process has terminated. Returns returncode
1431 attribute.
1432
1433 This method is called by __del__, so it cannot reference anything
1434 outside of the local scope (nor can any methods it calls).
1435
1436 """
1437 if self.returncode is None:
1438 try:
1439 pid, sts = _waitpid(self.pid, _WNOHANG)
1440 if pid == self.pid:
1441 self._handle_exitstatus(sts)
1442 except _os_error as e:
1443 if _deadstate is not None:
1444 self.returncode = _deadstate
1445 if e.errno == _ECHILD:
1446 # This happens if SIGCLD is set to be ignored or
1447 # waiting for child processes has otherwise been
1448 # disabled for our process. This child is dead, we
1449 # can't get the status.
1450 # http://bugs.python.org/issue15756
1451 self.returncode = 0
1452 return self.returncode
1453
1454
1455 def wait(self):
1456 """Wait for child process to terminate. Returns returncode
1457 attribute."""
1458 while self.returncode is None:
1459 try:
1460 pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
1461 except OSError as e:
1462 if e.errno != errno.ECHILD:
1463 raise
1464 # This happens if SIGCLD is set to be ignored or waiting
1465 # for child processes has otherwise been disabled for our
1466 # process. This child is dead, we can't get the status.
1467 pid = self.pid
1468 sts = 0
1469 # Check the pid and loop as waitpid has been known to return
1470 # 0 even without WNOHANG in odd situations. issue14396.
1471 if pid == self.pid:
1472 self._handle_exitstatus(sts)
1473 return self.returncode
1474
1475
1476 def _communicate(self, input):
1477 if os2:
1478 def _readerthread(file, buffer):
1479 while True:
1480 data = _eintr_retry_call(file.read)
1481 if data == "":
1482 file.close()
1483 break
1484 buffer.append(data)
1485
1486 stdout = None # Return
1487 stderr = None # Return
1488
1489 if self.stdout:
1490 stdout = []
1491 stdout_thread = threading.Thread(target=_readerthread,
1492 args=(self.stdout, stdout))
1493 stdout_thread.setDaemon(True)
1494 stdout_thread.start()
1495
1496 if self.stderr:
1497 stderr = []
1498 stderr_thread = threading.Thread(target=_readerthread,
1499 args=(self.stderr, stderr))
1500 stderr_thread.setDaemon(True)
1501 stderr_thread.start()
1502
1503 if self.stdin:
1504 if input is not None:
1505 try:
1506 self.stdin.write(input)
1507 except IOError as e:
1508 if e.errno != errno.EPIPE:
1509 raise
1510 self.stdin.close()
1511
1512 if self.stdout:
1513 stdout_thread.join()
1514 if self.stderr:
1515 stderr_thread.join()
1516
1517 else:
1518 if self.stdin:
1519 # Flush stdio buffer. This might block, if the user has
1520 # been writing to .stdin in an uncontrolled fashion.
1521 self.stdin.flush()
1522 if not input:
1523 self.stdin.close()
1524
1525 if _has_poll:
1526 stdout, stderr = self._communicate_with_poll(input)
1527 else:
1528 stdout, stderr = self._communicate_with_select(input)
1529
1530 # All data exchanged. Translate lists into strings.
1531 if stdout is not None:
1532 stdout = ''.join(stdout)
1533 if stderr is not None:
1534 stderr = ''.join(stderr)
1535
1536 # Translate newlines, if requested. We cannot let the file
1537 # object do the translation: It is based on stdio, which is
1538 # impossible to combine with select (unless forcing no
1539 # buffering).
1540 if self.universal_newlines and hasattr(file, 'newlines'):
1541 if stdout:
1542 stdout = self._translate_newlines(stdout)
1543 if stderr:
1544 stderr = self._translate_newlines(stderr)
1545
1546 self.wait()
1547 return (stdout, stderr)
1548
1549
1550 def _communicate_with_poll(self, input):
1551 stdout = None # Return
1552 stderr = None # Return
1553 fd2file = {}
1554 fd2output = {}
1555
1556 poller = select.poll()
1557 def register_and_append(file_obj, eventmask):
1558 poller.register(file_obj.fileno(), eventmask)
1559 fd2file[file_obj.fileno()] = file_obj
1560
1561 def close_unregister_and_remove(fd):
1562 poller.unregister(fd)
1563 fd2file[fd].close()
1564 fd2file.pop(fd)
1565
1566 if self.stdin and input:
1567 register_and_append(self.stdin, select.POLLOUT)
1568
1569 select_POLLIN_POLLPRI = select.POLLIN | select.POLLPRI
1570 if self.stdout:
1571 register_and_append(self.stdout, select_POLLIN_POLLPRI)
1572 fd2output[self.stdout.fileno()] = stdout = []
1573 if self.stderr:
1574 register_and_append(self.stderr, select_POLLIN_POLLPRI)
1575 fd2output[self.stderr.fileno()] = stderr = []
1576
1577 input_offset = 0
1578 while fd2file:
1579 try:
1580 ready = poller.poll()
1581 except select.error, e:
1582 if e.args[0] == errno.EINTR:
1583 continue
1584 raise
1585
1586 for fd, mode in ready:
1587 if mode & select.POLLOUT:
1588 chunk = input[input_offset : input_offset + _PIPE_BUF]
1589 try:
1590 input_offset += os.write(fd, chunk)
1591 except OSError as e:
1592 if e.errno == errno.EPIPE:
1593 close_unregister_and_remove(fd)
1594 else:
1595 raise
1596 else:
1597 if input_offset >= len(input):
1598 close_unregister_and_remove(fd)
1599 elif mode & select_POLLIN_POLLPRI:
1600 data = os.read(fd, 4096)
1601 if not data:
1602 close_unregister_and_remove(fd)
1603 fd2output[fd].append(data)
1604 else:
1605 # Ignore hang up or errors.
1606 close_unregister_and_remove(fd)
1607
1608 return (stdout, stderr)
1609
1610
1611 def _communicate_with_select(self, input):
1612 read_set = []
1613 write_set = []
1614 stdout = None # Return
1615 stderr = None # Return
1616
1617 if self.stdin and input:
1618 write_set.append(self.stdin)
1619 if self.stdout:
1620 read_set.append(self.stdout)
1621 stdout = []
1622 if self.stderr:
1623 read_set.append(self.stderr)
1624 stderr = []
1625
1626 input_offset = 0
1627 while read_set or write_set:
1628 try:
1629 rlist, wlist, xlist = select.select(read_set, write_set, [])
1630 except select.error, e:
1631 if e.args[0] == errno.EINTR:
1632 continue
1633 raise
1634
1635 if self.stdin in wlist:
1636 chunk = input[input_offset : input_offset + _PIPE_BUF]
1637 try:
1638 bytes_written = os.write(self.stdin.fileno(), chunk)
1639 except OSError as e:
1640 if e.errno == errno.EPIPE:
1641 self.stdin.close()
1642 write_set.remove(self.stdin)
1643 else:
1644 raise
1645 else:
1646 input_offset += bytes_written
1647 if input_offset >= len(input):
1648 self.stdin.close()
1649 write_set.remove(self.stdin)
1650
1651 if self.stdout in rlist:
1652 data = os.read(self.stdout.fileno(), 1024)
1653 if data == "":
1654 self.stdout.close()
1655 read_set.remove(self.stdout)
1656 stdout.append(data)
1657
1658 if self.stderr in rlist:
1659 data = os.read(self.stderr.fileno(), 1024)
1660 if data == "":
1661 self.stderr.close()
1662 read_set.remove(self.stderr)
1663 stderr.append(data)
1664
1665 return (stdout, stderr)
1666
1667
1668 def send_signal(self, sig):
1669 """Send a signal to the process
1670 """
1671 os.kill(self.pid, sig)
1672
1673 def terminate(self):
1674 """Terminate the process with SIGTERM
1675 """
1676 self.send_signal(signal.SIGTERM)
1677
1678 def kill(self):
1679 """Kill the process with SIGKILL
1680 """
1681 self.send_signal(signal.SIGKILL)
1682
1683
1684def _demo_posix():
1685 #
1686 # Example 1: Simple redirection: Get process list
1687 #
1688 plist = Popen(["ps"], stdout=PIPE).communicate()[0]
1689 print "Process list:"
1690 print plist
1691
1692 #
1693 # Example 2: Change uid before executing child
1694 #
1695 if os.getuid() == 0:
1696 p = Popen(["id"], preexec_fn=lambda: os.setuid(100))
1697 p.wait()
1698
1699 #
1700 # Example 3: Connecting several subprocesses
1701 #
1702 print "Looking for 'hda'..."
1703 p1 = Popen(["dmesg"], stdout=PIPE)
1704 p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
1705 print repr(p2.communicate()[0])
1706
1707 #
1708 # Example 4: Catch execution error
1709 #
1710 print
1711 print "Trying a weird file..."
1712 try:
1713 print Popen(["/this/path/does/not/exist"]).communicate()
1714 except OSError, e:
1715 if e.errno == errno.ENOENT:
1716 print "The file didn't exist. I thought so..."
1717 print "Child traceback:"
1718 print e.child_traceback
1719 else:
1720 print "Error", e.errno
1721 else:
1722 print >>sys.stderr, "Gosh. No error."
1723
1724
1725def _demo_windows():
1726 #
1727 # Example 1: Connecting several subprocesses
1728 #
1729 print "Looking for 'PROMPT' in set output..."
1730 p1 = Popen("set", stdout=PIPE, shell=True)
1731 p2 = Popen('find "PROMPT"', stdin=p1.stdout, stdout=PIPE)
1732 print repr(p2.communicate()[0])
1733
1734 #
1735 # Example 2: Simple execution of program
1736 #
1737 print "Executing calc..."
1738 p = Popen("calc")
1739 p.wait()
1740
1741
1742def _demo_os2():
1743 #
1744 # Example 1: Simple redirection: Get current process data
1745 #
1746 pdata = Popen(["pstat", "/p:%s" % hex(os.getpid())[2:]], stdout=PIPE, stderr=PIPE).communicate()[0]
1747 print "Current process data:"
1748 print pdata
1749 #
1750 # Example 2: Connecting several subprocesses
1751 #
1752 p1 = Popen(["pstat", "/p:%s" % hex(os.getpid())[2:]], stdout=PIPE)
1753 p2 = Popen(["grep", "\.EXE"], stdin=p1.stdout, stdout=PIPE)
1754 print "Current .EXE:"
1755 print p2.communicate()[0]
1756
1757
1758if __name__ == "__main__":
1759 if mswindows:
1760 _demo_windows()
1761 elif os2:
1762 _demo_os2()
1763 else:
1764 _demo_posix()
Note: See TracBrowser for help on using the repository browser.