asyncio-subprocess.rst 11 KB
Newer Older
1 2
.. currentmodule:: asyncio

3 4
.. _asyncio-subprocess:

5 6 7
============
Subprocesses
============
8

9 10
This section describes high-level async/await asyncio APIs to
create and manage subprocesses.
11

12 13
.. _asyncio_example_subprocess_shell:

14
Here's an example of how asyncio can run a shell command and
15
obtain its result::
16

17
    import asyncio
18

19 20 21 22 23
    async def run(cmd):
        proc = await asyncio.create_subprocess_shell(
            cmd,
            stdout=asyncio.subprocess.PIPE,
            stderr=asyncio.subprocess.PIPE)
24

25
        stdout, stderr = await proc.communicate()
26

27 28 29 30 31
        print(f'[{cmd!r} exited with {proc.returncode}]')
        if stdout:
            print(f'[stdout]\n{stdout.decode()}')
        if stderr:
            print(f'[stderr]\n{stderr.decode()}')
32

33
    asyncio.run(run('ls /zzz'))
34

35
will print::
36

37 38 39
    ['ls /zzz' exited with 1]
    [stderr]
    ls: /zzz: No such file or directory
40

41 42 43
Because all asyncio subprocess functions are asynchronous and asyncio
provides many tools to work with such functions, it is easy to execute
and monitor multiple subprocesses in parallel.  It is indeed trivial
44
to modify the above example to run several commands simultaneously::
45

46 47 48 49
    async def main():
        await asyncio.gather(
            run('ls /zzz'),
            run('sleep 1; echo "hello"'))
50

51
    asyncio.run(main())
52

53
See also the `Examples`_ subsection.
54

55

56 57
Creating Subprocesses
=====================
58

59
.. coroutinefunction:: create_subprocess_exec(program, \*args, stdin=None, \
60 61
                          stdout=None, stderr=None, loop=None, \
                          limit=None, \*\*kwds)
62

63
   Create a subprocess.
64

65 66
   The *limit* argument sets the buffer limit for :class:`StreamReader`
   wrappers for :attr:`Process.stdout` and :attr:`Process.stderr`
67
   (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments).
68

69
   Return a :class:`~asyncio.subprocess.Process` instance.
70

71 72
   See the documentation of :meth:`loop.subprocess_exec` for other
   parameters.
73

74 75 76
.. coroutinefunction:: create_subprocess_shell(cmd, stdin=None, \
                          stdout=None, stderr=None, loop=None, \
                          limit=None, \*\*kwds)
77

78
   Run the *cmd* shell command.
79

80 81
   The *limit* argument sets the buffer limit for :class:`StreamReader`
   wrappers for :attr:`Process.stdout` and :attr:`Process.stderr`
82
   (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments).
83

84
   Return a :class:`~asyncio.subprocess.Process` instance.
85

86 87
   See the documentation of :meth:`loop.subprocess_shell` for other
   parameters.
88

89
.. important::
90

91
   It is the application's responsibility to ensure that all whitespace and
92
   special characters are quoted appropriately to avoid `shell injection
93
   <https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_
94
   vulnerabilities. The :func:`shlex.quote` function can be used to properly
95 96
   escape whitespace and special shell characters in strings that are going
   to be used to construct shell commands.
97

98 99
.. note::

100 101 102
   The default asyncio event loop implementation on **Windows** does not
   support subprocesses. Subprocesses are available for Windows if a
   :class:`ProactorEventLoop` is used.
103 104
   See :ref:`Subprocess Support on Windows <asyncio-windows-subprocess>`
   for details.
105

106 107
.. seealso::

108
   asyncio also has the following *low-level* APIs to work with subprocesses:
109 110 111 112
   :meth:`loop.subprocess_exec`, :meth:`loop.subprocess_shell`,
   :meth:`loop.connect_read_pipe`, :meth:`loop.connect_write_pipe`,
   as well as the :ref:`Subprocess Transports <asyncio-subprocess-transports>`
   and :ref:`Subprocess Protocols <asyncio-subprocess-protocols>`.
113 114 115


Constants
116
=========
117 118 119

.. data:: asyncio.subprocess.PIPE

120 121 122 123 124 125 126 127 128 129
   Can be passed to the *stdin*, *stdout* or *stderr* parameters.

   If *PIPE* is passed to *stdin* argument, the
   :attr:`Process.stdin <asyncio.subprocess.Process.stdin>` attribute
   will point to a :class:`StreamWriter` instance.

   If *PIPE* is passed to *stdout* or *stderr* arguments, the
   :attr:`Process.stdout <asyncio.subprocess.Process.stdout>` and
   :attr:`Process.stderr <asyncio.subprocess.Process.stderr>`
   attributes will point to :class:`StreamReader` instances.
130 131 132

.. data:: asyncio.subprocess.STDOUT

133 134
   Special value that can be used as the *stderr* argument and indicates
   that standard error should be redirected into standard output.
135 136 137

.. data:: asyncio.subprocess.DEVNULL

138 139 140
   Special value that can be used as the *stdin*, *stdout* or *stderr* argument
   to process creation functions.  It indicates that the special file
   :data:`os.devnull` will be used for the corresponding subprocess stream.
141 142


143 144 145 146
Interacting with Subprocesses
=============================

Both :func:`create_subprocess_exec` and :func:`create_subprocess_shell`
147 148 149
functions return instances of the *Process* class.  *Process* is a high-level
wrapper that allows communicating with subprocesses and watching for
their completion.
150 151 152

.. class:: asyncio.subprocess.Process

153 154 155 156 157 158 159 160 161 162 163 164
   An object that wraps OS processes created by the
   :func:`create_subprocess_exec` and :func:`create_subprocess_shell`
   functions.

   This class is designed to have a similar API to the
   :class:`subprocess.Popen` class, but there are some
   notable differences:

   * unlike Popen, Process instances do not have an equivalent to
     the :meth:`~subprocess.Popen.poll` method;

   * the :meth:`~asyncio.subprocess.Process.communicate` and
165
     :meth:`~asyncio.subprocess.Process.wait` methods don't have a
166 167 168 169 170
     *timeout* parameter: use the :func:`wait_for` function;

   * the :meth:`Process.wait() <asyncio.subprocess.Process.wait>` method
     is asynchronous, whereas :meth:`subprocess.Popen.wait` method
     is implemented as a blocking busy loop;
171

172
   * the *universal_newlines* parameter is not supported.
173

174
   This class is :ref:`not thread safe <asyncio-multithreading>`.
175

176 177
   See also the :ref:`Subprocess and Threads <asyncio-subprocess-threads>`
   section.
178

179
   .. coroutinemethod:: wait()
180

181
      Wait for the child process to terminate.
182

183
      Set and return the :attr:`returncode` attribute.
184

185
      .. note::
186

187 188 189 190 191
         This method can deadlock when using ``stdout=PIPE`` or
         ``stderr=PIPE`` and the child process generates so much output
         that it blocks waiting for the OS pipe buffer to accept
         more data. Use the :meth:`communicate` method when using pipes
         to avoid this condition.
192

193
   .. coroutinemethod:: communicate(input=None)
194

195
      Interact with process:
196

197 198 199
      1. send data to *stdin* (if *input* is not ``None``);
      2. read data from *stdout* and *stderr*, until EOF is reached;
      3. wait for process to terminate.
200

201 202
      The optional *input* argument is the data (:class:`bytes` object)
      that will be sent to the child process.
203

204
      Return a tuple ``(stdout_data, stderr_data)``.
205

206 207 208 209
      If either :exc:`BrokenPipeError` or :exc:`ConnectionResetError`
      exception is raised when writing *input* into *stdin*, the
      exception is ignored.  This condition occurs when the process
      exits before all data are written into *stdin*.
210

211
      If it is desired to send data to the process' *stdin*,
212 213 214 215
      the process needs to be created with ``stdin=PIPE``.  Similarly,
      to get anything other than ``None`` in the result tuple, the
      process has to be created with ``stdout=PIPE`` and/or
      ``stderr=PIPE`` arguments.
216

217 218
      Note, that the data read is buffered in memory, so do not use
      this method if the data size is large or unlimited.
219

220
   .. method:: send_signal(signal)
221 222 223 224 225 226 227 228 229 230 231 232

      Sends the signal *signal* to the child process.

      .. note::

         On Windows, :py:data:`SIGTERM` is an alias for :meth:`terminate`.
         ``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` can be sent to processes
         started with a *creationflags* parameter which includes
         ``CREATE_NEW_PROCESS_GROUP``.

   .. method:: terminate()

233
      Stop the child process.
234

235
      On POSIX systems this method sends :py:data:`signal.SIGTERM` to the
236 237 238 239
      child process.

      On Windows the Win32 API function :c:func:`TerminateProcess` is
      called to stop the child process.
240

241
   .. method:: kill()
242

243 244
      Kill the child.

245
      On POSIX systems this method sends :py:data:`SIGKILL` to the child
246 247 248
      process.

      On Windows this method is an alias for :meth:`terminate`.
249

250
   .. attribute:: stdin
251

252 253
      Standard input stream (:class:`StreamWriter`) or ``None``
      if the process was created with ``stdin=None``.
254

255 256
   .. attribute:: stdout

257 258
      Standard output stream (:class:`StreamReader`) or ``None``
      if the process was created with ``stdout=None``.
259 260 261

   .. attribute:: stderr

262 263
      Standard error stream (:class:`StreamReader`) or ``None``
      if the process was created with ``stderr=None``.
264 265 266

   .. warning::

267 268 269
      Use the :meth:`communicate` method rather than
      :attr:`process.stdin.write() <stdin>`,
      :attr:`await process.stdout.read() <stdout>` or
270 271
      :attr:`await process.stderr.read <stderr>`.
      This avoids deadlocks due to streams pausing reading or writing
272
      and blocking the child process.
273 274 275

   .. attribute:: pid

276
      Process identification number (PID).
277 278

      Note that for processes created by the :func:`create_subprocess_shell`
279
      function, this attribute is the PID of the spawned shell.
280 281 282

   .. attribute:: returncode

283 284 285
      Return code of the process when it exits.

      A ``None`` value indicates that the process has not terminated yet.
286

287
      A negative value ``-N`` indicates that the child was terminated
288
      by signal ``N`` (POSIX only).
289 290


291 292
.. _asyncio-subprocess-threads:

293
Subprocess and Threads
294
----------------------
295

296 297
Standard asyncio event loop supports running subprocesses from
different threads, but there are limitations:
298

299
* An event loop must run in the main thread.
300

301
* The child watcher must be instantiated in the main thread
302 303 304 305
  before executing subprocesses from other threads. Call the
  :func:`get_child_watcher` function in the main thread to instantiate
  the child watcher.

306
Note that alternative event loop implementations might not share
307
the above limitations; please refer to their documentation.
308

309 310 311 312 313 314
.. seealso::

   The :ref:`Concurrency and multithreading in asyncio
   <asyncio-multithreading>` section.


315 316
Examples
--------
317

318 319
An example using the :class:`~asyncio.subprocess.Process` class to
control a subprocess and the :class:`StreamReader` class to read from
320
its standard output.
321

322 323
.. _asyncio_example_create_subprocess_exec:

324
The subprocess is created by the :func:`create_subprocess_exec`
325 326
function::

327
    import asyncio
328 329
    import sys

330
    async def get_date():
331 332
        code = 'import datetime; print(datetime.datetime.now())'

333 334
        # Create the subprocess; redirect the standard output
        # into a pipe.
335 336 337
        proc = await asyncio.create_subprocess_exec(
            sys.executable, '-c', code,
            stdout=asyncio.subprocess.PIPE)
338

339
        # Read one line of output.
340
        data = await proc.stdout.readline()
341 342
        line = data.decode('ascii').rstrip()

343
        # Wait for the subprocess exit.
344
        await proc.wait()
345 346
        return line

347 348 349 350
    date = asyncio.run(get_date())
    print(f"Current date: {date}")


351
See also the :ref:`same example <asyncio_example_subprocess_proto>`
352
written using low-level APIs.