bpo-35621: Support running subprocesses in asyncio when loop is executed in…

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread  (GH-14344)

Doc/library/asyncio-policy.rst

@@ -117,6 +117,7 @@ asyncio ships with the following built-in policies:
 
    .. availability:: Windows.
 
+.. _asyncio-watchers:
 
 Process Watchers
 ================
@@ -129,10 +130,11 @@ In asyncio, child processes are created with
 :func:`create_subprocess_exec` and :meth:`loop.subprocess_exec`
 functions.
 
-asyncio defines the :class:`AbstractChildWatcher` abstract base class,
-which child watchers should implement, and has two different
-implementations: :class:`SafeChildWatcher` (configured to be used
-by default) and :class:`FastChildWatcher`.
+asyncio defines the :class:`AbstractChildWatcher` abstract base class, which child
+watchers should implement, and has four different implementations:
+:class:`ThreadedChildWatcher` (configured to be used by default),
+:class:`MultiLoopChildWatcher`, :class:`SafeChildWatcher`, and
+:class:`FastChildWatcher`.
 
 See also the :ref:`Subprocess and Threads <asyncio-subprocess-threads>`
 section.
@@ -184,6 +186,15 @@ implementation used by the asyncio event loop:
 
       Note: loop may be ``None``.
 
+   .. method:: is_active()
+
+      Return ``True`` if the watcher is ready to use.
+
+      Spawning a subprocess with *inactive* current child watcher raises
+      :exc:`RuntimeError`.
+
+      .. versionadded:: 3.8
+
    .. method:: close()
 
       Close the watcher.
@@ -191,16 +202,48 @@ implementation used by the asyncio event loop:
       This method has to be called to ensure that underlying
       resources are cleaned-up.
 
-.. class:: SafeChildWatcher
+.. class:: ThreadedChildWatcher
+
+   This implementation starts a new waiting thread for every subprocess spawn.
+
+   It works reliably even when the asyncio event loop is run in a non-main OS thread.
+
+   There is no noticeable overhead when handling a big number of children (*O(1)* each
+   time a child terminates), but stating a thread per process requires extra memory.
+
+   This watcher is used by default.
+
+   .. versionadded:: 3.8
 
-   This implementation avoids disrupting other code spawning processes
+.. class:: MultiLoopChildWatcher
+
+   This implementation registers a :py:data:`SIGCHLD` signal handler on
+   instantiation. That can break third-party code that installs a custom handler for
+   `SIGCHLD`.  signal).
+
+   The watcher avoids disrupting other code spawning processes
    by polling every process explicitly on a :py:data:`SIGCHLD` signal.
 
-   This is a safe solution but it has a significant overhead when
+   There is no limitation for running subprocesses from different threads once the
+   watcher is installed.
+
+   The solution is safe but it has a significant overhead when
    handling a big number of processes (*O(n)* each time a
    :py:data:`SIGCHLD` is received).
 
-   asyncio uses this safe implementation by default.
+   .. versionadded:: 3.8
+
+.. class:: SafeChildWatcher
+
+   This implementation uses active event loop from the main thread to handle
+   :py:data:`SIGCHLD` signal. If the main thread has no running event loop another
+   thread cannot spawn a subprocess (:exc:`RuntimeError` is raised).
+
+   The watcher avoids disrupting other code spawning processes
+   by polling every process explicitly on a :py:data:`SIGCHLD` signal.
+
+   This solution is as safe as :class:`MultiLoopChildWatcher` and has the same *O(N)*
+   complexity but requires a running event loop in the main thread to work.
 
 .. class:: FastChildWatcher
 
@@ -211,6 +254,9 @@ implementation used by the asyncio event loop:
    There is no noticeable overhead when handling a big number of
    children (*O(1)* each time a child terminates).
 
+   This solution requires a running event loop in the main thread to work, as
+   :class:`SafeChildWatcher`.
+
 
 Custom Policies
 ===============

Doc/library/asyncio-subprocess.rst

@@ -293,18 +293,26 @@ their completion.
 Subprocess and Threads
 ----------------------
 
-Standard asyncio event loop supports running subprocesses from
-different threads, but there are limitations:
+Standard asyncio event loop supports running subprocesses from different threads by
+default.
 
-* An event loop must run in the main thread.
+On Windows subprocesses are provided by :class:`ProactorEventLoop` only (default),
+:class:`SelectorEventLoop` has no subprocess support.
 
-* The child watcher must be instantiated in the main thread
-  before executing subprocesses from other threads. Call the
-  :func:`get_child_watcher` function in the main thread to instantiate
-  the child watcher.
+On UNIX *child watchers* are used for subprocess finish waiting, see
+:ref:`asyncio-watchers` for more info.
 
-Note that alternative event loop implementations might not share
-the above limitations; please refer to their documentation.
+
+.. versionchanged:: 3.8
+
+   UNIX switched to use :class:`ThreadedChildWatcher` for spawning subprocesses from
+   different threads without any limitation.
+
+   Spawning a subprocess with *inactive* current child watcher raises
+   :exc:`RuntimeError`.
+
+Note that alternative event loop implementations might have own limitations;
+please refer to their documentation.
 
 .. seealso::
 

Lib/test/test_asyncio/test_subprocess.py

@@ -633,6 +633,7 @@ class SubprocessMixin:
 
         self.assertIsNone(self.loop.run_until_complete(execute()))
 
+
 if sys.platform != 'win32':
     # Unix
     class SubprocessWatcherMixin(SubprocessMixin):
@@ -648,7 +649,24 @@ if sys.platform != 'win32':
             watcher = self.Watcher()
             watcher.attach_loop(self.loop)
             policy.set_child_watcher(watcher)
-            self.addCleanup(policy.set_child_watcher, None)
+
+        def tearDown(self):
+            super().tearDown()
+            policy = asyncio.get_event_loop_policy()
+            watcher = policy.get_child_watcher()
+            policy.set_child_watcher(None)
+            watcher.attach_loop(None)
+            watcher.close()
+
+    class SubprocessThreadedWatcherTests(SubprocessWatcherMixin,
+                                         test_utils.TestCase):
+
+        Watcher = unix_events.ThreadedChildWatcher
+
+    class SubprocessMultiLoopWatcherTests(SubprocessWatcherMixin,
+                                          test_utils.TestCase):
+
+        Watcher = unix_events.MultiLoopChildWatcher
 
     class SubprocessSafeWatcherTests(SubprocessWatcherMixin,
                                      test_utils.TestCase):
@@ -670,5 +688,25 @@ else:
             self.set_event_loop(self.loop)
 
 
+class GenericWatcherTests:
+
+    def test_create_subprocess_fails_with_inactive_watcher(self):
+
+        async def execute():
+            watcher = mock.create_authspec(asyncio.AbstractChildWatcher)
+            watcher.is_active.return_value = False
+            asyncio.set_child_watcher(watcher)
+
+            with self.assertRaises(RuntimeError):
+                await subprocess.create_subprocess_exec(
+                    support.FakePath(sys.executable), '-c', 'pass')
+
+            watcher.add_child_handler.assert_not_called()
+
+        self.assertIsNone(self.loop.run_until_complete(execute()))
+
+
+
+
 if __name__ == '__main__':
     unittest.main()

Lib/test/test_asyncio/test_unix_events.py

@@ -1082,6 +1082,8 @@ class AbstractChildWatcherTests(unittest.TestCase):
             NotImplementedError, watcher.attach_loop, f)
         self.assertRaises(
             NotImplementedError, watcher.close)
+        self.assertRaises(
+            NotImplementedError, watcher.is_active)
         self.assertRaises(
             NotImplementedError, watcher.__enter__)
         self.assertRaises(
@@ -1784,15 +1786,6 @@ class ChildWatcherTestsMixin:
                 if isinstance(self.watcher, asyncio.FastChildWatcher):
                     self.assertFalse(self.watcher._zombies)
 
-    @waitpid_mocks
-    def test_add_child_handler_with_no_loop_attached(self, m):
-        callback = mock.Mock()
-        with self.create_watcher() as watcher:
-            with self.assertRaisesRegex(
-                    RuntimeError,
-                    'the child watcher does not have a loop attached'):
-                watcher.add_child_handler(100, callback)
-
 
 class SafeChildWatcherTests (ChildWatcherTestsMixin, test_utils.TestCase):
     def create_watcher(self):
@@ -1809,17 +1802,16 @@ class PolicyTests(unittest.TestCase):
     def create_policy(self):
         return asyncio.DefaultEventLoopPolicy()
 
-    def test_get_child_watcher(self):
+    def test_get_default_child_watcher(self):
         policy = self.create_policy()
         self.assertIsNone(policy._watcher)
 
         watcher = policy.get_child_watcher()
-        self.assertIsInstance(watcher, asyncio.SafeChildWatcher)
+        self.assertIsInstance(watcher, asyncio.ThreadedChildWatcher)
 
         self.assertIs(policy._watcher, watcher)
 
         self.assertIs(watcher, policy.get_child_watcher())
-        self.assertIsNone(watcher._loop)
 
     def test_get_child_watcher_after_set(self):
         policy = self.create_policy()
@@ -1829,18 +1821,6 @@ class PolicyTests(unittest.TestCase):
         self.assertIs(policy._watcher, watcher)
         self.assertIs(watcher, policy.get_child_watcher())
 
-    def test_get_child_watcher_with_mainloop_existing(self):
-        policy = self.create_policy()
-        loop = policy.get_event_loop()
-
-        self.assertIsNone(policy._watcher)
-        watcher = policy.get_child_watcher()
-
-        self.assertIsInstance(watcher, asyncio.SafeChildWatcher)
-        self.assertIs(watcher._loop, loop)
-
-        loop.close()
-
     def test_get_child_watcher_thread(self):
 
         def f():
@@ -1866,7 +1846,11 @@ class PolicyTests(unittest.TestCase):
         policy = self.create_policy()
         loop = policy.get_event_loop()
 
-        watcher = policy.get_child_watcher()
+        # Explicitly setup SafeChildWatcher,
+        # default ThreadedChildWatcher has no _loop property
+        watcher = asyncio.SafeChildWatcher()
+        policy.set_child_watcher(watcher)
+        watcher.attach_loop(loop)
 
         self.assertIs(watcher._loop, loop)
 

Lib/test/test_asyncio/utils.py

 """Utilities shared by tests."""
 
+import asyncio
 import collections
 import contextlib
 import io
@@ -512,6 +513,18 @@ class TestCase(unittest.TestCase):
         if executor is not None:
             executor.shutdown(wait=True)
         loop.close()
+        policy = support.maybe_get_event_loop_policy()
+        if policy is not None:
+            try:
+                watcher = policy.get_child_watcher()
+            except NotImplementedError:
+                # watcher is not implemented by EventLoopPolicy, e.g. Windows
+                pass
+            else:
+                if isinstance(watcher, asyncio.ThreadedChildWatcher):
+                    threads = list(watcher._threads.values())
+                    for thread in threads:
+                        thread.join()
 
     def set_event_loop(self, loop, *, cleanup=True):
         assert loop is not None

Misc/NEWS.d/next/Library/2019-05-28-19-03-46.bpo-35621.Abc1lf.rst

+Support running asyncio subprocesses when execution event loop in a thread
+on UNIX.