merge heads

Doc/library/doctest.rst

+:keepdoctest:
+
 :mod:`doctest` --- Test interactive Python examples
 ===================================================
 
@@ -318,7 +320,8 @@ The fine print:
   Tabs in output generated by the tested code are not modified.  Because any
   hard tabs in the sample output *are* expanded, this means that if the code
   output includes hard tabs, the only way the doctest can pass is if the
-  :const:`NORMALIZE_WHITESPACE` option or directive is in effect.
+  :const:`NORMALIZE_WHITESPACE` option or :ref:`directive <doctest-directives>`
+  is in effect.
   Alternatively, the test can be rewritten to capture the output and compare it
   to an expected value as part of the test.  This handling of tabs in the
   source was arrived at through trial and error, and has proven to be the least
@@ -483,15 +486,16 @@ Some details you should read once, but won't need to remember:
      SyntaxError: invalid syntax
 
 
+.. _option-flags-and-directives:
 .. _doctest-options:
 
-Option Flags and Directives
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Option Flags
+^^^^^^^^^^^^
 
 A number of option flags control various aspects of doctest's behavior.
 Symbolic names for the flags are supplied as module constants, which can be
 or'ed together and passed to various functions.  The names can also be used in
-doctest directives (see below).
+:ref:`doctest directives <doctest-directives>`.
 
 The first group of options define test semantics, controlling aspects of how
 doctest decides whether actual output matches an example's expected output:
@@ -545,14 +549,14 @@ doctest decides whether actual output matches an example's expected output:
    :exc:`TypeError` is raised.
 
    It will also ignore the module name used in Python 3 doctest reports. Hence
-   both these variations will work regardless of whether the test is run under
-   Python 2.7 or Python 3.2 (or later versions):
+   both of these variations will work with the flag specified, regardless of
+   whether the test is run under Python 2.7 or Python 3.2 (or later versions)::
 
-      >>> raise CustomError('message') #doctest: +IGNORE_EXCEPTION_DETAIL
+      >>> raise CustomError('message')
       Traceback (most recent call last):
       CustomError: message
 
-      >>> raise CustomError('message') #doctest: +IGNORE_EXCEPTION_DETAIL
+      >>> raise CustomError('message')
       Traceback (most recent call last):
       my_module.CustomError: message
 
@@ -562,15 +566,16 @@ doctest decides whether actual output matches an example's expected output:
    exception name. Using :const:`IGNORE_EXCEPTION_DETAIL` and the details
    from Python 2.3 is also the only clear way to write a doctest that doesn't
    care about the exception detail yet continues to pass under Python 2.3 or
-   earlier (those releases do not support doctest directives and ignore them
-   as irrelevant comments). For example, ::
+   earlier (those releases do not support :ref:`doctest directives
+   <doctest-directives>` and ignore them as irrelevant comments). For example::
 
-      >>> (1, 2)[3] = 'moo' #doctest: +IGNORE_EXCEPTION_DETAIL
+      >>> (1, 2)[3] = 'moo'
       Traceback (most recent call last):
         File "<stdin>", line 1, in ?
       TypeError: object doesn't support item assignment
 
-   passes under Python 2.3 and later Python versions, even though the detail
+   passes under Python 2.3 and later Python versions with the flag specified,
+   even though the detail
    changed in Python 2.4 to say "does not" instead of "doesn't".
 
    .. versionchanged:: 3.2
@@ -632,9 +637,30 @@ The second group of options controls how test failures are reported:
 
    A bitmask or'ing together all the reporting flags above.
 
-"Doctest directives" may be used to modify the option flags for individual
-examples.  Doctest directives are expressed as a special Python comment
-following an example's source code:
+
+There is also a way to register new option flag names, though this isn't
+useful unless you intend to extend :mod:`doctest` internals via subclassing:
+
+
+.. function:: register_optionflag(name)
+
+   Create a new option flag with a given name, and return the new flag's integer
+   value.  :func:`register_optionflag` can be used when subclassing
+   :class:`OutputChecker` or :class:`DocTestRunner` to create new options that are
+   supported by your subclasses.  :func:`register_optionflag` should always be
+   called using the following idiom::
+
+      MY_FLAG = register_optionflag('MY_FLAG')
+
+
+.. _doctest-directives:
+
+Directives
+^^^^^^^^^^
+
+Doctest directives may be used to modify the :ref:`option flags
+<doctest-options>` for an individual example.  Doctest directives are
+special Python comments following an example's source code:
 
 .. productionlist:: doctest
    directive: "#" "doctest:" `directive_options`
@@ -650,43 +676,28 @@ above.
 An example's doctest directives modify doctest's behavior for that single
 example.  Use ``+`` to enable the named behavior, or ``-`` to disable it.
 
-.. note::
-   Due to an `unfortunate limitation`_ of our current documentation
-   publishing process, syntax highlighting has been disabled in the examples
-   below in order to ensure the doctest directives are correctly displayed.
-
-   .. _unfortunate limitation: http://bugs.python.org/issue12947
+For example, this test passes::
 
-For example, this test passes:
-
-.. code-block:: text
-
-   >>> print(list(range(20))) #doctest: +NORMALIZE_WHITESPACE
+   >>> print(list(range(20))) # doctest: +NORMALIZE_WHITESPACE
    [0,   1,  2,  3,  4,  5,  6,  7,  8,  9,
    10,  11, 12, 13, 14, 15, 16, 17, 18, 19]
 
 Without the directive it would fail, both because the actual output doesn't have
 two blanks before the single-digit list elements, and because the actual output
 is on a single line.  This test also passes, and also requires a directive to do
-so:
-
-.. code-block:: text
+so::
 
    >>> print(list(range(20))) # doctest: +ELLIPSIS
    [0, 1, ..., 18, 19]
 
 Multiple directives can be used on a single physical line, separated by
-commas:
-
-.. code-block:: text
+commas::
 
    >>> print(list(range(20))) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
    [0,    1, ...,   18,    19]
 
 If multiple directive comments are used for a single example, then they are
-combined:
-
-.. code-block:: text
+combined::
 
    >>> print(list(range(20))) # doctest: +ELLIPSIS
    ...                        # doctest: +NORMALIZE_WHITESPACE
@@ -694,9 +705,7 @@ combined:
 
 As the previous example shows, you can add ``...`` lines to your example
 containing only directives.  This can be useful when an example is too long for
-a directive to comfortably fit on the same line:
-
-.. code-block:: text
+a directive to comfortably fit on the same line::
 
    >>> print(list(range(5)) + list(range(10, 20)) + list(range(30, 40)))
    ... # doctest: +ELLIPSIS
@@ -708,20 +717,6 @@ usually the only meaningful choice.  However, option flags can also be passed to
 functions that run doctests, establishing different defaults.  In such cases,
 disabling an option via ``-`` in a directive can be useful.
 
-There's also a way to register new option flag names, although this isn't useful
-unless you intend to extend :mod:`doctest` internals via subclassing:
-
-
-.. function:: register_optionflag(name)
-
-   Create a new option flag with a given name, and return the new flag's integer
-   value.  :func:`register_optionflag` can be used when subclassing
-   :class:`OutputChecker` or :class:`DocTestRunner` to create new options that are
-   supported by your subclasses.  :func:`register_optionflag` should always be
-   called using the following idiom::
-
-      MY_FLAG = register_optionflag('MY_FLAG')
-
 
 .. _doctest-warnings:
 

Doc/tools/sphinxext/pyspecific.py

@@ -33,9 +33,38 @@ def new_visit_versionmodified(self, node):
     self.body.append('<span class="versionmodified">%s</span> ' % text)
 
 from sphinx.writers.html import HTMLTranslator
+from sphinx.writers.latex import LaTeXTranslator
 from sphinx.locale import versionlabels
 HTMLTranslator.visit_versionmodified = new_visit_versionmodified
+HTMLTranslator.visit_versionmodified = new_visit_versionmodified
+
+# monkey-patch HTML and LaTeX translators to keep doctest blocks in the
+# doctest docs themselves
+orig_visit_literal_block = HTMLTranslator.visit_literal_block
+def new_visit_literal_block(self, node):
+    meta = self.builder.env.metadata[self.builder.current_docname]
+    old_trim_doctest_flags = self.highlighter.trim_doctest_flags
+    if 'keepdoctest' in meta:
+        self.highlighter.trim_doctest_flags = False
+    try:
+        orig_visit_literal_block(self, node)
+    finally:
+        self.highlighter.trim_doctest_flags = old_trim_doctest_flags
+
+HTMLTranslator.visit_literal_block = new_visit_literal_block
+
+orig_depart_literal_block = LaTeXTranslator.depart_literal_block
+def new_depart_literal_block(self, node):
+    meta = self.builder.env.metadata[self.curfilestack[-1]]
+    old_trim_doctest_flags = self.highlighter.trim_doctest_flags
+    if 'keepdoctest' in meta:
+        self.highlighter.trim_doctest_flags = False
+    try:
+        orig_depart_literal_block(self, node)
+    finally:
+        self.highlighter.trim_doctest_flags = old_trim_doctest_flags
 
+LaTeXTranslator.depart_literal_block = new_depart_literal_block
 
 # Support for marking up and linking to bugs.python.org issues
 

Lib/subprocess.py

@@ -1327,6 +1327,7 @@ class Popen(object):
 
             if executable is None:
                 executable = args[0]
+            orig_executable = executable
 
             # For transferring possible exec failure from child to parent.
             # Data format: "exception name:hex errno:description"
@@ -1409,10 +1410,17 @@ class Popen(object):
                 err_msg = err_msg.decode(errors="surrogatepass")
                 if issubclass(child_exception_type, OSError) and hex_errno:
                     errno_num = int(hex_errno, 16)
+                    child_exec_never_called = (err_msg == "noexec")
+                    if child_exec_never_called:
+                        err_msg = ""
                     if errno_num != 0:
                         err_msg = os.strerror(errno_num)
                         if errno_num == errno.ENOENT:
-                            err_msg += ': ' + repr(args[0])
+                            if child_exec_never_called:
+                                # The error must be from chdir(cwd).
+                                err_msg += ': ' + repr(cwd)
+                            else:
+                                err_msg += ': ' + repr(orig_executable)
                     raise child_exception_type(errno_num, err_msg)
                 raise child_exception_type(err_msg)
 

Lib/test/test_subprocess.py

@@ -200,13 +200,16 @@ class ProcessTestCase(BaseTestCase):
         p.wait()
         self.assertEqual(47, p.returncode)
 
-    # TODO: make this test work on Linux.
-    # This may be failing on Linux because of issue #7774.
-    @unittest.skipIf(sys.platform not in ('win32', 'darwin'),
-                     "possible bug using executable argument on Linux")
     def test_executable(self):
         # Check that the executable argument works.
-        self._assert_python(["doesnotexist", "-c"], executable=sys.executable)
+        #
+        # On Unix (non-Mac and non-Windows), Python looks at args[0] to
+        # determine where its standard library is, so we need the directory
+        # of args[0] to be valid for the Popen() call to Python to succeed.
+        # See also issue #16170 and issue #7774.
+        doesnotexist = os.path.join(os.path.dirname(sys.executable),
+                                    "doesnotexist")
+        self._assert_python([doesnotexist, "-c"], executable=sys.executable)
 
     def test_executable_takes_precedence(self):
         # Check that the executable argument takes precedence over args[0].
@@ -1035,24 +1038,30 @@ class _SuppressCoreFiles(object):
 @unittest.skipIf(mswindows, "POSIX specific tests")
 class POSIXProcessTestCase(BaseTestCase):
 
-    def test_exceptions(self):
-        nonexistent_dir = "/_this/pa.th/does/not/exist"
+    def setUp(self):
+        super().setUp()
+        self._nonexistent_dir = "/_this/pa.th/does/not/exist"
+
+    def _get_chdir_exception(self):
         try:
-            os.chdir(nonexistent_dir)
+            os.chdir(self._nonexistent_dir)
         except OSError as e:
             # This avoids hard coding the errno value or the OS perror()
             # string and instead capture the exception that we want to see
             # below for comparison.
             desired_exception = e
-            desired_exception.strerror += ': ' + repr(sys.executable)
+            desired_exception.strerror += ': ' + repr(self._nonexistent_dir)
         else:
             self.fail("chdir to nonexistant directory %s succeeded." %
-                      nonexistent_dir)
+                      self._nonexistent_dir)
+        return desired_exception
 
-        # Error in the child re-raised in the parent.
+    def test_exception_cwd(self):
+        """Test error in the child raised in the parent for a bad cwd."""
+        desired_exception = self._get_chdir_exception()
         try:
             p = subprocess.Popen([sys.executable, "-c", ""],
-                                 cwd=nonexistent_dir)
+                                 cwd=self._nonexistent_dir)
         except OSError as e:
             # Test that the child process chdir failure actually makes
             # it up to the parent process as the correct exception.
@@ -1061,6 +1070,33 @@ class POSIXProcessTestCase(BaseTestCase):
         else:
             self.fail("Expected OSError: %s" % desired_exception)
 
+    def test_exception_bad_executable(self):
+        """Test error in the child raised in the parent for a bad executable."""
+        desired_exception = self._get_chdir_exception()
+        try:
+            p = subprocess.Popen([sys.executable, "-c", ""],
+                                 executable=self._nonexistent_dir)
+        except OSError as e:
+            # Test that the child process exec failure actually makes
+            # it up to the parent process as the correct exception.
+            self.assertEqual(desired_exception.errno, e.errno)
+            self.assertEqual(desired_exception.strerror, e.strerror)
+        else:
+            self.fail("Expected OSError: %s" % desired_exception)
+
+    def test_exception_bad_args_0(self):
+        """Test error in the child raised in the parent for a bad args[0]."""
+        desired_exception = self._get_chdir_exception()
+        try:
+            p = subprocess.Popen([self._nonexistent_dir, "-c", ""])
+        except OSError as e:
+            # Test that the child process exec failure actually makes
+            # it up to the parent process as the correct exception.
+            self.assertEqual(desired_exception.errno, e.errno)
+            self.assertEqual(desired_exception.strerror, e.strerror)
+        else:
+            self.fail("Expected OSError: %s" % desired_exception)
+
     def test_restore_signals(self):
         # Code coverage for both values of restore_signals to make sure it
         # at least does not blow up.

Misc/NEWS

@@ -42,6 +42,10 @@ Core and Builtins
 Library
 -------
 
+- Issue #16114: The subprocess module no longer provides a misleading error
+  message stating that args[0] did not exist when either the cwd or executable
+  keyword arguments specified a path that did not exist.
+
 - Issue #16169: Fix ctypes.WinError()'s confusion between errno and winerror.
 
 - Issue #1492704: shutil.copyfile() raises a distinct SameFileError now if

Modules/_posixsubprocess.c

@@ -356,7 +356,7 @@ child_exec(char *const exec_array[],
            PyObject *preexec_fn,
            PyObject *preexec_fn_args_tuple)
 {
-    int i, saved_errno, unused;
+    int i, saved_errno, unused, reached_preexec = 0;
     PyObject *result;
     const char* err_msg = "";
     /* Buffer large enough to hold a hex integer.  We can't malloc. */
@@ -440,6 +440,7 @@ child_exec(char *const exec_array[],
         POSIX_CALL(setsid());
 #endif
 
+    reached_preexec = 1;
     if (preexec_fn != Py_None && preexec_fn_args_tuple) {
         /* This is where the user has asked us to deadlock their program. */
         result = PyObject_Call(preexec_fn, preexec_fn_args_tuple, NULL);
@@ -489,6 +490,10 @@ error:
         }
         unused = write(errpipe_write, cur, hex_errno + sizeof(hex_errno) - cur);
         unused = write(errpipe_write, ":", 1);
+        if (!reached_preexec) {
+            /* Indicate to the parent that the error happened before exec(). */
+            unused = write(errpipe_write, "noexec", 6);
+        }
         /* We can't call strerror(saved_errno).  It is not async signal safe.
          * The parent process will look the error message up. */
     } else {