(cherry picked from commit aba24ff3)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

(cherry picked from commit 993030aa)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

PyErr_Print() will not return when the exception is a SystemExit, so
decref the __main__ module object in that case.
(cherry picked from commit d8cba5d1)
Co-authored-by: Zackery Spytz <zspytz@gmail.com>

Directly executed pyc files were being kept open longer than necessary.
(cherry picked from commit ea737751)
Co-authored-by: Zackery Spytz <zspytz@gmail.com>

Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.

While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:

- The core changes to importlib to understand how to read, validate, and
  regenerate hash-based pycs.

- Support for generating hash-based pycs in py_compile and compileall.

- Modifications to our siphash implementation to support passing a custom
  key. We then expose it to importlib through _imp.

- Updates to all places in the interpreter, standard library, and tests that
  manually generate or parse pyc files to grok the new format.

- Support in the interpreter command line code for long options like
  --check-hash-based-pycs.

- Tests and documentation for all of the above.

Py_Main() now handles two more -X options:

* -X showrefcount: new _PyCoreConfig.show_ref_count field
* -X showalloccount: new _PyCoreConfig.show_alloc_count field

Resolves bpo-31420.

(This was accidentally reverted when in #3565.)

PR #1638, for bpo-28411, causes problems in some (very) edge cases. Until that gets sorted out, we're reverting the merge. PR #3506, a fix on top of #1638, is also getting reverted.

* Drop warnoptions from PyInterpreterState.

* Drop xoptions from PyInterpreterState.

* Don't set warnoptions and _xoptions again.

* Decref after adding to sys.__dict__.

* Drop an unused macro.

* Check sys.xoptions *before* we delete it.

* group the (stateful) runtime globals into various topical structs
* consolidate the topical structs under a single top-level _PyRuntimeState struct
* add a check-c-globals.py script that helps identify runtime globals

Other globals are excluded (see globals.txt and check-c-globals.py).

(patch by Nick Coghlan)

bltinmodule.c: Added in b744ba1d and no longer necessary since d64e8a75
posixmodule.c: Added in d1cd4d41 and no longer necessary since efb00c0c
pythonrun.c:   Added in 73d538b9 and no longer necessary since d6009517
sysmodule.c:   Added in 5467d4c0 and no longer necessary since a2c17c58

Based on patch by Michael Layzell.

UTF-8 represenatation of Unicode objects.

with PyUnicode_AsUTF8 and PyUnicode_AsUTF8AndSize.

The latter function is more readable, faster and doesn't raise exceptions.

Based on patch by Xiang Zhang.

Directly pass NULL rather than an empty string.

Issue #27809:

* Rename _PyObject_FastCall() function to _PyObject_FastCallDict()
* Add _PyObject_FastCall(), _PyObject_CallNoArg() and _PyObject_CallArg1()
  macros calling _PyObject_FastCallDict()

Issue #27128.

Most fixes to Doc/ and Lib/ directories by Ville Skyttä.

It is very unlikely that they can occur in real code for now.

Issue #24891: Fix a race condition at Python startup if the file descriptor
of stdin (0), stdout (1) or stderr (2) is closed while Python is creating
sys.stdin, sys.stdout and sys.stderr objects. These attributes are now set
to None if the creation of the object failed, instead of raising an OSError
exception. Initial patch written by Marco Paolini.

Patch by Louis Dassy.

The concept of .pyo files no longer exists. Now .pyc files have an
optional `opt-` tag which specifies if any extra optimizations beyond
the peepholer were applied.

Flushing sys.stdout and sys.stderr in Py_FatalError() can call again
Py_FatalError(). Add a reentrant flag to detect this case and just abort at the
second call.

It should help to see exceptions when stderr if buffered: PyErr_Display() calls
sys.stderr.write(), it doesn't write into stderr file descriptor directly.

* Display the current Python stack if an exception was raised but the exception
  has no traceback
* Disable faulthandler if an exception was raised (before it was only disabled
  if no exception was raised)
* To display the current Python stack, call PyGILState_GetThisThreadState()
  which works even if the GIL was released

Detect also earlier PyMarshal_Read*() errors in zipimport.

- interpreter startup and shutdown code moved to a new
  pylifecycle.c module
- Py_OptimizeFlag moved into the new module with the other
  global flags

MAXPATHLEN is now preferred.