* Prevent low-grade poplib REDOS (CVE-2018-1060)

The regex to test a mail server's timestamp is susceptible to
catastrophic backtracking on long evil responses from the server.

Happily, the maximum length of malicious inputs is 2K thanks
to a limit introduced in the fix for CVE-2013-1752.

A 2KB evil response from the mail server would result in small slowdowns
(milliseconds vs. microseconds) accumulated over many apop calls.
This is a potential DOS vector via accumulated slowdowns.

Replace it with a similar non-vulnerable regex.

The new regex is RFC compliant.
The old regex was non-compliant in edge cases.

* Prevent difflib REDOS (CVE-2018-1061)

The default regex for IS_LINE_JUNK is susceptible to
catastrophic backtracking.
This is a potential DOS vector.

Replace it with an equivalent non-vulnerable regex.

Also introduce unit and REDOS tests for difflib.
Co-authored-by: Tim Peters <tim.peters@gmail.com>
Co-authored-by: Christian Heimes <christian@python.org>

Some applications (e.g. traditional Unix diff, version control
systems) neither know nor care about the encodings of the files they
are comparing. They are textual, but to the diff utility they are just
bytes. This worked fine under Python 2, because all of the hardcoded
strings in difflib.py are ASCII, so could safely be combined with
old-style u'' strings. But it stopped working in 3.x.

The solution is to use surrogate escapes for a lossless
bytes->str->bytes roundtrip. That means {unified,context}_diff() can
continue to just handle strings without worrying about bytes. Callers
who have to deal with bytes will need to change to using diff_bytes().

Use case: Mercurial's test runner uses difflib to compare current hg
output with known good output. But Mercurial's output is just bytes,
since it can contain:
  * file contents (arbitrary unknown encoding)
  * filenames (arbitrary unknown encoding)
  * usernames and commit messages (usually UTF-8, but not guaranteed
    because old versions of Mercurial did not enforce it)
  * user messages (locale encoding)

Since the output of any given hg command can include text in multiple
encodings, it is hopeless to try to treat it as decodable Unicode
text. It's just bytes, all the way down.

This is an elaboration of a patch by Terry Reedy.

Issue 10534, difflib: tweak doc; test new SequenceMatcher instance attributes; avoid unneeded lists of SM.b2j keys and items in .__chain_b. Do not backport.

Issue 2986: Add autojunk paramater to SequenceMatcher to turn off heuristic. Patch by Terry Reedy, Eli Bendersky, and Simon Cross

svn+ssh://pythondev@svn.python.org/python/trunk

........
  r80004 | r.david.murray | 2010-04-12 12:35:19 -0400 (Mon, 12 Apr 2010) | 13 lines

  Issue #7585: use tab between components in unified and context diff headers.

  Instead of spaces between the filename and date (or whatever the string
  is that follows the filename, if any) use tabs.  This is what the unix
  'diff' command does, for example, and difflib was intended to follow
  the 'standard' way of doing diffs.  This improves compatibility with
  patch tools.  The docs and examples are also changed to recommended that
  the date format used be the ISO 8601 format, which is what modern diff
  tools emit by default.

  Patch by Anatoly Techtonik.
........

Instead of spaces between the filename and date (or whatever the string
is that follows the filename, if any) use tabs.  This is what the unix
'diff' command does, for example, and difflib was intended to follow
the 'standard' way of doing diffs.  This improves compatibility with
patch tools.  The docs and examples are also changed to recommended that
the date format used be the ISO 8601 format, which is what modern diff
tools emit by default.

Patch by Anatoly Techtonik.

svn+ssh://pythondev@svn.python.org/python/branches/py3k

................
  r76469 | senthil.kumaran | 2009-11-24 00:32:52 +0530 (Tue, 24 Nov 2009) | 10 lines

  Merged revisions 76464 via svnmerge from
  svn+ssh://pythondev@svn.python.org/python/trunk

  ........
    r76464 | senthil.kumaran | 2009-11-24 00:11:31 +0530 (Tue, 24 Nov 2009) | 4 lines

    Fix for issue1488943 - difflib.Differ() doesn't always add hints for tab
    characters.
  ........
................

svn+ssh://pythondev@svn.python.org/python/trunk

........
  r76464 | senthil.kumaran | 2009-11-24 00:11:31 +0530 (Tue, 24 Nov 2009) | 4 lines

  Fix for issue1488943 - difflib.Differ() doesn't always add hints for tab
  characters.
........

svn+ssh://pythondev@svn.python.org/python/trunk

........
  r76464 | senthil.kumaran | 2009-11-24 00:11:31 +0530 (Tue, 24 Nov 2009) | 4 lines

  Fix for issue1488943 - difflib.Differ() doesn't always add hints for tab
  characters.
........

characters.

svn+ssh://pythondev@svn.python.org/python/trunk

........
  r72979 | philip.jenvey | 2009-05-27 22:58:44 -0700 (Wed, 27 May 2009) | 2 lines

  explicitly close files
........

number of tests, all because of the codecs/_multibytecodecs issue described
here (it's not a Py3K issue, just something Py3K discovers):
http://mail.python.org/pipermail/python-dev/2006-April/064051.html

Hye-Shik Chang promised to look for a fix, so no need to fix it here. The
tests that are expected to break are:

test_codecencodings_cn
test_codecencodings_hk
test_codecencodings_jp
test_codecencodings_kr
test_codecencodings_tw
test_codecs
test_multibytecodec

This merge fixes an actual test failure (test_weakref) in this branch,
though, so I believe merging is the right thing to do anyway.

touch the recursion limit. The applied patch inlines the recursive
__helper method in a non-recursive way.

test depends on invisible trailing whitespace in .py files.  The author will
have to repair that.

that behaves as if both lists has an empty string in each of them.

Closes bug #979794 (and duplicate bug #980117).

unittest environment.  Since his extension finds docstrings in private
functions, it exposed a bug in the difflib doctests.

Backport candidate

imports e.g. test_support must do so using an absolute package name
such as "import test.test_support" or "from test import test_support".

This also updates the README in Lib/test, and gets rid of the
duplicate data dirctory in Lib/test/data (replaced by
Lib/email/test/data).

Now Tim and Jack can have at it. :)

horridly inefficient hack in regrtest's Compare class, but it's about as
clean as can be:  regrtest has to set up the Compare instance before
importing a test module, and by the time the module *is* imported it's too
late to change that decision.  The good news is that the more tests we
convert to unittest and doctest, the less the inefficiency here matters.
Even now there are few tests with large expected-output files (the new
cost here is a Python-level call per .write() when there's an expected-
output file).

Change test_doctest and test_difflib to pass regrtest's notion of
verbosity on to doctest.
Add explanation for a dozen "new" things to test/README.

generating it.  Since this is purely a doctest, the output file never
served a good purpose.

Guido told me to do this <wink>.
Greatly expanded docstrings, and fleshed out with examples.
New std test.
Added new get_close_matches() function for ESR.
Needs docs, but LaTeXification of the module docstring is all it needs.
\CVS: ----------------------------------------------------------------------