Fix two (in my opinion) spurious failure conditions in the lib2to3.tests.test_parser.TestParserIdempotency test_parser test.

    Use the same encoding found in the initial file to write a temp file for a diff. This retains the BOM if the encoding was initially utf-8-sig.

    If the file cannot be parsed using the normal grammar, try again with no print statement which should succeed for valid files using future print_function

For case (1), the driver was correctly handling a BOM in a utf-8 file, but then the test was not writing a comparison file using 'utf-8-sig' to diff against, so the BOM got removed. I don't think that is the fault of the parser, and lib2to3 will retain the BOM.

For case (2), lib2to3 pre-detects the use of from __future__ import print_function or allows the user to force this interpretation with a -p flag, and then selects a different grammar with the print statement removed. That makes the test cases unfair to this test as the driver itself doesn't know which grammar to use. As a minimal fix, the test will try using a grammar with the print statement, and if that fails fall back on a grammar without it. A more thorough handling of the idempotency test would to be to parse all files using both grammars and ignore if one of the two failed but otherwise check both. I didn't think this was necessary but can change.

between runs given the same Grammar.txt input regardless of the hash
randomization setting.

svn+ssh://pythondev@svn.python.org/python/trunk

................
  r73370 | benjamin.peterson | 2009-06-11 17:06:46 -0500 (Thu, 11 Jun 2009) | 105 lines

  Merged revisions 72523,72950-72951,72994,73003,73033,73036-73040,73091-73093,73096,73179-73181,73192,73231,73244,73255-73256,73365 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r72523 | benjamin.peterson | 2009-05-09 14:42:26 -0500 (Sat, 09 May 2009) | 1 line

    remove parenthesis
  ........
    r72950 | benjamin.peterson | 2009-05-26 18:19:45 -0500 (Tue, 26 May 2009) | 1 line

    remove unused imports
  ........
    r72951 | benjamin.peterson | 2009-05-26 18:27:00 -0500 (Tue, 26 May 2009) | 1 line

    this is no longer executable
  ........
    r72994 | benjamin.peterson | 2009-05-28 15:32:54 -0500 (Thu, 28 May 2009) | 1 line

    fix test_all_fixers on Windows #6134
  ........
    r73003 | benjamin.peterson | 2009-05-28 21:57:28 -0500 (Thu, 28 May 2009) | 4 lines

    make 2to3 test utilities easier to use with other applications (3to2)

    Patch by Joe Amenta
  ........
    r73033 | benjamin.peterson | 2009-05-29 16:58:32 -0500 (Fri, 29 May 2009) | 1 line

    update grammar for multi with statement
  ........
    r73036 | benjamin.peterson | 2009-05-29 17:33:20 -0500 (Fri, 29 May 2009) | 1 line

    simplify fix_unicode
  ........
    r73037 | benjamin.peterson | 2009-05-29 17:53:03 -0500 (Fri, 29 May 2009) | 1 line

    add custom error for pattern syntax errors
  ........
    r73038 | benjamin.peterson | 2009-05-29 17:55:00 -0500 (Fri, 29 May 2009) | 1 line

    complain if details are attached to a token
  ........
    r73039 | benjamin.peterson | 2009-05-29 18:00:28 -0500 (Fri, 29 May 2009) | 1 line

    add a test for whitespace
  ........
    r73040 | benjamin.peterson | 2009-05-29 18:01:17 -0500 (Fri, 29 May 2009) | 1 line

    a fix for emacs highlighting
  ........
    r73091 | benjamin.peterson | 2009-05-31 20:55:25 -0500 (Sun, 31 May 2009) | 1 line

    deprecate set_prefix() and get_prefix() in favor of a prefix property
  ........
    r73092 | benjamin.peterson | 2009-05-31 21:00:51 -0500 (Sun, 31 May 2009) | 1 line

    change hideous java naming scheme
  ........
    r73093 | benjamin.peterson | 2009-05-31 21:01:39 -0500 (Sun, 31 May 2009) | 1 line

    remove dated comment
  ........
    r73096 | benjamin.peterson | 2009-05-31 21:40:53 -0500 (Sun, 31 May 2009) | 1 line

    group tests
  ........
    r73179 | benjamin.peterson | 2009-06-03 13:09:53 -0500 (Wed, 03 Jun 2009) | 1 line

    handle the case where there's multiple trailers #6185
  ........
    r73180 | benjamin.peterson | 2009-06-03 13:18:05 -0500 (Wed, 03 Jun 2009) | 1 line

    scrap __main__ section
  ........
    r73181 | benjamin.peterson | 2009-06-03 13:24:48 -0500 (Wed, 03 Jun 2009) | 1 line

    remove shebang lines and __main__ sections
  ........
    r73192 | benjamin.peterson | 2009-06-03 19:16:30 -0500 (Wed, 03 Jun 2009) | 4 lines

    actually test something here

    Thanks to Joe Amenta for noticing.y
  ........
    r73231 | benjamin.peterson | 2009-06-04 13:38:50 -0500 (Thu, 04 Jun 2009) | 1 line

    remove unused variable
  ........
    r73244 | benjamin.peterson | 2009-06-05 08:39:25 -0500 (Fri, 05 Jun 2009) | 1 line

    allow fixers to give different options in setUp
  ........
    r73255 | benjamin.peterson | 2009-06-06 11:23:46 -0500 (Sat, 06 Jun 2009) | 1 line

    fix the except fixer on one line suites #6222
  ........
    r73256 | benjamin.peterson | 2009-06-06 11:27:40 -0500 (Sat, 06 Jun 2009) | 1 line

    test one-line else and finally clauses
  ........
    r73365 | benjamin.peterson | 2009-06-11 17:01:32 -0500 (Thu, 11 Jun 2009) | 1 line

    normalize whitespace
  ........
................

Merged revisions 72523,72950-72951,72994,73003,73033,73036-73040,73091-73093,73096,73179-73181,73192,73231,73244,73255-73256,73365 via svnmerge from
svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

........
  r72523 | benjamin.peterson | 2009-05-09 14:42:26 -0500 (Sat, 09 May 2009) | 1 line

  remove parenthesis
........
  r72950 | benjamin.peterson | 2009-05-26 18:19:45 -0500 (Tue, 26 May 2009) | 1 line

  remove unused imports
........
  r72951 | benjamin.peterson | 2009-05-26 18:27:00 -0500 (Tue, 26 May 2009) | 1 line

  this is no longer executable
........
  r72994 | benjamin.peterson | 2009-05-28 15:32:54 -0500 (Thu, 28 May 2009) | 1 line

  fix test_all_fixers on Windows #6134
........
  r73003 | benjamin.peterson | 2009-05-28 21:57:28 -0500 (Thu, 28 May 2009) | 4 lines

  make 2to3 test utilities easier to use with other applications (3to2)

  Patch by Joe Amenta
........
  r73033 | benjamin.peterson | 2009-05-29 16:58:32 -0500 (Fri, 29 May 2009) | 1 line

  update grammar for multi with statement
........
  r73036 | benjamin.peterson | 2009-05-29 17:33:20 -0500 (Fri, 29 May 2009) | 1 line

  simplify fix_unicode
........
  r73037 | benjamin.peterson | 2009-05-29 17:53:03 -0500 (Fri, 29 May 2009) | 1 line

  add custom error for pattern syntax errors
........
  r73038 | benjamin.peterson | 2009-05-29 17:55:00 -0500 (Fri, 29 May 2009) | 1 line

  complain if details are attached to a token
........
  r73039 | benjamin.peterson | 2009-05-29 18:00:28 -0500 (Fri, 29 May 2009) | 1 line

  add a test for whitespace
........
  r73040 | benjamin.peterson | 2009-05-29 18:01:17 -0500 (Fri, 29 May 2009) | 1 line

  a fix for emacs highlighting
........
  r73091 | benjamin.peterson | 2009-05-31 20:55:25 -0500 (Sun, 31 May 2009) | 1 line

  deprecate set_prefix() and get_prefix() in favor of a prefix property
........
  r73092 | benjamin.peterson | 2009-05-31 21:00:51 -0500 (Sun, 31 May 2009) | 1 line

  change hideous java naming scheme
........
  r73093 | benjamin.peterson | 2009-05-31 21:01:39 -0500 (Sun, 31 May 2009) | 1 line

  remove dated comment
........
  r73096 | benjamin.peterson | 2009-05-31 21:40:53 -0500 (Sun, 31 May 2009) | 1 line

  group tests
........
  r73179 | benjamin.peterson | 2009-06-03 13:09:53 -0500 (Wed, 03 Jun 2009) | 1 line

  handle the case where there's multiple trailers #6185
........
  r73180 | benjamin.peterson | 2009-06-03 13:18:05 -0500 (Wed, 03 Jun 2009) | 1 line

  scrap __main__ section
........
  r73181 | benjamin.peterson | 2009-06-03 13:24:48 -0500 (Wed, 03 Jun 2009) | 1 line

  remove shebang lines and __main__ sections
........
  r73192 | benjamin.peterson | 2009-06-03 19:16:30 -0500 (Wed, 03 Jun 2009) | 4 lines

  actually test something here

  Thanks to Joe Amenta for noticing.y
........
  r73231 | benjamin.peterson | 2009-06-04 13:38:50 -0500 (Thu, 04 Jun 2009) | 1 line

  remove unused variable
........
  r73244 | benjamin.peterson | 2009-06-05 08:39:25 -0500 (Fri, 05 Jun 2009) | 1 line

  allow fixers to give different options in setUp
........
  r73255 | benjamin.peterson | 2009-06-06 11:23:46 -0500 (Sat, 06 Jun 2009) | 1 line

  fix the except fixer on one line suites #6222
........
  r73256 | benjamin.peterson | 2009-06-06 11:27:40 -0500 (Sat, 06 Jun 2009) | 1 line

  test one-line else and finally clauses
........
  r73365 | benjamin.peterson | 2009-06-11 17:01:32 -0500 (Thu, 11 Jun 2009) | 1 line

  normalize whitespace
........

svn+ssh://pythondev@svn.python.org/python/trunk

................
  r72494 | benjamin.peterson | 2009-05-08 20:01:14 -0500 (Fri, 08 May 2009) | 21 lines

  Merged revisions 72491-72493 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r72491 | benjamin.peterson | 2009-05-08 19:33:27 -0500 (Fri, 08 May 2009) | 7 lines

    make 2to3 use unicode internally on 2.x

    This started out as a fix for #2660, but became this large refactoring
    when I realized the dire state this was in. 2to3 now uses
    tokenize.detect_encoding to decode the files correctly into unicode.
  ........
    r72492 | benjamin.peterson | 2009-05-08 19:35:38 -0500 (Fri, 08 May 2009) | 1 line

    remove compat code
  ........
    r72493 | benjamin.peterson | 2009-05-08 19:54:15 -0500 (Fri, 08 May 2009) | 1 line

    add a test for \r\n newlines
  ........
................

svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

........
  r72491 | benjamin.peterson | 2009-05-08 19:33:27 -0500 (Fri, 08 May 2009) | 7 lines

  make 2to3 use unicode internally on 2.x

  This started out as a fix for #2660, but became this large refactoring
  when I realized the dire state this was in. 2to3 now uses
  tokenize.detect_encoding to decode the files correctly into unicode.
........
  r72492 | benjamin.peterson | 2009-05-08 19:35:38 -0500 (Fri, 08 May 2009) | 1 line

  remove compat code
........
  r72493 | benjamin.peterson | 2009-05-08 19:54:15 -0500 (Fri, 08 May 2009) | 1 line

  add a test for \r\n newlines
........

svn+ssh://pythondev@svn.python.org/python/trunk

................
  r66174 | benjamin.peterson | 2008-09-02 19:21:32 -0500 (Tue, 02 Sep 2008) | 15 lines

  Merged revisions 66173 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r66173 | benjamin.peterson | 2008-09-02 18:57:48 -0500 (Tue, 02 Sep 2008) | 8 lines

    A little 2to3 refactoring #3637

    This moves command line logic from refactor.py to a new file called
    main.py.  RefactoringTool now merely deals with the actual fixers and
    refactoring; options processing for example is abstracted out.

    This patch was reviewed by Gregory P. Smith.
  ........
................
  r66175 | benjamin.peterson | 2008-09-02 20:53:28 -0500 (Tue, 02 Sep 2008) | 1 line

  update 2to3 script from 2to3 trunk
................
  r66177 | benjamin.peterson | 2008-09-02 21:14:03 -0500 (Tue, 02 Sep 2008) | 9 lines

  Merged revisions 66176 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r66176 | benjamin.peterson | 2008-09-02 21:04:06 -0500 (Tue, 02 Sep 2008) | 1 line

    fix typo
  ........
................

svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

........
  r66173 | benjamin.peterson | 2008-09-02 18:57:48 -0500 (Tue, 02 Sep 2008) | 8 lines

  A little 2to3 refactoring #3637

  This moves command line logic from refactor.py to a new file called
  main.py.  RefactoringTool now merely deals with the actual fixers and
  refactoring; options processing for example is abstracted out.

  This patch was reviewed by Gregory P. Smith.
........

svn+ssh://pythondev@svn.python.org/python/trunk

........
  r61596 | martin.v.loewis | 2008-03-18 23:43:46 -0500 (Di, 18 Mär 2008) | 2 lines

  Import lib2to3.
........
  r61597 | martin.v.loewis | 2008-03-18 23:58:04 -0500 (Di, 18 Mär 2008) | 3 lines

  Initialized merge tracking via "svnmerge" with revisions "1-61595" from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3
........