• Jeremy Hylton's avatar
    Bug fixes large and small for tokenize. · 76467ba6
    Jeremy Hylton yazdı
    Small: Always generate a NL or NEWLINE token following
           a COMMENT token.  The old code did not generate an NL token if
           the comment was on a line by itself.
    
    Large: The output of untokenize() will now match the
           input exactly if it is passed the full token sequence.  The
           old, crufty output is still generated if a limited input
           sequence is provided, where limited means that it does not
           include position information for tokens.
    
    Remaining bug: There is no CONTINUATION token (\) so there is no way
    for untokenize() to handle such code.
    
    Also, expanded the number of doctests in hopes of eventually removing
    the old-style tests that compare against a golden file.
    
    Bug fix candidate for Python 2.5.1. (Sigh.)
    76467ba6
test_tokenize 16.4 KB