regrtest.py 39.8 KB
Newer Older
1 2 3 4 5 6 7 8 9 10
#! /usr/bin/env python

"""Regression test.

This will find all modules whose name is "test_*" in the test
directory, and run them.  Various command line options provide
additional facilities.

Command line options:

11
-v: verbose    -- run tests in verbose mode with output to stdout
12
-w: verbose2   -- re-run failed tests in verbose mode
13 14 15 16 17 18 19 20 21 22 23
-q: quiet      -- don't print anything except if a test fails
-g: generate   -- write the output file for a test instead of comparing it
-x: exclude    -- arguments are tests to *exclude*
-s: single     -- run only a single test (see below)
-r: random     -- randomize test execution order
-f: fromfile   -- read names of tests to run from a file (see below)
-l: findleaks  -- if GC is available detect tests that leak memory
-u: use        -- specify which special resource intensive tests to run
-h: help       -- print this text and exit
-t: threshold  -- call gc.set_threshold(N)
-T: coverage   -- turn on code coverage using the trace module
24 25
-D: coverdir   -- Directory where coverage files are put
-N: nocoverdir -- Put coverage files alongside modules
26 27
-L: runleaks   -- run the leaks(1) command just before exit
-R: huntrleaks -- search for reference leaks (needs debug build, v. slow)
28
-M: memlimit   -- run very large memory-consuming tests
29 30 31 32

If non-option arguments are present, they are names for tests to run,
unless -x is given, in which case they are names for tests not to run.
If no test names are given, all tests are run.
33

34
-v is incompatible with -g and does not compare test output files.
35

36 37
-T turns on code coverage tracing with the trace module.

38 39 40 41
-D specifies the directory where coverage files are put.

-N Put coverage files alongside modules.

42 43 44 45 46 47 48
-s means to run only a single test and exit.  This is useful when
doing memory analysis on the Python interpreter (which tend to consume
too many resources to run the full regression test non-stop).  The
file /tmp/pynexttest is read to find the next test to run.  If this
file is missing, the first test_*.py file in testdir or on the command
line is used.  (actually tempfile.gettempdir() is used instead of
/tmp).
49

50 51 52 53
-f reads the names of tests from the file given as f's argument, one
or more test names per line.  Whitespace is ignored.  Blank lines and
lines beginning with '#' are ignored.  This is especially useful for
whittling down failures involving interactions among tests.
Tim Peters's avatar
Tim Peters committed
54

55 56 57 58
-L causes the leaks(1) command to be run just before exit if it exists.
leaks(1) is available on Mac OS X and presumably on some other
FreeBSD-derived systems.

59 60 61 62 63 64 65 66
-R runs each test several times and examines sys.gettotalrefcount() to
see if the test appears to be leaking references.  The argument should
be of the form stab:run:fname where 'stab' is the number of times the
test is run to let gettotalrefcount settle down, 'run' is the number
of times further it is run and 'fname' is the name of the file the
reports are written to.  These parameters all have defaults (5, 4 and
"reflog.txt" respectively), so the minimal invocation is '-R ::'.

67 68 69 70 71 72 73 74 75 76 77 78 79
-M runs tests that require an exorbitant amount of memory. These tests
typically try to ascertain containers keep working when containing more than
2 bilion objects, and only work on 64-bit systems. The passed-in memlimit,
which is a string in the form of '2.5Gb', determines howmuch memory the
tests will limit themselves to (but they may go slightly over.) The number
shouldn't be more memory than the machine has (including swap memory). You
should also keep in mind that swap memory is generally much, much slower
than RAM, and setting memlimit to all available RAM or higher will heavily
tax the machine. On the other hand, it is no use running these tests with a
limit of less than 2.5Gb, and many require more than 20Gb. Tests that expect
to use more than memlimit memory will be skipped. The big-memory tests
generally run very, very long.

80 81 82 83
-u is used to specify which special resource intensive tests to run,
such as those requiring large file support or network connectivity.
The argument is a comma-separated list of words indicating the
resources to test.  Currently only the following are defined:
84

85 86
    all -       Enable all special resources.

Guido van Rossum's avatar
Guido van Rossum committed
87 88 89 90
    audio -     Tests that use the audio device.  (There are known
                cases of broken audio drivers that can crash Python or
                even the Linux kernel.)

91 92
    curses -    Tests that use curses and will modify the terminal's
                state and output modes.
Tim Peters's avatar
Tim Peters committed
93

94 95 96
    largefile - It is okay to run some test that may create huge
                files.  These tests can take a long time and may
                consume >2GB of disk space temporarily.
97

98 99
    network -   It is okay to run tests that use external network
                resource, e.g. testing SSL support for sockets.
100 101 102

    bsddb -     It is okay to run the bsddb testsuite, which takes
                a long time to complete.
103

104 105 106
    decimal -   Test the decimal module against a large suite that
                verifies compliance with standards.

107 108
    compiler -  Test the compiler package by compiling all the source
                in the standard library and test suite.  This takes
109 110 111
                a long time.  Enabling this resource also allows
                test_tokenize to verify round-trip lexing on every
                file in the test library.
112

Tim Peters's avatar
Tim Peters committed
113
    subprocess  Run all tests for the subprocess module.
114

115 116
    urlfetch -  It is okay to download files required on testing.

117 118 119
To enable all resources except one, use '-uall,-<resource>'.  For
example, to run all the tests except for the bsddb tests, give the
option '-uall,-bsddb'.
120 121 122
"""

import os
123
import sys
124
import getopt
125
import random
126
import warnings
Neal Norwitz's avatar
Neal Norwitz committed
127
import re
128 129
import cStringIO
import traceback
130 131 132

# I see no other way to suppress these warnings;
# putting them in test_grammar.py has no effect:
133
warnings.filterwarnings("ignore", "hex/oct constants", FutureWarning,
134
                        ".*test.test_grammar$")
135 136 137 138 139
if sys.maxint > 0x7fffffff:
    # Also suppress them in <string>, because for 64-bit platforms,
    # that's where test_grammar.py hides them.
    warnings.filterwarnings("ignore", "hex/oct constants", FutureWarning,
                            "<string>")
140

141 142
# Ignore ImportWarnings that only occur in the source tree,
# (because of modules with the same name as source-directories in Modules/)
143 144 145
for mod in ("ctypes", "gzip", "zipfile", "tarfile", "encodings.zlib_codec",
            "test.test_zipimport", "test.test_zlib", "test.test_zipfile",
            "test.test_codecs", "test.string_tests"):
146 147 148
    warnings.filterwarnings(module=".*%s$" % (mod,),
                            action="ignore", category=ImportWarning)

149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164
# MacOSX (a.k.a. Darwin) has a default stack size that is too small
# for deeply recursive regular expressions.  We see this as crashes in
# the Python test suite when running test_re.py and test_sre.py.  The
# fix is to set the stack limit to 2048.
# This approach may also be useful for other Unixy platforms that
# suffer from small default stack limits.
if sys.platform == 'darwin':
    try:
        import resource
    except ImportError:
        pass
    else:
        soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
        newsoft = min(hard, max(soft, 1024*2048))
        resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard))

165
from test import test_support
166

167
RESOURCE_NAMES = ('audio', 'curses', 'largefile', 'network', 'bsddb',
168
                  'decimal', 'compiler', 'subprocess', 'urlfetch')
169 170


171 172 173 174 175 176
def usage(code, msg=''):
    print __doc__
    if msg: print msg
    sys.exit(code)


177 178
def main(tests=None, testdir=None, verbose=0, quiet=False, generate=False,
         exclude=False, single=False, randomize=False, fromfile=None,
179
         findleaks=False, use_resources=None, trace=False, coverdir='coverage',
180
         runleaks=False, huntrleaks=False, verbose2=False):
181 182
    """Execute a test suite.

183
    This also parses command-line options and modifies its behavior
184
    accordingly.
185 186 187 188 189 190

    tests -- a list of strings containing test names (optional)
    testdir -- the directory in which to look for tests (optional)

    Users other than the Python test suite will certainly want to
    specify testdir; if it's omitted, the directory containing the
191
    Python test suite is searched for.
192 193 194 195

    If the tests argument is omitted, the tests listed on the
    command-line will be used.  If that's empty, too, then all *.py
    files beginning with test_ will be used.
196

197
    The other default arguments (verbose, quiet, generate, exclude, single,
198 199 200
    randomize, findleaks, use_resources, trace and coverdir) allow programmers
    calling main() directly to set the values that would normally be set by
    flags on the command line.
201
    """
202

203
    test_support.record_original_stdout(sys.stdout)
204
    try:
205
        opts, args = getopt.getopt(sys.argv[1:], 'hvgqxsrf:lu:t:TD:NLR:wM:',
206
                                   ['help', 'verbose', 'quiet', 'generate',
Tim Peters's avatar
Tim Peters committed
207
                                    'exclude', 'single', 'random', 'fromfile',
208
                                    'findleaks', 'use=', 'threshold=', 'trace',
209
                                    'coverdir=', 'nocoverdir', 'runleaks',
210
                                    'huntrleaks=', 'verbose2', 'memlimit=',
211
                                    ])
212
    except getopt.error, msg:
213 214 215 216 217
        usage(2, msg)

    # Defaults
    if use_resources is None:
        use_resources = []
218
    for o, a in opts:
219 220 221 222
        if o in ('-h', '--help'):
            usage(0)
        elif o in ('-v', '--verbose'):
            verbose += 1
223 224
        elif o in ('-w', '--verbose2'):
            verbose2 = True
225
        elif o in ('-q', '--quiet'):
226
            quiet = True;
227 228
            verbose = 0
        elif o in ('-g', '--generate'):
229
            generate = True
230
        elif o in ('-x', '--exclude'):
231
            exclude = True
232
        elif o in ('-s', '--single'):
233
            single = True
234
        elif o in ('-r', '--randomize'):
235
            randomize = True
Tim Peters's avatar
Tim Peters committed
236 237
        elif o in ('-f', '--fromfile'):
            fromfile = a
238
        elif o in ('-l', '--findleaks'):
239
            findleaks = True
240 241
        elif o in ('-L', '--runleaks'):
            runleaks = True
242 243 244
        elif o in ('-t', '--threshold'):
            import gc
            gc.set_threshold(int(a))
245 246
        elif o in ('-T', '--coverage'):
            trace = True
247 248 249 250
        elif o in ('-D', '--coverdir'):
            coverdir = os.path.join(os.getcwd(), a)
        elif o in ('-N', '--nocoverdir'):
            coverdir = None
251 252 253 254 255 256 257 258 259 260 261 262 263 264 265
        elif o in ('-R', '--huntrleaks'):
            huntrleaks = a.split(':')
            if len(huntrleaks) != 3:
                print a, huntrleaks
                usage(2, '-R takes three colon-separated arguments')
            if len(huntrleaks[0]) == 0:
                huntrleaks[0] = 5
            else:
                huntrleaks[0] = int(huntrleaks[0])
            if len(huntrleaks[1]) == 0:
                huntrleaks[1] = 4
            else:
                huntrleaks[1] = int(huntrleaks[1])
            if len(huntrleaks[2]) == 0:
                huntrleaks[2] = "reflog.txt"
266 267
        elif o in ('-M', '--memlimit'):
            test_support.set_memlimit(a)
268
        elif o in ('-u', '--use'):
269 270
            u = [x.lower() for x in a.split(',')]
            for r in u:
271
                if r == 'all':
272 273 274 275 276 277
                    use_resources[:] = RESOURCE_NAMES
                    continue
                remove = False
                if r[0] == '-':
                    remove = True
                    r = r[1:]
278 279
                if r not in RESOURCE_NAMES:
                    usage(1, 'Invalid -u/--use option: ' + a)
280 281 282 283
                if remove:
                    if r in use_resources:
                        use_resources.remove(r)
                elif r not in use_resources:
284
                    use_resources.append(r)
285
    if generate and verbose:
286
        usage(2, "-g and -v don't go together!")
Tim Peters's avatar
Tim Peters committed
287 288
    if single and fromfile:
        usage(2, "-s and -f don't go together!")
289

290 291 292
    good = []
    bad = []
    skipped = []
293
    resource_denieds = []
294

295
    if findleaks:
296 297 298
        try:
            import gc
        except ImportError:
299
            print 'No GC available, disabling findleaks.'
300
            findleaks = False
301
        else:
302 303 304 305
            # Uncomment the line below to report garbage that is not
            # freeable by reference counting alone.  By default only
            # garbage that is not collectable by the GC is reported.
            #gc.set_debug(gc.DEBUG_SAVEALL)
306
            found_garbage = []
307

308 309 310 311 312
    if single:
        from tempfile import gettempdir
        filename = os.path.join(gettempdir(), 'pynexttest')
        try:
            fp = open(filename, 'r')
313
            next = fp.read().strip()
314 315 316 317
            tests = [next]
            fp.close()
        except IOError:
            pass
Tim Peters's avatar
Tim Peters committed
318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333

    if fromfile:
        tests = []
        fp = open(fromfile)
        for line in fp:
            guts = line.split() # assuming no test has whitespace in its name
            if guts and not guts[0].startswith('#'):
                tests.extend(guts)
        fp.close()

    # Strip .py extensions.
    if args:
        args = map(removepy, args)
    if tests:
        tests = map(removepy, tests)

334 335
    stdtests = STDTESTS[:]
    nottests = NOTTESTS[:]
336
    if exclude:
337 338 339 340
        for arg in args:
            if arg in stdtests:
                stdtests.remove(arg)
        nottests[:0] = args
341
        args = []
342
    tests = tests or args or findtests(testdir, stdtests, nottests)
343 344
    if single:
        tests = tests[:1]
345 346
    if randomize:
        random.shuffle(tests)
347 348 349 350
    if trace:
        import trace
        tracer = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix],
                             trace=False, count=True)
351
    test_support.verbose = verbose      # Tell tests to be moderately quiet
352
    test_support.use_resources = use_resources
353
    save_modules = sys.modules.keys()
354
    for test in tests:
355 356
        if not quiet:
            print test
357
            sys.stdout.flush()
358 359 360 361 362
        if trace:
            # If we're tracing code coverage, then we don't exit with status
            # if on a false return value from main.
            tracer.runctx('runtest(test, generate, verbose, quiet, testdir)',
                          globals=globals(), locals=vars())
363
        else:
364 365 366 367 368 369 370 371 372
            try:
                ok = runtest(test, generate, verbose, quiet, testdir,
                             huntrleaks)
            except KeyboardInterrupt:
                # print a newline separate from the ^C
                print
                break
            except:
                raise
373 374 375 376 377 378 379 380
            if ok > 0:
                good.append(test)
            elif ok == 0:
                bad.append(test)
            else:
                skipped.append(test)
                if ok == -2:
                    resource_denieds.append(test)
381 382 383
        if findleaks:
            gc.collect()
            if gc.garbage:
384 385 386 387
                print "Warning: test created", len(gc.garbage),
                print "uncollectable object(s)."
                # move the uncollectable objects somewhere so we don't see
                # them again
388 389
                found_garbage.extend(gc.garbage)
                del gc.garbage[:]
390 391
        # Unload the newly imported modules (best effort finalization)
        for module in sys.modules.keys():
392
            if module not in save_modules and module.startswith("test."):
393
                test_support.unload(module)
394 395 396 397 398

    # The lists won't be sorted if running with -r
    good.sort()
    bad.sort()
    skipped.sort()
Tim Peters's avatar
Tim Peters committed
399

400
    if good and not quiet:
401 402 403
        if not bad and not skipped and len(good) > 1:
            print "All",
        print count(len(good), "test"), "OK."
404
        if verbose:
405 406
            print "CAUTION:  stdout isn't compared in verbose mode:"
            print "a test that passes in verbose mode may fail without it."
407
    if bad:
408 409
        print count(len(bad), "test"), "failed:"
        printlist(bad)
410
    if skipped and not quiet:
411 412
        print count(len(skipped), "test"), "skipped:"
        printlist(skipped)
413

414
        e = _ExpectedSkips()
415
        plat = sys.platform
416
        if e.isvalid():
417
            surprise = set(skipped) - e.getexpected() - set(resource_denieds)
418 419
            if surprise:
                print count(len(surprise), "skip"), \
420 421
                      "unexpected on", plat + ":"
                printlist(surprise)
422 423 424 425 426 427
            else:
                print "Those skips are all expected on", plat + "."
        else:
            print "Ask someone to teach regrtest.py about which tests are"
            print "expected to get skipped on", plat + "."

428 429 430
    if verbose2 and bad:
        print "Re-running failed tests in verbose mode"
        for test in bad:
431 432
            print "Re-running test %r in verbose mode" % test
            sys.stdout.flush()
433 434 435 436 437 438 439 440 441 442 443
            try:
                test_support.verbose = 1
                ok = runtest(test, generate, 1, quiet, testdir,
                             huntrleaks)
            except KeyboardInterrupt:
                # print a newline separate from the ^C
                print
                break
            except:
                raise

444 445 446 447 448 449 450 451 452 453 454 455 456 457
    if single:
        alltests = findtests(testdir, stdtests, nottests)
        for i in range(len(alltests)):
            if tests[0] == alltests[i]:
                if i == len(alltests) - 1:
                    os.unlink(filename)
                else:
                    fp = open(filename, 'w')
                    fp.write(alltests[i+1] + '\n')
                    fp.close()
                break
        else:
            os.unlink(filename)

458 459 460 461
    if trace:
        r = tracer.results()
        r.write_results(show_missing=True, summary=True, coverdir=coverdir)

462 463 464
    if runleaks:
        os.system("leaks %d" % os.getpid())

465
    sys.exit(len(bad) > 0)
466

467

468
STDTESTS = [
469 470 471 472 473 474 475 476
    'test_grammar',
    'test_opcodes',
    'test_operations',
    'test_builtin',
    'test_exceptions',
    'test_types',
   ]

477
NOTTESTS = [
478
    'test_support',
479 480
    'test_future1',
    'test_future2',
481
    'test_future3',
482 483
    ]

484
def findtests(testdir=None, stdtests=STDTESTS, nottests=NOTTESTS):
485
    """Return a list of all applicable test modules."""
486
    if not testdir: testdir = findtestdir()
487 488 489
    names = os.listdir(testdir)
    tests = []
    for name in names:
490
        if name[:5] == "test_" and name[-3:] == os.extsep+"py":
491 492 493
            modname = name[:-3]
            if modname not in stdtests and modname not in nottests:
                tests.append(modname)
494 495 496
    tests.sort()
    return stdtests + tests

497
def runtest(test, generate, verbose, quiet, testdir=None, huntrleaks=False):
498
    """Run a single test.
499

500 501
    test -- the name of the test
    generate -- if true, generate output, instead of running the test
502
                and comparing it to a previously created output file
503
    verbose -- if true, print more messages
504
    quiet -- if true, don't print 'skipped' messages (probably redundant)
505
    testdir -- test directory
506 507 508 509 510 511 512
    huntrleaks -- run multiple times to test for leaks; requires a debug
                  build; a triple corresponding to -R's three arguments
    Return:
        -2  test skipped because resource denied
        -1  test skipped for some other reason
         0  test failed
         1  test passed
513
    """
514

515 516 517 518 519 520 521 522
    try:
        return runtest_inner(test, generate, verbose, quiet, testdir,
                             huntrleaks)
    finally:
        cleanup_test_droppings(test, verbose)

def runtest_inner(test, generate, verbose, quiet,
                     testdir=None, huntrleaks=False):
523
    test_support.unload(test)
524 525
    if not testdir:
        testdir = findtestdir()
526 527
    outputdir = os.path.join(testdir, "output")
    outputfile = os.path.join(outputdir, test)
528
    if verbose:
529
        cfp = None
530
    else:
531
        cfp = cStringIO.StringIO()
532

533
    try:
534
        save_stdout = sys.stdout
535 536 537 538
        try:
            if cfp:
                sys.stdout = cfp
                print test              # Output file starts with test name
539 540 541 542 543 544 545
            if test.startswith('test.'):
                abstest = test
            else:
                # Always import it from the test package
                abstest = 'test.' + test
            the_package = __import__(abstest, globals(), locals(), [])
            the_module = getattr(the_package, test)
546 547 548 549 550 551 552
            # Most tests run to completion simply as a side-effect of
            # being imported.  For the benefit of tests that can't run
            # that way (like test_threaded_import), explicitly invoke
            # their test_main() function (if it exists).
            indirect_test = getattr(the_module, "test_main", None)
            if indirect_test is not None:
                indirect_test()
553
            if huntrleaks:
554
                dash_R(the_module, test, indirect_test, huntrleaks)
555
        finally:
556
            sys.stdout = save_stdout
557 558 559 560 561
    except test_support.ResourceDenied, msg:
        if not quiet:
            print test, "skipped --", msg
            sys.stdout.flush()
        return -2
562
    except (ImportError, test_support.TestSkipped), msg:
563
        if not quiet:
564
            print test, "skipped --", msg
565
            sys.stdout.flush()
566
        return -1
567 568
    except KeyboardInterrupt:
        raise
569
    except test_support.TestFailed, msg:
570
        print "test", test, "failed --", msg
571
        sys.stdout.flush()
572
        return 0
573
    except:
574
        type, value = sys.exc_info()[:2]
575
        print "test", test, "crashed --", str(type) + ":", value
576
        sys.stdout.flush()
577 578
        if verbose:
            traceback.print_exc(file=sys.stdout)
579
            sys.stdout.flush()
580
        return 0
581
    else:
582 583 584
        if not cfp:
            return 1
        output = cfp.getvalue()
585 586 587 588 589 590 591 592
        if generate:
            if output == test + "\n":
                if os.path.exists(outputfile):
                    # Write it since it already exists (and the contents
                    # may have changed), but let the user know it isn't
                    # needed:
                    print "output file", outputfile, \
                          "is no longer needed; consider removing it"
593 594 595 596 597 598 599 600 601 602 603 604 605
                else:
                    # We don't need it, so don't create it.
                    return 1
            fp = open(outputfile, "w")
            fp.write(output)
            fp.close()
            return 1
        if os.path.exists(outputfile):
            fp = open(outputfile, "r")
            expected = fp.read()
            fp.close()
        else:
            expected = test + "\n"
606
        if output == expected or huntrleaks:
607 608
            return 1
        print "test", test, "produced unexpected output:"
609
        sys.stdout.flush()
610
        reportdiff(expected, output)
611
        sys.stdout.flush()
612 613
        return 0

614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644
def cleanup_test_droppings(testname, verbose):
    import shutil

    # Try to clean up junk commonly left behind.  While tests shouldn't leave
    # any files or directories behind, when a test fails that can be tedious
    # for it to arrange.  The consequences can be especially nasty on Windows,
    # since if a test leaves a file open, it cannot be deleted by name (while
    # there's nothing we can do about that here either, we can display the
    # name of the offending test, which is a real help).
    for name in (test_support.TESTFN,
                 "db_home",
                ):
        if not os.path.exists(name):
            continue

        if os.path.isdir(name):
            kind, nuker = "directory", shutil.rmtree
        elif os.path.isfile(name):
            kind, nuker = "file", os.unlink
        else:
            raise SystemError("os.path says %r exists but is neither "
                              "directory nor file" % name)

        if verbose:
            print "%r left behind %s %r" % (testname, kind, name)
        try:
            nuker(name)
        except Exception, msg:
            print >> sys.stderr, ("%r left behind %s %r and it couldn't be "
                "removed: %s" % (testname, kind, name, msg))

645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686
def dash_R(the_module, test, indirect_test, huntrleaks):
    # This code is hackish and inelegant, but it seems to do the job.
    import copy_reg

    if not hasattr(sys, 'gettotalrefcount'):
        raise Exception("Tracking reference leaks requires a debug build "
                        "of Python")

    # Save current values for dash_R_cleanup() to restore.
    fs = warnings.filters[:]
    ps = copy_reg.dispatch_table.copy()
    pic = sys.path_importer_cache.copy()

    if indirect_test:
        def run_the_test():
            indirect_test()
    else:
        def run_the_test():
            reload(the_module)

    deltas = []
    nwarmup, ntracked, fname = huntrleaks
    repcount = nwarmup + ntracked
    print >> sys.stderr, "beginning", repcount, "repetitions"
    print >> sys.stderr, ("1234567890"*(repcount//10 + 1))[:repcount]
    dash_R_cleanup(fs, ps, pic)
    for i in range(repcount):
        rc = sys.gettotalrefcount()
        run_the_test()
        sys.stderr.write('.')
        dash_R_cleanup(fs, ps, pic)
        if i >= nwarmup:
            deltas.append(sys.gettotalrefcount() - rc - 2)
    print >> sys.stderr
    if any(deltas):
        print >> sys.stderr, test, 'leaked', deltas, 'references'
        refrep = open(fname, "a")
        print >> refrep, test, 'leaked', deltas, 'references'
        refrep.close()

def dash_R_cleanup(fs, ps, pic):
    import gc, copy_reg
687
    import _strptime, linecache, dircache
688
    import urlparse, urllib, urllib2, mimetypes, doctest
689
    import struct, filecmp
690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709
    from distutils.dir_util import _path_created

    # Restore some original values.
    warnings.filters[:] = fs
    copy_reg.dispatch_table.clear()
    copy_reg.dispatch_table.update(ps)
    sys.path_importer_cache.clear()
    sys.path_importer_cache.update(pic)

    # Clear assorted module caches.
    _path_created.clear()
    re.purge()
    _strptime._regex_cache.clear()
    urlparse.clear_cache()
    urllib.urlcleanup()
    urllib2.install_opener(None)
    dircache.reset()
    linecache.clearcache()
    mimetypes._default_mime_types()
    struct._cache.clear()
710
    filecmp._cache.clear()
711 712 713 714 715
    doctest.master = None

    # Collect cyclic trash.
    gc.collect()

716 717
def reportdiff(expected, output):
    import difflib
718 719 720
    print "*" * 70
    a = expected.splitlines(1)
    b = output.splitlines(1)
721 722
    sm = difflib.SequenceMatcher(a=a, b=b)
    tuples = sm.get_opcodes()
723

724
    def pair(x0, x1):
725
        # x0:x1 are 0-based slice indices; convert to 1-based line indices.
726 727
        x0 += 1
        if x0 >= x1:
728
            return "line " + str(x0)
729
        else:
730 731
            return "lines %d-%d" % (x0, x1)

732 733 734
    for op, a0, a1, b0, b1 in tuples:
        if op == 'equal':
            pass
735

736
        elif op == 'delete':
737
            print "***", pair(a0, a1), "of expected output missing:"
738
            for line in a[a0:a1]:
739 740
                print "-", line,

741
        elif op == 'replace':
742 743 744 745 746
            print "*** mismatch between", pair(a0, a1), "of expected", \
                  "output and", pair(b0, b1), "of actual output:"
            for line in difflib.ndiff(a[a0:a1], b[b0:b1]):
                print line,

747
        elif op == 'insert':
748 749
            print "***", pair(b0, b1), "of actual output doesn't appear", \
                  "in expected output after line", str(a1)+":"
750
            for line in b[b0:b1]:
751 752
                print "+", line,

753 754
        else:
            print "get_opcodes() returned bad tuple?!?!", (op, a0, a1, b0, b1)
755

756
    print "*" * 70
757 758 759

def findtestdir():
    if __name__ == '__main__':
760
        file = sys.argv[0]
761
    else:
762
        file = __file__
763 764 765
    testdir = os.path.dirname(file) or os.curdir
    return testdir

Tim Peters's avatar
Tim Peters committed
766 767 768 769 770
def removepy(name):
    if name.endswith(os.extsep + "py"):
        name = name[:-3]
    return name

771 772
def count(n, word):
    if n == 1:
773
        return "%d %s" % (n, word)
774
    else:
775
        return "%d %ss" % (n, word)
776

777
def printlist(x, width=70, indent=4):
778
    """Print the elements of iterable x to stdout.
779 780 781 782 783 784

    Optional arg width (default 70) is the maximum line length.
    Optional arg indent (default 4) is the number of blanks with which to
    begin each line.
    """

785 786 787 788
    from textwrap import fill
    blanks = ' ' * indent
    print fill(' '.join(map(str, x)), width,
               initial_indent=blanks, subsequent_indent=blanks)
789

790 791
# Map sys.platform to a string containing the basenames of tests
# expected to be skipped on that platform.
792 793 794 795 796
#
# Special cases:
#     test_pep277
#         The _ExpectedSkips constructor adds this to the set of expected
#         skips if not os.path.supports_unicode_filenames.
797 798 799
#     test_socket_ssl
#         Controlled by test_socket_ssl.skip_expected.  Requires the network
#         resource, and a socket module with ssl support.
800 801 802
#     test_timeout
#         Controlled by test_timeout.skip_expected.  Requires the network
#         resource and a socket module.
803

804 805 806
_expectations = {
    'win32':
        """
807
        test__locale
808
        test_applesingle
809
        test_al
810
        test_bsddb185
811
        test_bsddb3
812 813 814 815
        test_cd
        test_cl
        test_commands
        test_crypt
816
        test_curses
817 818 819 820 821 822 823 824
        test_dbm
        test_dl
        test_fcntl
        test_fork1
        test_gdbm
        test_gl
        test_grp
        test_imgfile
825
        test_ioctl
826 827 828 829 830
        test_largefile
        test_linuxaudiodev
        test_mhlib
        test_nis
        test_openpty
831
        test_ossaudiodev
832
        test_poll
833
        test_posix
834 835
        test_pty
        test_pwd
836
        test_resource
837 838
        test_signal
        test_sunaudiodev
839
        test_threadsignals
840
        test_timing
841 842
        test_wait3
        test_wait4
843 844 845 846
        """,
    'linux2':
        """
        test_al
847
        test_applesingle
848
        test_bsddb185
849 850
        test_cd
        test_cl
851
        test_curses
852 853 854 855
        test_dl
        test_gl
        test_imgfile
        test_largefile
856
        test_linuxaudiodev
857 858
        test_nis
        test_ntpath
859
        test_ossaudiodev
860
        test_sqlite
861
        test_startfile
862 863
        test_sunaudiodev
        """,
864
   'mac':
865 866
        """
        test_al
867
        test_atexit
868
        test_bsddb
869
        test_bsddb185
870 871
        test_bsddb3
        test_bz2
872 873 874 875
        test_cd
        test_cl
        test_commands
        test_crypt
876
        test_curses
877 878 879 880 881 882
        test_dbm
        test_dl
        test_fcntl
        test_fork1
        test_gl
        test_grp
883
        test_ioctl
884 885 886 887 888 889 890 891
        test_imgfile
        test_largefile
        test_linuxaudiodev
        test_locale
        test_mmap
        test_nis
        test_ntpath
        test_openpty
892
        test_ossaudiodev
893
        test_poll
894
        test_popen
895
        test_popen2
896
        test_posix
897 898
        test_pty
        test_pwd
899
        test_resource
900
        test_signal
901
        test_sqlite
902
        test_startfile
903 904
        test_sunaudiodev
        test_sundry
905
        test_tarfile
906 907
        test_timing
        """,
908
    'unixware7':
909 910
        """
        test_al
911
        test_applesingle
912
        test_bsddb
913
        test_bsddb185
914 915 916 917 918 919 920 921 922 923 924 925 926
        test_cd
        test_cl
        test_dl
        test_gl
        test_imgfile
        test_largefile
        test_linuxaudiodev
        test_minidom
        test_nis
        test_ntpath
        test_openpty
        test_pyexpat
        test_sax
927
        test_startfile
928
        test_sqlite
929 930 931
        test_sunaudiodev
        test_sundry
        """,
932 933 934
    'openunix8':
        """
        test_al
935
        test_applesingle
936
        test_bsddb
937
        test_bsddb185
938 939 940 941 942 943 944 945 946 947 948 949 950
        test_cd
        test_cl
        test_dl
        test_gl
        test_imgfile
        test_largefile
        test_linuxaudiodev
        test_minidom
        test_nis
        test_ntpath
        test_openpty
        test_pyexpat
        test_sax
951
        test_sqlite
952
        test_startfile
953 954 955 956 957 958
        test_sunaudiodev
        test_sundry
        """,
    'sco_sv3':
        """
        test_al
959
        test_applesingle
960 961
        test_asynchat
        test_bsddb
962
        test_bsddb185
963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979
        test_cd
        test_cl
        test_dl
        test_fork1
        test_gettext
        test_gl
        test_imgfile
        test_largefile
        test_linuxaudiodev
        test_locale
        test_minidom
        test_nis
        test_ntpath
        test_openpty
        test_pyexpat
        test_queue
        test_sax
980
        test_sqlite
981
        test_startfile
982 983 984 985 986 987 988
        test_sunaudiodev
        test_sundry
        test_thread
        test_threaded_import
        test_threadedtempfile
        test_threading
        """,
989 990 991
    'riscos':
        """
        test_al
992
        test_applesingle
993
        test_asynchat
994
        test_atexit
995
        test_bsddb
996
        test_bsddb185
997
        test_bsddb3
998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021
        test_cd
        test_cl
        test_commands
        test_crypt
        test_dbm
        test_dl
        test_fcntl
        test_fork1
        test_gdbm
        test_gl
        test_grp
        test_imgfile
        test_largefile
        test_linuxaudiodev
        test_locale
        test_mmap
        test_nis
        test_ntpath
        test_openpty
        test_poll
        test_popen2
        test_pty
        test_pwd
        test_strop
1022
        test_sqlite
1023
        test_startfile
1024 1025 1026 1027 1028 1029 1030 1031
        test_sunaudiodev
        test_sundry
        test_thread
        test_threaded_import
        test_threadedtempfile
        test_threading
        test_timing
        """,
1032
    'darwin':
1033
        """
1034
        test__locale
1035
        test_al
1036
        test_bsddb
1037
        test_bsddb3
1038 1039 1040 1041 1042 1043 1044 1045
        test_cd
        test_cl
        test_curses
        test_gdbm
        test_gl
        test_imgfile
        test_largefile
        test_linuxaudiodev
1046
        test_locale
1047 1048 1049
        test_minidom
        test_nis
        test_ntpath
1050
        test_ossaudiodev
1051
        test_poll
1052
        test_sqlite
1053
        test_startfile
1054 1055
        test_sunaudiodev
        """,
1056 1057 1058
    'sunos5':
        """
        test_al
1059
        test_applesingle
1060
        test_bsddb
1061
        test_bsddb185
1062 1063 1064 1065 1066 1067 1068 1069 1070 1071
        test_cd
        test_cl
        test_curses
        test_dbm
        test_gdbm
        test_gl
        test_gzip
        test_imgfile
        test_linuxaudiodev
        test_openpty
1072
        test_sqlite
1073
        test_startfile
1074 1075
        test_zipfile
        test_zlib
Jeremy Hylton's avatar
Jeremy Hylton committed
1076
        """,
1077 1078 1079
    'hp-ux11':
        """
        test_al
1080
        test_applesingle
1081
        test_bsddb
1082
        test_bsddb185
1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099
        test_cd
        test_cl
        test_curses
        test_dl
        test_gdbm
        test_gl
        test_gzip
        test_imgfile
        test_largefile
        test_linuxaudiodev
        test_locale
        test_minidom
        test_nis
        test_ntpath
        test_openpty
        test_pyexpat
        test_sax
1100
        test_sqlite
1101
        test_startfile
1102 1103 1104 1105
        test_sunaudiodev
        test_zipfile
        test_zlib
        """,
1106
    'atheos':
Tim Peters's avatar
Tim Peters committed
1107 1108
        """
        test_al
1109
        test_applesingle
1110
        test_bsddb185
Tim Peters's avatar
Tim Peters committed
1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126
        test_cd
        test_cl
        test_curses
        test_dl
        test_gdbm
        test_gl
        test_imgfile
        test_largefile
        test_linuxaudiodev
        test_locale
        test_mhlib
        test_mmap
        test_nis
        test_poll
        test_popen2
        test_resource
1127
        test_sqlite
1128
        test_startfile
Tim Peters's avatar
Tim Peters committed
1129 1130
        test_sunaudiodev
        """,
1131 1132 1133
    'cygwin':
        """
        test_al
1134
        test_applesingle
1135
        test_bsddb185
1136
        test_bsddb3
1137 1138 1139 1140 1141 1142
        test_cd
        test_cl
        test_curses
        test_dbm
        test_gl
        test_imgfile
1143
        test_ioctl
1144 1145 1146 1147
        test_largefile
        test_linuxaudiodev
        test_locale
        test_nis
1148
        test_ossaudiodev
1149
        test_socketserver
1150
        test_sqlite
1151 1152
        test_sunaudiodev
        """,
1153 1154 1155
    'os2emx':
        """
        test_al
1156
        test_applesingle
1157
        test_audioop
1158
        test_bsddb185
1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176
        test_bsddb3
        test_cd
        test_cl
        test_commands
        test_curses
        test_dl
        test_gl
        test_imgfile
        test_largefile
        test_linuxaudiodev
        test_mhlib
        test_mmap
        test_nis
        test_openpty
        test_ossaudiodev
        test_pty
        test_resource
        test_signal
1177
        test_sqlite
1178
        test_startfile
1179 1180
        test_sunaudiodev
        """,
1181 1182 1183 1184 1185 1186 1187 1188 1189
    'freebsd4':
        """
        test_aepack
        test_al
        test_applesingle
        test_bsddb
        test_bsddb3
        test_cd
        test_cl
1190
        test_gdbm
1191 1192 1193 1194 1195 1196 1197 1198 1199 1200
        test_gl
        test_imgfile
        test_linuxaudiodev
        test_locale
        test_macfs
        test_macostools
        test_nis
        test_ossaudiodev
        test_pep277
        test_plistlib
1201
        test_pty
1202 1203 1204
        test_scriptpackages
        test_socket_ssl
        test_socketserver
1205
        test_sqlite
1206
        test_startfile
1207
        test_sunaudiodev
1208
        test_tcl
1209 1210 1211 1212 1213
        test_timeout
        test_unicode_file
        test_urllibnet
        test_winreg
        test_winsound
1214
        """,
1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235
    'aix5':
        """
        test_aepack
        test_al
        test_applesingle
        test_bsddb
        test_bsddb185
        test_bsddb3
        test_bz2
        test_cd
        test_cl
        test_dl
        test_gdbm
        test_gl
        test_gzip
        test_imgfile
        test_linuxaudiodev
        test_macfs
        test_macostools
        test_nis
        test_ossaudiodev
1236
        test_sqlite
1237
        test_startfile
1238 1239 1240 1241 1242 1243 1244
        test_sunaudiodev
        test_tcl
        test_winreg
        test_winsound
        test_zipimport
        test_zlib
        """,
1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269
    'openbsd3':
        """
        test_aepack
        test_al
        test_applesingle
        test_bsddb
        test_bsddb3
        test_cd
        test_cl
        test_ctypes
        test_dl
        test_gdbm
        test_gl
        test_imgfile
        test_linuxaudiodev
        test_locale
        test_macfs
        test_macostools
        test_nis
        test_normalization
        test_ossaudiodev
        test_pep277
        test_plistlib
        test_scriptpackages
        test_tcl
1270
        test_sqlite
1271
        test_startfile
1272 1273 1274 1275 1276
        test_sunaudiodev
        test_unicode_file
        test_winreg
        test_winsound
        """,
1277
}
1278
_expectations['freebsd5'] = _expectations['freebsd4']
1279
_expectations['freebsd6'] = _expectations['freebsd4']
1280
_expectations['freebsd7'] = _expectations['freebsd4']
1281

1282 1283
class _ExpectedSkips:
    def __init__(self):
1284
        import os.path
1285
        from test import test_socket_ssl
1286
        from test import test_timeout
1287

1288
        self.valid = False
1289
        if sys.platform in _expectations:
1290
            s = _expectations[sys.platform]
1291
            self.expected = set(s.split())
1292

1293 1294
            if not os.path.supports_unicode_filenames:
                self.expected.add('test_pep277')
1295

1296 1297 1298
            if test_socket_ssl.skip_expected:
                self.expected.add('test_socket_ssl')

1299 1300 1301
            if test_timeout.skip_expected:
                self.expected.add('test_timeout')

1302 1303 1304 1305
            if sys.maxint == 9223372036854775807L:
                self.expected.add('test_rgbimg')
                self.expected.add('test_imageop')

1306
            if not sys.platform in ("mac", "darwin"):
1307 1308 1309 1310
                MAC_ONLY = ["test_macostools", "test_macfs", "test_aepack",
                            "test_plistlib", "test_scriptpackages"]
                for skip in MAC_ONLY:
                    self.expected.add(skip)
1311 1312

            if sys.platform != "win32":
1313 1314 1315 1316
                WIN_ONLY = ["test_unicode_file", "test_winreg",
                            "test_winsound"]
                for skip in WIN_ONLY:
                    self.expected.add(skip)
Tim Peters's avatar
Tim Peters committed
1317

1318
            self.valid = True
1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332

    def isvalid(self):
        "Return true iff _ExpectedSkips knows about the current platform."
        return self.valid

    def getexpected(self):
        """Return set of test names we expect to skip on current platform.

        self.isvalid() must be true.
        """

        assert self.isvalid()
        return self.expected

1333
if __name__ == '__main__':
1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347
    # Remove regrtest.py's own directory from the module search path.  This
    # prevents relative imports from working, and relative imports will screw
    # up the testing framework.  E.g. if both test.test_support and
    # test_support are imported, they will not contain the same globals, and
    # much of the testing framework relies on the globals in the
    # test.test_support module.
    mydir = os.path.abspath(os.path.normpath(os.path.dirname(sys.argv[0])))
    i = pathlen = len(sys.path)
    while i >= 0:
        i -= 1
        if os.path.abspath(os.path.normpath(sys.path[i])) == mydir:
            del sys.path[i]
    if len(sys.path) == pathlen:
        print 'Could not find %r in sys.path to remove it' % mydir
1348
    main()