implement test skipping and expected failures

patch by myself #1034053

Doc/library/unittest.rst

@@ -11,6 +11,9 @@
 
 
 .. versionadded:: 2.1
+.. versionchanged:: 2.7
+
+   Added :ref:`skipping and expected failures <unittest-skipping>`.
 
 The Python unit testing framework, sometimes referred to as "PyUnit," is a
 Python language version of JUnit, by Kent Beck and Erich Gamma. JUnit is, in
@@ -60,7 +63,8 @@ fixture is created for each test.
 
 Test suites are implemented by the :class:`TestSuite` class.  This class allows
 individual tests and test suites to be aggregated; when the suite is executed,
-all tests added directly to the suite and in "child" test suites are run.
+all tests added directly to the suite and in "child" test suites are run.  A
+:class:`ClassTestSuite` contains the test cases of a class.
 
 A test runner is an object that provides a single method, :meth:`run`, which
 accepts a :class:`TestCase` or :class:`TestSuite` object as a parameter, and
@@ -408,6 +412,78 @@ may treat :exc:`AssertionError` differently.
    make future test refactorings infinitely easier.
 
 
+.. _unittest-skipping:
+
+Skipping tests and expected failures
+------------------------------------
+
+Unittest supports skipping individual test methods and even whole classes of
+tests.  In addition, it supports marking a test as a "expected failure," a test
+that is broken and will fail, but shouldn't be counted as a failure on a
+:class:`TestResult`.
+
+Skipping a test is simply a matter of using the :func:`skip` :term:`decorator`
+or one of its conditional variants.
+
+Basic skipping looks like this: ::
+
+   class MyTestCase(unittest.TestCase):
+
+       @unittest.skip("demonstrating skipping")
+       def test_nothing(self):
+           self.fail("shouldn't happen")
+
+This is the output of running the example above in verbose mode: ::
+
+   test_nothing (__main__.MyTestCase) ... skipped 'demonstrating skipping'
+
+   ----------------------------------------------------------------------
+   Ran 1 test in 0.072s
+
+Classes can be skipped just like methods: ::
+
+   @skip("showing class skipping")
+   class MySkippedTestCase(unittest.TestCase):
+       def test_not_run(self):
+           pass
+
+Expected failures use the :func:`expectedFailure` decorator. ::
+
+   class ExpectedFailureTestCase(unittest.TestCase):
+       @unittest.expectedFailure
+       def test_fail(self):
+           self.assertEqual(1, 0, "broken")
+
+It's easy to roll your own skipping decorators by making a decorator that calls
+:func:`skip` on the test when it wants it to be skipped.  This decorator skips
+the test unless the passed object has a certain attribute: ::
+
+   def skipUnlessHasattr(obj, attr):
+       if hasattr(obj, attr):
+           return lambda func: func
+       return unittest.skip("{0!r} doesn't have {1!r}".format(obj, attr))
+
+The following decorators implement test skipping and expected failures:
+
+.. function:: skip(reason)
+
+   Unconditionally skip the decorated test.  *reason* should describe why the
+   test is being skipped.
+
+.. function:: skipIf(condition, reason)
+
+   Skip the decorated test if *condition* is true.
+
+.. function:: skipUnless(condition, reason)
+
+   Skip the decoratored test unless *condition* is true.
+
+.. function:: expectedFailure
+
+   Mark the test as an expected failure.  If the test fails when run, the test
+   is not counted as a failure.
+
+
 .. _unittest-contents:
 
 Classes and functions
@@ -459,6 +535,13 @@ Classes and functions
    test suites that will be used to build the suite initially. Additional methods
    are provided to add test cases and suites to the collection later on.
 
+.. class:: ClassTestSuite(tests, collected_from)
+
+   This subclass of :class:`TestSuite` repesents an aggregation of individuals
+   tests from one :class:`TestCase` class.  *tests* is an iterable of
+   :class:`TestCase` instances created from the class.  *collected_from* is the
+   class they came from.
+
 
 .. class:: TestLoader()
 
@@ -553,6 +636,11 @@ Methods in the first group (running the test) are:
    The same effect may be had by simply calling the :class:`TestCase` instance.
 
 
+.. method:: TestCase.skip(reason)
+
+   Skips the current test.  See :ref:`unittest-skipping`.
+
+
 .. method:: TestCase.debug()
 
    Run the test without collecting the result.  This allows exceptions raised by
@@ -690,10 +778,11 @@ test:
 TestSuite Objects
 -----------------
 
-:class:`TestSuite` objects behave much like :class:`TestCase` objects, except
-they do not actually implement a test.  Instead, they are used to aggregate
-tests into groups of tests that should be run together. Some additional methods
-are available to add tests to :class:`TestSuite` instances:
+:class:`TestSuite` (including :class:`ClassTestSuite`) objects behave much like
+:class:`TestCase` objects, except they do not actually implement a test.
+Instead, they are used to aggregate tests into groups of tests that should be
+run together. Some additional methods are available to add tests to
+:class:`TestSuite` instances:
 
 
 .. method:: TestSuite.addTest(test)
@@ -846,6 +935,34 @@ tools which support interactive reporting while tests are being run.
    The default implementation does nothing.
 
 
+.. method:: TestResult.addSkip(test, reason)
+
+   Called when the test case *test* is skipped.  *reason* is the reason the test
+   gave for skipping.
+
+   The default implementation appends a tuple ``(test, reason)`` to the
+   instance's ``skipped`` attribute.
+
+
+.. method:: TestResult.addExpectedFailure(test, err)
+
+   Called when the test case *test* fails, but was marked with the
+   :func:`expectedFailure` decorator.
+
+   The default implementation appends a tuple ``(test, formatted_err)`` to the
+   instance's ``expected_failures`` attribute, where *formatted_err* is a
+   formatted traceback derived from *err*.
+
+
+.. method:: TestResult.addUnexpectedSuccess(test)
+
+   Called when the test case *test* was marked with the :func:`expectedFailure`
+   decorator, but succeeded.
+
+   The default implementation appends the test to the instance's
+   ``unexpected_successes`` attribute.
+
+
 .. _testloader-objects:
 
 TestLoader Objects
@@ -946,3 +1063,9 @@ subclassing or assignment on an instance:
 
    This affects all the :meth:`loadTestsFrom\*` methods.
 
+
+.. attribute:: TestLoader.classSuiteClass
+
+   Callable object that constructs a test suite for the tests cases from one
+   class.  The default value is :class:`ClassTestSuite`.
+

Lib/test/test_unittest.py

@@ -31,10 +31,27 @@ class LoggingResult(unittest.TestResult):
         self._events.append('addFailure')
         super(LoggingResult, self).addFailure(*args)
 
+    def addSuccess(self, *args):
+        self._events.append('addSuccess')
+        super(LoggingResult, self).addSuccess(*args)
+
     def addError(self, *args):
         self._events.append('addError')
         super(LoggingResult, self).addError(*args)
 
+    def addSkip(self, *args):
+        self._events.append('addSkip')
+        super(LoggingResult, self).addSkip(*args)
+
+    def addExpectedFailure(self, *args):
+        self._events.append('addExpectedFailure')
+        super(LoggingResult, self).addExpectedFailure(*args)
+
+    def addUnexpectedSuccess(self, *args):
+        self._events.append('addUnexpectedSuccess')
+        super(LoggingResult, self).addUnexpectedSuccess(*args)
+
+
 class TestEquality(object):
     # Check for a valid __eq__ implementation
     def test_eq(self):
@@ -72,6 +89,13 @@ class TestHashing(object):
                 self.fail("Problem hashing %s and %s: %s" % (obj_1, obj_2, e))
 
 
+# List subclass we can add attributes to.
+class MyClassSuite(list):
+
+    def __init__(self, tests, klass):
+        super(MyClassSuite, self).__init__(tests)
+
+
 ################################################################
 ### /Support code
 
@@ -1223,7 +1247,7 @@ class Test_TestLoader(TestCase):
         tests = [Foo('test_1'), Foo('test_2')]
 
         loader = unittest.TestLoader()
-        loader.suiteClass = list
+        loader.classSuiteClass = MyClassSuite
         self.assertEqual(loader.loadTestsFromTestCase(Foo), tests)
 
     # It is implicit in the documentation for TestLoader.suiteClass that
@@ -1236,7 +1260,7 @@ class Test_TestLoader(TestCase):
             def foo_bar(self): pass
         m.Foo = Foo
 
-        tests = [[Foo('test_1'), Foo('test_2')]]
+        tests = [unittest.ClassTestSuite([Foo('test_1'), Foo('test_2')], Foo)]
 
         loader = unittest.TestLoader()
         loader.suiteClass = list
@@ -1255,7 +1279,7 @@ class Test_TestLoader(TestCase):
         tests = [Foo('test_1'), Foo('test_2')]
 
         loader = unittest.TestLoader()
-        loader.suiteClass = list
+        loader.classSuiteClass = MyClassSuite
         self.assertEqual(loader.loadTestsFromName('Foo', m), tests)
 
     # It is implicit in the documentation for TestLoader.suiteClass that
@@ -1268,7 +1292,7 @@ class Test_TestLoader(TestCase):
             def foo_bar(self): pass
         m.Foo = Foo
 
-        tests = [[Foo('test_1'), Foo('test_2')]]
+        tests = [unittest.ClassTestSuite([Foo('test_1'), Foo('test_2')], Foo)]
 
         loader = unittest.TestLoader()
         loader.suiteClass = list
@@ -2261,9 +2285,103 @@ class Test_TestCase(TestCase, TestEquality, TestHashing):
         # Make run() find a result object on its own
         Foo('test').run()
 
-        expected = ['startTest', 'test', 'stopTest']
+        expected = ['startTest', 'test', 'addSuccess', 'stopTest']
         self.assertEqual(events, expected)
 
+
+class Test_TestSkipping(TestCase):
+
+    def test_skipping(self):
+        class Foo(unittest.TestCase):
+            def test_skip_me(self):
+                self.skip("skip")
+        events = []
+        result = LoggingResult(events)
+        test = Foo("test_skip_me")
+        test.run(result)
+        self.assertEqual(events, ['startTest', 'addSkip', 'stopTest'])
+        self.assertEqual(result.skipped, [(test, "skip")])
+
+        # Try letting setUp skip the test now.
+        class Foo(unittest.TestCase):
+            def setUp(self):
+                self.skip("testing")
+            def test_nothing(self): pass
+        events = []
+        result = LoggingResult(events)
+        test = Foo("test_nothing")
+        test.run(result)
+        self.assertEqual(events, ['startTest', 'addSkip', 'stopTest'])
+        self.assertEqual(result.skipped, [(test, "testing")])
+        self.assertEqual(result.testsRun, 1)
+
+    def test_skipping_decorators(self):
+        op_table = ((unittest.skipUnless, False, True),
+                    (unittest.skipIf, True, False))
+        for deco, do_skip, dont_skip in op_table:
+            class Foo(unittest.TestCase):
+                @deco(do_skip, "testing")
+                def test_skip(self): pass
+
+                @deco(dont_skip, "testing")
+                def test_dont_skip(self): pass
+            test_do_skip = Foo("test_skip")
+            test_dont_skip = Foo("test_dont_skip")
+            suite = unittest.ClassTestSuite([test_do_skip, test_dont_skip], Foo)
+            events = []
+            result = LoggingResult(events)
+            suite.run(result)
+            self.assertEqual(len(result.skipped), 1)
+            expected = ['startTest', 'addSkip', 'stopTest',
+                        'startTest', 'addSuccess', 'stopTest']
+            self.assertEqual(events, expected)
+            self.assertEqual(result.testsRun, 2)
+            self.assertEqual(result.skipped, [(test_do_skip, "testing")])
+            self.assertTrue(result.wasSuccessful())
+
+    def test_skip_class(self):
+        @unittest.skip("testing")
+        class Foo(unittest.TestCase):
+            def test_1(self):
+                record.append(1)
+        record = []
+        result = unittest.TestResult()
+        suite = unittest.ClassTestSuite([Foo("test_1")], Foo)
+        suite.run(result)
+        self.assertEqual(result.skipped, [(suite, "testing")])
+        self.assertEqual(record, [])
+
+    def test_expected_failure(self):
+        class Foo(unittest.TestCase):
+            @unittest.expectedFailure
+            def test_die(self):
+                self.fail("help me!")
+        events = []
+        result = LoggingResult(events)
+        test = Foo("test_die")
+        test.run(result)
+        self.assertEqual(events,
+                         ['startTest', 'addExpectedFailure', 'stopTest'])
+        self.assertEqual(result.expected_failures[0][0], test)
+        self.assertTrue(result.wasSuccessful())
+
+    def test_unexpected_success(self):
+        class Foo(unittest.TestCase):
+            @unittest.expectedFailure
+            def test_die(self):
+                pass
+        events = []
+        result = LoggingResult(events)
+        test = Foo("test_die")
+        test.run(result)
+        self.assertEqual(events,
+                         ['startTest', 'addUnexpectedSuccess', 'stopTest'])
+        self.assertFalse(result.failures)
+        self.assertEqual(result.unexpected_successes, [test])
+        self.assertTrue(result.wasSuccessful())
+
+
+
 class Test_Assertions(TestCase):
     def test_AlmostEqual(self):
         self.failUnlessAlmostEqual(1.00000001, 1.0)
@@ -2328,7 +2446,7 @@ class Test_Assertions(TestCase):
 def test_main():
     test_support.run_unittest(Test_TestCase, Test_TestLoader,
         Test_TestSuite, Test_TestResult, Test_FunctionTestCase,
-        Test_Assertions)
+        Test_TestSkipping, Test_Assertions)
 
 if __name__ == "__main__":
     test_main()

Misc/NEWS

@@ -197,6 +197,8 @@ Core and Builtins
 Library
 -------
 
+- Issue #1034053: unittest now supports skipping tests and expected failures.
+
 - Issue #5068: Fixed the tarfile._BZ2Proxy.read() method that would loop
   forever on incomplete input. That caused tarfile.open() to hang when used
   with mode 'r' or 'r:bz2' and a fileobj argument that contained no data or