libtest.tex 10.1 KB
Newer Older
1 2 3 4 5
\section{\module{test} ---
         Regression tests package for Python}

\declaremodule{standard}{test}
\sectionauthor{Brett Cannon}{brett@python.org}
Fred Drake's avatar
Fred Drake committed
6 7
\modulesynopsis{Regression tests package containing the testing suite
                for Python.}
8 9


Fred Drake's avatar
Fred Drake committed
10 11 12 13
The \module{test} package contains all regression tests for Python as
well as the modules \module{test.test_support} and
\module{test.regrtest}.  \module{test.test_support} is used to enhance
your tests while \module{test.regrtest} drives the testing suite.
14 15

Each module in the \module{test} package whose name starts with
Fred Drake's avatar
Fred Drake committed
16 17 18 19 20 21
\samp{test_} is a testing suite for a specific module or feature.
All new tests should be written using the \refmodule{unittest} module;
using \refmodule{unittest} is not required but makes the tests more
flexible and maintenance of the tests easier.  Some older tests are
written to use \refmodule{doctest} and a ``traditional'' testing
style; these styles of tests will not be covered.
22 23 24 25 26 27 28

\begin{seealso}
\seemodule{unittest}{Writing PyUnit regression tests.}
\seemodule{doctest}{Tests embedded in documentation strings.}
\end{seealso}


Fred Drake's avatar
Fred Drake committed
29 30
\subsection{Writing Unit Tests for the \module{test} package%
            \label{writing-tests}}
31 32

It is preferred that tests for the \module{test} package use the
Fred Drake's avatar
Fred Drake committed
33
\refmodule{unittest} module and follow a few guidelines.
34 35 36 37
One is to name the test module by starting it with \samp{test_} and end it with
the name of the module being tested.
The test methods in the test module should start with \samp{test_} and end with
a description of what the method is testing.
38 39 40 41
This is needed so that the methods are recognized by the test driver as
test methods.
Also, no documentation string for the method should be included.
A comment (such as
Fred Drake's avatar
Fred Drake committed
42
\samp{\# Tests function returns only True or False}) should be used to provide
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87
documentation for test methods.
This is done because documentation strings get printed out if they exist and
thus what test is being run is not stated.

A basic boilerplate is often used:

\begin{verbatim}
import unittest
from test import test_support

class MyTestCase1(unittest.TestCase):

    # Only use setUp() and tearDown() if necessary

    def setUp(self):
        ... code to execute in preparation for tests ...

    def tearDown(self):
        ... code to execute to clean up after tests ...

    def test_feature_one(self):
        # Test feature one.
        ... testing code ...

    def test_feature_two(self):
        # Test feature two.
        ... testing code ...

    ... more test methods ...

class MyTestCase2(unittest.TestCase):
    ... same structure as MyTestCase1 ...

... more test classes ...

def test_main():
    test_support.run_unittest(MyTestCase1,
                              MyTestCase2,
                              ... list other tests ...
                             )

if __name__ == '__main__':
    test_main()
\end{verbatim}

Fred Drake's avatar
Fred Drake committed
88 89
This boilerplate code allows the testing suite to be run by
\module{test.regrtest} as well as on its own as a script.
90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113

The goal for regression testing is to try to break code.
This leads to a few guidelines to be followed:

\begin{itemize}
\item The testing suite should exercise all classes, functions, and
      constants.
      This includes not just the external API that is to be presented to the
      outside world but also "private" code.
\item Whitebox testing (examining the code being tested when the tests are
      being written) is preferred.
      Blackbox testing (testing only the published user interface) is not
      complete enough to make sure all boundary and edge cases are tested.
\item Make sure all possible values are tested including invalid ones.
      This makes sure that not only all valid values are acceptable but also
      that improper values are handled correctly.
\item Exhaust as many code paths as possible.
      Test where branching occurs and thus tailor input to make sure as many
      different paths through the code are taken.
\item Add an explicit test for any bugs discovered for the tested code.
      This will make sure that the error does not crop up again if the code is
      changed in the future.
\item Make sure to clean up after your tests (such as close and remove all
      temporary files).
114 115
\item If a test is dependent on a specific condition of the operating system
      then verify the condition already exists before attempting the test.
116 117 118 119
\item Import as few modules as possible and do it as soon as possible.
      This minimizes external dependencies of tests and also minimizes possible
      anomalous behavior from side-effects of importing a module.
\item Try to maximize code reuse.
Fred Drake's avatar
Fred Drake committed
120 121
      On occasion, tests will vary by something as small as what type
      of input is used.
122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143
      Minimize code duplication by subclassing a basic test class with a class
      that specifies the input:
\begin{verbatim}
class TestFuncAcceptsSequences(unittest.TestCase):

    func = mySuperWhammyFunction

    def test_func(self):
        self.func(self.arg)

class AcceptLists(TestFuncAcceptsSequences):
    arg = [1,2,3]

class AcceptStrings(TestFuncAcceptsSequences):
    arg = 'abc'

class AcceptTuples(TestFuncAcceptsSequences):
    arg = (1,2,3)
\end{verbatim}
\end{itemize}

\begin{seealso}
Fred Drake's avatar
Fred Drake committed
144 145
\seetitle{Test Driven Development}
         {A book by Kent Beck on writing tests before code.}
146 147 148
\end{seealso}


149
\subsection{Running tests using \module{test.regrtest} \label{regrtest}}
150

Fred Drake's avatar
Fred Drake committed
151 152
\module{test.regrtest} can be used as a script to drive Python's
regression test suite.
153 154 155
Running the script by itself automatically starts running all
regression tests in the \module{test} package.
It does this by finding all modules in the package whose name starts with
Fred Drake's avatar
Fred Drake committed
156 157
\samp{test_}, importing them, and executing the function
\function{test_main()} if present.
158
The names of tests to execute may also be passed to the script.
Fred Drake's avatar
Fred Drake committed
159 160 161
Specifying a single regression test (\program{python regrtest.py}
\programopt{test_spam.py}) will minimize output and only print whether
the test passed or failed and thus minimize output.
162

Fred Drake's avatar
Fred Drake committed
163
Running \module{test.regrtest} directly allows what resources are
164
available for tests to use to be set.
Fred Drake's avatar
Fred Drake committed
165 166 167 168
You do this by using the \programopt{-u} command-line option.
Run \program{python regrtest.py} \programopt{-uall} to turn on all
resources; specifying \programopt{all} as an option for
\programopt{-u} enables all possible resources.
169 170
If all but one resource is desired (a more common case), a
comma-separated list of resources that are not desired may be listed after
Fred Drake's avatar
Fred Drake committed
171 172 173 174 175
\programopt{all}.
The command \program{python regrtest.py}
\programopt{-uall,-audio,-largefile} will run \module{test.regrtest}
with all resources except the \programopt{audio} and
\programopt{largefile} resources.
176
For a list of all resources and more command-line options, run
Fred Drake's avatar
Fred Drake committed
177
\program{python regrtest.py} \programopt{-h}.
178 179 180

Some other ways to execute the regression tests depend on what platform the
tests are being executed on.
Fred Drake's avatar
Fred Drake committed
181 182 183 184 185 186 187 188 189 190 191 192 193
On \UNIX{}, you can run \program{make} \programopt{test} at the
top-level directory where Python was built.
On Windows, executing \program{rt.bat} from your \file{PCBuild}
directory will run all regression tests.


\section{\module{test.test_support} ---
         Utility functions for tests}

\declaremodule[test.testsupport]{standard}{test.test_support}
\modulesynopsis{Support for Python regression tests.}

The \module{test.test_support} module provides support for Python's
194 195
regression tests.

Fred Drake's avatar
Fred Drake committed
196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278
This module defines the following exceptions:

\begin{excdesc}{TestFailed}
Exception to be raised when a test fails.
\end{excdesc}

\begin{excdesc}{TestSkipped}
Subclass of \exception{TestFailed}.
Raised when a test is skipped.
This occurs when a needed resource (such as a network connection) is not
available at the time of testing.
\end{excdesc}

\begin{excdesc}{ResourceDenied}
Subclass of \exception{TestSkipped}.
Raised when a resource (such as a network connection) is not available.
Raised by the \function{requires()} function.
\end{excdesc}


The \module{test.test_support} module defines the following constants:

\begin{datadesc}{verbose}
\constant{True} when verbose output is enabled.
Should be checked when more detailed information is desired about a running
test.
\var{verbose} is set by \module{test.regrtest}.
\end{datadesc}

\begin{datadesc}{have_unicode}
\constant{True} when Unicode support is available.
\end{datadesc}

\begin{datadesc}{is_jython}
\constant{True} if the running interpreter is Jython.
\end{datadesc}

\begin{datadesc}{TESTFN}
Set to the path that a temporary file may be created at.
Any temporary that is created should be closed and unlinked (removed).
\end{datadesc}


The \module{test.test_support} module defines the following functions:

\begin{funcdesc}{forget}{module_name}
Removes the module named \var{module_name} from \code{sys.modules} and deletes
any byte-compiled files of the module.
\end{funcdesc}

\begin{funcdesc}{is_resource_enabled}{resource}
Returns \constant{True} if \var{resource} is enabled and available.
The list of available resources is only set when \module{test.regrtest}
is executing the tests.
\end{funcdesc}

\begin{funcdesc}{requires}{resource\optional{, msg}}
Raises \exception{ResourceDenied} if \var{resource} is not available.
\var{msg} is the argument to \exception{ResourceDenied} if it is raised.
Always returns true if called by a function whose \code{__name__} is
\code{'__main__'}.
Used when tests are executed by \module{test.regrtest}.
\end{funcdesc}

\begin{funcdesc}{findfile}{filename}
Return the path to the file named \var{filename}.
If no match is found \var{filename} is returned.
This does not equal a failure since it could be the path to the file.
\end{funcdesc}

\begin{funcdesc}{run_unittest}{*classes}
Execute \class{unittest.TestCase} subclasses passed to the function.
The function scans the classes for methods starting with the prefix
\samp{test_} and executes the tests individually.
This is the preferred way to execute tests.
\end{funcdesc}

\begin{funcdesc}{run_suite}{suite\optional{, testclass}}
Execute the \class{unittest.TestSuite} instance \var{suite}.
The optional argument \var{testclass} accepts one of the test classes in the
suite so as to print out more detailed information on where the testing suite
originated from.
\end{funcdesc}