stdlib2.rst 14.8 KB
Newer Older
1 2
.. _tut-brieftourtwo:

3
**********************************************
4 5
Brief Tour of the Standard Library --- Part II
**********************************************
6 7 8 9 10 11 12 13 14 15

This second tour covers more advanced modules that support professional
programming needs.  These modules rarely occur in small scripts.


.. _tut-output-formatting:

Output Formatting
=================

16
The :mod:`reprlib` module provides a version of :func:`repr` customized for
17 18
abbreviated displays of large or deeply nested containers::

19 20
   >>> import reprlib
   >>> reprlib.repr(set('supercalifragilisticexpialidocious'))
21
   "{'a', 'c', 'd', 'e', 'f', 'g', ...}"
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

The :mod:`pprint` module offers more sophisticated control over printing both
built-in and user defined objects in a way that is readable by the interpreter.
When the result is longer than one line, the "pretty printer" adds line breaks
and indentation to more clearly reveal data structure::

   >>> import pprint
   >>> t = [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta',
   ...     'yellow'], 'blue']]]
   ...
   >>> pprint.pprint(t, width=30)
   [[[['black', 'cyan'],
      'white',
      ['green', 'red']],
     [['magenta', 'yellow'],
      'blue']]]

The :mod:`textwrap` module formats paragraphs of text to fit a given screen
width::

   >>> import textwrap
   >>> doc = """The wrap() method is just like fill() except that it returns
   ... a list of strings instead of one big string with newlines to separate
   ... the wrapped lines."""
   ...
47
   >>> print(textwrap.fill(doc, width=40))
48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
   The wrap() method is just like fill()
   except that it returns a list of strings
   instead of one big string with newlines
   to separate the wrapped lines.

The :mod:`locale` module accesses a database of culture specific data formats.
The grouping attribute of locale's format function provides a direct way of
formatting numbers with group separators::

   >>> import locale
   >>> locale.setlocale(locale.LC_ALL, 'English_United States.1252')
   'English_United States.1252'
   >>> conv = locale.localeconv()          # get a mapping of conventions
   >>> x = 1234567.8
   >>> locale.format("%d", x, grouping=True)
   '1,234,567'
64 65
   >>> locale.format_string("%s%.*f", (conv['currency_symbol'],
   ...                      conv['frac_digits'], x), grouping=True)
66 67 68 69 70 71 72 73
   '$1,234,567.80'


.. _tut-templating:

Templating
==========

74 75 76
The :mod:`string` module includes a versatile :class:`~string.Template` class
with a simplified syntax suitable for editing by end-users.  This allows users
to customize their applications without having to alter the application.
77 78 79 80 81 82 83 84 85 86 87

The format uses placeholder names formed by ``$`` with valid Python identifiers
(alphanumeric characters and underscores).  Surrounding the placeholder with
braces allows it to be followed by more alphanumeric letters with no intervening
spaces.  Writing ``$$`` creates a single escaped ``$``::

   >>> from string import Template
   >>> t = Template('${village}folk send $$10 to $cause.')
   >>> t.substitute(village='Nottingham', cause='the ditch fund')
   'Nottinghamfolk send $10 to the ditch fund.'

88 89 90 91 92
The :meth:`~string.Template.substitute` method raises a :exc:`KeyError` when a
placeholder is not supplied in a dictionary or a keyword argument.  For
mail-merge style applications, user supplied data may be incomplete and the
:meth:`~string.Template.safe_substitute` method may be more appropriate ---
it will leave placeholders unchanged if data is missing::
93 94 95 96 97

   >>> t = Template('Return the $item to $owner.')
   >>> d = dict(item='unladen swallow')
   >>> t.substitute(d)
   Traceback (most recent call last):
98
     ...
99 100 101 102 103 104 105 106
   KeyError: 'owner'
   >>> t.safe_substitute(d)
   'Return the unladen swallow to $owner.'

Template subclasses can specify a custom delimiter.  For example, a batch
renaming utility for a photo browser may elect to use percent signs for
placeholders such as the current date, image sequence number, or file format::

107
   >>> import time, os.path
108 109 110
   >>> photofiles = ['img_1074.jpg', 'img_1076.jpg', 'img_1077.jpg']
   >>> class BatchRename(Template):
   ...     delimiter = '%'
111
   >>> fmt = input('Enter rename style (%d-date %n-seqnum %f-format):  ')
112 113 114 115 116 117 118
   Enter rename style (%d-date %n-seqnum %f-format):  Ashley_%n%f

   >>> t = BatchRename(fmt)
   >>> date = time.strftime('%d%b%y')
   >>> for i, filename in enumerate(photofiles):
   ...     base, ext = os.path.splitext(filename)
   ...     newname = t.substitute(d=date, n=i, f=ext)
119
   ...     print('{0} --> {1}'.format(filename, newname))
120 121 122 123 124 125 126 127 128 129 130 131 132 133 134

   img_1074.jpg --> Ashley_0.jpg
   img_1076.jpg --> Ashley_1.jpg
   img_1077.jpg --> Ashley_2.jpg

Another application for templating is separating program logic from the details
of multiple output formats.  This makes it possible to substitute custom
templates for XML files, plain text reports, and HTML web reports.


.. _tut-binary-formats:

Working with Binary Data Record Layouts
=======================================

135 136 137
The :mod:`struct` module provides :func:`~struct.pack` and
:func:`~struct.unpack` functions for working with variable length binary
record formats.  The following example shows
138 139 140 141
how to loop through header information in a ZIP file without using the
:mod:`zipfile` module.  Pack codes ``"H"`` and ``"I"`` represent two and four
byte unsigned numbers respectively.  The ``"<"`` indicates that they are
standard size and in little-endian byte order::
142 143 144

   import struct

145 146 147
   with open('myfile.zip', 'rb') as f:
       data = f.read()

148 149 150
   start = 0
   for i in range(3):                      # show the first 3 file headers
       start += 14
151
       fields = struct.unpack('<IIIHH', data[start:start+16])
152 153 154 155 156 157
       crc32, comp_size, uncomp_size, filenamesize, extra_size = fields

       start += 16
       filename = data[start:start+filenamesize]
       start += filenamesize
       extra = data[start:start+extra_size]
158
       print(filename, hex(crc32), comp_size, uncomp_size)
159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179

       start += extra_size + comp_size     # skip to the next header


.. _tut-multi-threading:

Multi-threading
===============

Threading is a technique for decoupling tasks which are not sequentially
dependent.  Threads can be used to improve the responsiveness of applications
that accept user input while other tasks run in the background.  A related use
case is running I/O in parallel with computations in another thread.

The following code shows how the high level :mod:`threading` module can run
tasks in background while the main program continues to run::

   import threading, zipfile

   class AsyncZip(threading.Thread):
       def __init__(self, infile, outfile):
180
           threading.Thread.__init__(self)
181 182
           self.infile = infile
           self.outfile = outfile
183

184 185 186 187
       def run(self):
           f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED)
           f.write(self.infile)
           f.close()
188
           print('Finished background zip of:', self.infile)
189 190 191

   background = AsyncZip('mydata.txt', 'myarchive.zip')
   background.start()
192
   print('The main program continues to run in foreground.')
193 194

   background.join()    # Wait for the background task to finish
195
   print('Main program waited until background was done.')
196 197 198 199 200 201 202 203 204

The principal challenge of multi-threaded applications is coordinating threads
that share data or other resources.  To that end, the threading module provides
a number of synchronization primitives including locks, events, condition
variables, and semaphores.

While those tools are powerful, minor design errors can result in problems that
are difficult to reproduce.  So, the preferred approach to task coordination is
to concentrate all access to a resource in a single thread and then use the
205
:mod:`queue` module to feed that thread with requests from other threads.
206
Applications using :class:`~queue.Queue` objects for inter-thread communication and
207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224
coordination are easier to design, more readable, and more reliable.


.. _tut-logging:

Logging
=======

The :mod:`logging` module offers a full featured and flexible logging system.
At its simplest, log messages are sent to a file or to ``sys.stderr``::

   import logging
   logging.debug('Debugging information')
   logging.info('Informational message')
   logging.warning('Warning:config file %s not found', 'server.conf')
   logging.error('Error occurred')
   logging.critical('Critical error -- shutting down')

225 226 227
This produces the following output:

.. code-block:: none
228 229 230 231 232 233 234 235

   WARNING:root:Warning:config file server.conf not found
   ERROR:root:Error occurred
   CRITICAL:root:Critical error -- shutting down

By default, informational and debugging messages are suppressed and the output
is sent to standard error.  Other output options include routing messages
through email, datagrams, sockets, or to an HTTP Server.  New filters can select
236 237 238
different routing based on message priority: :const:`~logging.DEBUG`,
:const:`~logging.INFO`, :const:`~logging.WARNING`, :const:`~logging.ERROR`,
and :const:`~logging.CRITICAL`.
239 240 241 242 243 244 245 246 247 248 249 250

The logging system can be configured directly from Python or can be loaded from
a user editable configuration file for customized logging without altering the
application.


.. _tut-weak-references:

Weak References
===============

Python does automatic memory management (reference counting for most objects and
251 252
:term:`garbage collection` to eliminate cycles).  The memory is freed shortly
after the last reference to it has been eliminated.
253 254 255 256 257 258 259 260 261 262 263 264

This approach works fine for most applications but occasionally there is a need
to track objects only as long as they are being used by something else.
Unfortunately, just tracking them creates a reference that makes them permanent.
The :mod:`weakref` module provides tools for tracking objects without creating a
reference.  When the object is no longer needed, it is automatically removed
from a weakref table and a callback is triggered for weakref objects.  Typical
applications include caching objects that are expensive to create::

   >>> import weakref, gc
   >>> class A:
   ...     def __init__(self, value):
265
   ...         self.value = value
266
   ...     def __repr__(self):
267
   ...         return str(self.value)
268 269 270 271 272 273 274 275 276 277 278
   ...
   >>> a = A(10)                   # create a reference
   >>> d = weakref.WeakValueDictionary()
   >>> d['primary'] = a            # does not create a reference
   >>> d['primary']                # fetch the object if it is still alive
   10
   >>> del a                       # remove the one reference
   >>> gc.collect()                # run garbage collection right away
   0
   >>> d['primary']                # entry was automatically removed
   Traceback (most recent call last):
Christian Heimes's avatar
Christian Heimes committed
279
     File "<stdin>", line 1, in <module>
280
       d['primary']                # entry was automatically removed
Ned Deily's avatar
Ned Deily committed
281
     File "C:/python38/lib/weakref.py", line 46, in __getitem__
282 283 284 285 286 287 288 289 290 291 292 293 294
       o = self.data[key]()
   KeyError: 'primary'


.. _tut-list-tools:

Tools for Working with Lists
============================

Many data structure needs can be met with the built-in list type. However,
sometimes there is a need for alternative implementations with different
performance trade-offs.

295 296 297 298 299
The :mod:`array` module provides an :class:`~array.array()` object that is like
a list that stores only homogeneous data and stores it more compactly.  The
following example shows an array of numbers stored as two byte unsigned binary
numbers (typecode ``"H"``) rather than the usual 16 bytes per entry for regular
lists of Python int objects::
300 301 302 303 304 305 306 307

   >>> from array import array
   >>> a = array('H', [4000, 10, 700, 22222])
   >>> sum(a)
   26932
   >>> a[1:3]
   array('H', [10, 700])

308 309 310 311
The :mod:`collections` module provides a :class:`~collections.deque()` object
that is like a list with faster appends and pops from the left side but slower
lookups in the middle. These objects are well suited for implementing queues
and breadth first tree searches::
312 313 314 315

   >>> from collections import deque
   >>> d = deque(["task1", "task2", "task3"])
   >>> d.append("task4")
316
   >>> print("Handling", d.popleft())
317 318
   Handling task1

319 320
::

321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356
   unsearched = deque([starting_node])
   def breadth_first_search(unsearched):
       node = unsearched.popleft()
       for m in gen_moves(node):
           if is_goal(m):
               return m
           unsearched.append(m)

In addition to alternative list implementations, the library also offers other
tools such as the :mod:`bisect` module with functions for manipulating sorted
lists::

   >>> import bisect
   >>> scores = [(100, 'perl'), (200, 'tcl'), (400, 'lua'), (500, 'python')]
   >>> bisect.insort(scores, (300, 'ruby'))
   >>> scores
   [(100, 'perl'), (200, 'tcl'), (300, 'ruby'), (400, 'lua'), (500, 'python')]

The :mod:`heapq` module provides functions for implementing heaps based on
regular lists.  The lowest valued entry is always kept at position zero.  This
is useful for applications which repeatedly access the smallest element but do
not want to run a full list sort::

   >>> from heapq import heapify, heappop, heappush
   >>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0]
   >>> heapify(data)                      # rearrange the list into heap order
   >>> heappush(data, -5)                 # add a new entry
   >>> [heappop(data) for i in range(3)]  # fetch the three smallest entries
   [-5, 0, 1]


.. _tut-decimal-fp:

Decimal Floating Point Arithmetic
=================================

357 358
The :mod:`decimal` module offers a :class:`~decimal.Decimal` datatype for
decimal floating point arithmetic.  Compared to the built-in :class:`float`
359 360 361 362 363 364 365 366 367
implementation of binary floating point, the class is especially helpful for

* financial applications and other uses which require exact decimal
  representation,
* control over precision,
* control over rounding to meet legal or regulatory requirements,
* tracking of significant decimal places, or
* applications where the user expects the results to match calculations done by
  hand.
368 369 370 371 372

For example, calculating a 5% tax on a 70 cent phone charge gives different
results in decimal floating point and binary floating point. The difference
becomes significant if the results are rounded to the nearest cent::

373
   >>> from decimal import *
374 375 376 377
   >>> round(Decimal('0.70') * Decimal('1.05'), 2)
   Decimal('0.74')
   >>> round(.70 * 1.05, 2)
   0.73
378

379 380 381 382 383
The :class:`~decimal.Decimal` result keeps a trailing zero, automatically
inferring four place significance from multiplicands with two place
significance.  Decimal reproduces mathematics as done by hand and avoids
issues that can arise when binary floating point cannot exactly represent
decimal quantities.
384

385 386 387
Exact representation enables the :class:`~decimal.Decimal` class to perform
modulo calculations and equality tests that are unsuitable for binary floating
point::
388 389

   >>> Decimal('1.00') % Decimal('.10')
390
   Decimal('0.00')
391 392 393 394 395 396
   >>> 1.00 % 0.10
   0.09999999999999995

   >>> sum([Decimal('0.1')]*10) == Decimal('1.0')
   True
   >>> sum([0.1]*10) == 1.0
397
   False
398 399 400 401 402

The :mod:`decimal` module provides arithmetic with as much precision as needed::

   >>> getcontext().prec = 36
   >>> Decimal(1) / Decimal(7)
403
   Decimal('0.142857142857142857142857142857142857')
404 405