pickle.rst 36.3 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13
:mod:`pickle` --- Python object serialization
=============================================

.. index::
   single: persistence
   pair: persistent; objects
   pair: serializing; objects
   pair: marshalling; objects
   pair: flattening; objects
   pair: pickling; objects

.. module:: pickle
   :synopsis: Convert Python objects to streams of bytes and back.
14
.. sectionauthor:: Jim Kerr <jbkerr@sr.hp.com>.
Andrew Kuchling's avatar
Andrew Kuchling committed
15
.. sectionauthor:: Barry Warsaw <barry@python.org>
16

17

18 19 20 21 22 23 24 25
The :mod:`pickle` module implements binary protocols for serializing and
de-serializing a Python object structure.  *"Pickling"* is the process
whereby a Python object hierarchy is converted into a byte stream, and
*"unpickling"* is the inverse operation, whereby a byte stream
(from a :term:`binary file` or :term:`bytes-like object`) is converted
back into an object hierarchy.  Pickling (and unpickling) is alternatively
known as "serialization", "marshalling," [#]_ or "flattening"; however, to
avoid confusion, the terms used here are "pickling" and "unpickling".
26

27 28 29 30 31 32
.. warning::

   The :mod:`pickle` module is not intended to be secure against erroneous or
   maliciously constructed data.  Never unpickle data received from an untrusted
   or unauthenticated source.

33 34 35 36

Relationship to other Python modules
------------------------------------

37 38
Comparison with ``marshal``
^^^^^^^^^^^^^^^^^^^^^^^^^^^
39 40 41 42 43 44

Python has a more primitive serialization module called :mod:`marshal`, but in
general :mod:`pickle` should always be the preferred way to serialize Python
objects.  :mod:`marshal` exists primarily to support Python's :file:`.pyc`
files.

Georg Brandl's avatar
Georg Brandl committed
45
The :mod:`pickle` module differs from :mod:`marshal` in several significant ways:
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71

* The :mod:`pickle` module keeps track of the objects it has already serialized,
  so that later references to the same object won't be serialized again.
  :mod:`marshal` doesn't do this.

  This has implications both for recursive objects and object sharing.  Recursive
  objects are objects that contain references to themselves.  These are not
  handled by marshal, and in fact, attempting to marshal recursive objects will
  crash your Python interpreter.  Object sharing happens when there are multiple
  references to the same object in different places in the object hierarchy being
  serialized.  :mod:`pickle` stores such objects only once, and ensures that all
  other references point to the master copy.  Shared objects remain shared, which
  can be very important for mutable objects.

* :mod:`marshal` cannot be used to serialize user-defined classes and their
  instances.  :mod:`pickle` can save and restore class instances transparently,
  however the class definition must be importable and live in the same module as
  when the object was stored.

* The :mod:`marshal` serialization format is not guaranteed to be portable
  across Python versions.  Because its primary job in life is to support
  :file:`.pyc` files, the Python implementers reserve the right to change the
  serialization format in non-backwards compatible ways should the need arise.
  The :mod:`pickle` serialization format is guaranteed to be backwards compatible
  across Python releases.

72 73
Comparison with ``json``
^^^^^^^^^^^^^^^^^^^^^^^^
74

75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
There are fundamental differences between the pickle protocols and
`JSON (JavaScript Object Notation) <http://json.org>`_:

* JSON is a text serialization format (it outputs unicode text, although
  most of the time it is then encoded to ``utf-8``), while pickle is
  a binary serialization format;

* JSON is human-readable, while pickle is not;

* JSON is interoperable and widely used outside of the Python ecosystem,
  while pickle is Python-specific;

* JSON, by default, can only represent a subset of the Python built-in
  types, and no custom classes; pickle can represent an extremely large
  number of Python types (many of them automatically, by clever usage
  of Python's introspection facilities; complex cases can be tackled by
  implementing :ref:`specific object APIs <pickle-inst>`).

.. seealso::
   The :mod:`json` module: a standard library module allowing JSON
   serialization and deserialization.
96

97 98 99

.. _pickle-protocols:

100 101 102 103 104 105 106 107
Data stream format
------------------

.. index::
   single: External Data Representation

The data format used by :mod:`pickle` is Python-specific.  This has the
advantage that there are no restrictions imposed by external standards such as
108 109 110 111 112 113
JSON or XDR (which can't represent pointer sharing); however it means that
non-Python programs may not be able to reconstruct pickled Python objects.

By default, the :mod:`pickle` data format uses a relatively compact binary
representation.  If you need optimal size characteristics, you can efficiently
:doc:`compress <archiving>` pickled data.
114

115
The module :mod:`pickletools` contains tools for analyzing data streams
116 117
generated by :mod:`pickle`.  :mod:`pickletools` source code has extensive
comments about opcodes used by pickle protocols.
118

119 120 121
There are currently 5 different protocols which can be used for pickling.
The higher the protocol used, the more recent the version of Python needed
to read the pickle produced.
122

123
* Protocol version 0 is the original "human-readable" protocol and is
124
  backwards compatible with earlier versions of Python.
125

126
* Protocol version 1 is an old binary format which is also compatible with
127 128 129
  earlier versions of Python.

* Protocol version 2 was introduced in Python 2.3.  It provides much more
130 131
  efficient pickling of :term:`new-style class`\es.  Refer to :pep:`307` for
  information about improvements brought by protocol 2.
132

133
* Protocol version 3 was added in Python 3.0.  It has explicit support for
134
  :class:`bytes` objects and cannot be unpickled by Python 2.x.  This is
135 136 137 138 139 140 141
  the default protocol, and the recommended protocol when compatibility with
  other Python 3 versions is required.

* Protocol version 4 was added in Python 3.4.  It adds support for very large
  objects, pickling more kinds of objects, and some data format
  optimizations.  Refer to :pep:`3154` for information about improvements
  brought by protocol 4.
142

143 144 145 146 147 148 149 150 151 152 153 154
.. note::
   Serialization is a more primitive notion than persistence; although
   :mod:`pickle` reads and writes file objects, it does not handle the issue of
   naming persistent objects, nor the (even more complicated) issue of concurrent
   access to persistent objects.  The :mod:`pickle` module can transform a complex
   object into a byte stream and it can transform the byte stream into an object
   with the same internal structure.  Perhaps the most obvious thing to do with
   these byte streams is to write them onto a file, but it is also conceivable to
   send them across a network or store them in a database.  The :mod:`shelve`
   module provides a simple interface to pickle and unpickle objects on
   DBM-style database files.

155

156 157
Module Interface
----------------
158

159 160 161 162 163 164
To serialize an object hierarchy, you simply call the :func:`dumps` function.
Similarly, to de-serialize a data stream, you call the :func:`loads` function.
However, if you want more control over serialization and de-serialization,
you can create a :class:`Pickler` or an :class:`Unpickler` object, respectively.

The :mod:`pickle` module provides the following constants:
165 166 167 168


.. data:: HIGHEST_PROTOCOL

169 170 171 172
   An integer, the highest :ref:`protocol version <pickle-protocols>`
   available.  This value can be passed as a *protocol* value to functions
   :func:`dump` and :func:`dumps` as well as the :class:`Pickler`
   constructor.
173

174 175
.. data:: DEFAULT_PROTOCOL

176 177
   An integer, the default :ref:`protocol version <pickle-protocols>` used
   for pickling.  May be less than :data:`HIGHEST_PROTOCOL`.  Currently the
178
   default protocol is 3, a new protocol designed for Python 3.
179 180


181 182 183
The :mod:`pickle` module provides the following functions to make the pickling
process more convenient:

184
.. function:: dump(obj, file, protocol=None, \*, fix_imports=True)
185

186 187
   Write a pickled representation of *obj* to the open :term:`file object` *file*.
   This is equivalent to ``Pickler(file, protocol).dump(obj)``.
188

189 190 191 192
   The optional *protocol* argument, an integer, tells the pickler to use
   the given protocol; supported protocols are 0 to :data:`HIGHEST_PROTOCOL`.
   If not specified, the default is :data:`DEFAULT_PROTOCOL`.  If a negative
   number is specified, :data:`HIGHEST_PROTOCOL` is selected.
193

194
   The *file* argument must have a write() method that accepts a single bytes
195 196 197
   argument.  It can thus be an on-disk file opened for binary writing, a
   :class:`io.BytesIO` instance, or any other custom object that meets this
   interface.
198

199
   If *fix_imports* is true and *protocol* is less than 3, pickle will try to
200 201
   map the new Python 3 names to the old module names used in Python 2, so
   that the pickle data stream is readable with Python 2.
202

203
.. function:: dumps(obj, protocol=None, \*, fix_imports=True)
204

205 206
   Return the pickled representation of the object as a :class:`bytes` object,
   instead of writing it to a file.
207

208 209
   Arguments *protocol* and *fix_imports* have the same meaning as in
   :func:`dump`.
210

211
.. function:: load(file, \*, fix_imports=True, encoding="ASCII", errors="strict")
212

213 214 215
   Read a pickled object representation from the open :term:`file object`
   *file* and return the reconstituted object hierarchy specified therein.
   This is equivalent to ``Unpickler(file).load()``.
216

217 218 219
   The protocol version of the pickle is detected automatically, so no
   protocol argument is needed.  Bytes past the pickled object's
   representation are ignored.
220

221 222
   The argument *file* must have two methods, a read() method that takes an
   integer argument, and a readline() method that requires no arguments.  Both
223 224
   methods should return bytes.  Thus *file* can be an on-disk file opened for
   binary reading, a :class:`io.BytesIO` object, or any other custom object
225
   that meets this interface.
226

227
   Optional keyword arguments are *fix_imports*, *encoding* and *errors*,
Georg Brandl's avatar
Georg Brandl committed
228
   which are used to control compatibility support for pickle stream generated
229 230
   by Python 2.  If *fix_imports* is true, pickle will try to map the old
   Python 2 names to the new names used in Python 3.  The *encoding* and
231
   *errors* tell pickle how to decode 8-bit string instances pickled by Python
232 233
   2; these default to 'ASCII' and 'strict', respectively.  The *encoding* can
   be 'bytes' to read these 8-bit string instances as bytes objects.
234

235
.. function:: loads(bytes_object, \*, fix_imports=True, encoding="ASCII", errors="strict")
236

237 238
   Read a pickled object hierarchy from a :class:`bytes` object and return the
   reconstituted object hierarchy specified therein
239

240 241 242
   The protocol version of the pickle is detected automatically, so no
   protocol argument is needed.  Bytes past the pickled object's
   representation are ignored.
243

244
   Optional keyword arguments are *fix_imports*, *encoding* and *errors*,
Georg Brandl's avatar
Georg Brandl committed
245
   which are used to control compatibility support for pickle stream generated
246 247
   by Python 2.  If *fix_imports* is true, pickle will try to map the old
   Python 2 names to the new names used in Python 3.  The *encoding* and
248
   *errors* tell pickle how to decode 8-bit string instances pickled by Python
249 250
   2; these default to 'ASCII' and 'strict', respectively.  The *encoding* can
   be 'bytes' to read these 8-bit string instances as bytes objects.
251 252 253


The :mod:`pickle` module defines three exceptions:
254 255 256

.. exception:: PickleError

257
   Common base class for the other pickling exceptions.  It inherits
258 259 260 261
   :exc:`Exception`.

.. exception:: PicklingError

262 263
   Error raised when an unpicklable object is encountered by :class:`Pickler`.
   It inherits :exc:`PickleError`.
264

265 266 267
   Refer to :ref:`pickle-picklable` to learn what kinds of objects can be
   pickled.

268 269
.. exception:: UnpicklingError

270
   Error raised when there is a problem unpickling an object, such as a data
271
   corruption or a security violation.  It inherits :exc:`PickleError`.
272

273 274 275
   Note that other exceptions may also be raised during unpickling, including
   (but not necessarily limited to) AttributeError, EOFError, ImportError, and
   IndexError.
276 277


278 279
The :mod:`pickle` module exports two classes, :class:`Pickler` and
:class:`Unpickler`:
280

281
.. class:: Pickler(file, protocol=None, \*, fix_imports=True)
282

283
   This takes a binary file for writing a pickle data stream.
284

285 286 287 288
   The optional *protocol* argument, an integer, tells the pickler to use
   the given protocol; supported protocols are 0 to :data:`HIGHEST_PROTOCOL`.
   If not specified, the default is :data:`DEFAULT_PROTOCOL`.  If a negative
   number is specified, :data:`HIGHEST_PROTOCOL` is selected.
289

290
   The *file* argument must have a write() method that accepts a single bytes
291
   argument.  It can thus be an on-disk file opened for binary writing, a
292 293
   :class:`io.BytesIO` instance, or any other custom object that meets this
   interface.
294

295
   If *fix_imports* is true and *protocol* is less than 3, pickle will try to
296 297
   map the new Python 3 names to the old module names used in Python 2, so
   that the pickle data stream is readable with Python 2.
298

299
   .. method:: dump(obj)
300

301 302 303 304
      Write a pickled representation of *obj* to the open file object given in
      the constructor.

   .. method:: persistent_id(obj)
305

306 307 308 309 310 311 312 313 314
      Do nothing by default.  This exists so a subclass can override it.

      If :meth:`persistent_id` returns ``None``, *obj* is pickled as usual.  Any
      other value causes :class:`Pickler` to emit the returned value as a
      persistent ID for *obj*.  The meaning of this persistent ID should be
      defined by :meth:`Unpickler.persistent_load`.  Note that the value
      returned by :meth:`persistent_id` cannot itself have a persistent ID.

      See :ref:`pickle-persistent` for details and examples of uses.
315

316 317 318 319 320 321 322
   .. attribute:: dispatch_table

      A pickler object's dispatch table is a registry of *reduction
      functions* of the kind which can be declared using
      :func:`copyreg.pickle`.  It is a mapping whose keys are classes
      and whose values are reduction functions.  A reduction function
      takes a single argument of the associated class and should
323
      conform to the same interface as a :meth:`__reduce__`
324 325 326 327 328 329 330 331 332 333 334 335 336 337 338
      method.

      By default, a pickler object will not have a
      :attr:`dispatch_table` attribute, and it will instead use the
      global dispatch table managed by the :mod:`copyreg` module.
      However, to customize the pickling for a specific pickler object
      one can set the :attr:`dispatch_table` attribute to a dict-like
      object.  Alternatively, if a subclass of :class:`Pickler` has a
      :attr:`dispatch_table` attribute then this will be used as the
      default dispatch table for instances of that class.

      See :ref:`pickle-dispatch` for usage examples.

      .. versionadded:: 3.3

339 340
   .. attribute:: fast

341 342 343 344 345
      Deprecated. Enable fast mode if set to a true value.  The fast mode
      disables the usage of memo, therefore speeding the pickling process by not
      generating superfluous PUT opcodes.  It should not be used with
      self-referential objects, doing otherwise will cause :class:`Pickler` to
      recurse infinitely.
346 347 348

      Use :func:`pickletools.optimize` if you need more compact pickles.

349

350
.. class:: Unpickler(file, \*, fix_imports=True, encoding="ASCII", errors="strict")
351

352
   This takes a binary file for reading a pickle data stream.
353

354 355
   The protocol version of the pickle is detected automatically, so no
   protocol argument is needed.
356

357 358
   The argument *file* must have two methods, a read() method that takes an
   integer argument, and a readline() method that requires no arguments.  Both
359 360 361
   methods should return bytes.  Thus *file* can be an on-disk file object
   opened for binary reading, a :class:`io.BytesIO` object, or any other
   custom object that meets this interface.
362

363
   Optional keyword arguments are *fix_imports*, *encoding* and *errors*,
Georg Brandl's avatar
Georg Brandl committed
364
   which are used to control compatibility support for pickle stream generated
365 366
   by Python 2.  If *fix_imports* is true, pickle will try to map the old
   Python 2 names to the new names used in Python 3.  The *encoding* and
367
   *errors* tell pickle how to decode 8-bit string instances pickled by Python
368 369
   2; these default to 'ASCII' and 'strict', respectively.  The *encoding* can
   be 'bytes' to read these ß8-bit string instances as bytes objects.
370

371
   .. method:: load()
372

373 374
      Read a pickled object representation from the open file object given in
      the constructor, and return the reconstituted object hierarchy specified
375
      therein.  Bytes past the pickled object's representation are ignored.
376

377
   .. method:: persistent_load(pid)
378

379
      Raise an :exc:`UnpicklingError` by default.
380

381
      If defined, :meth:`persistent_load` should return the object specified by
382
      the persistent ID *pid*.  If an invalid persistent ID is encountered, an
383
      :exc:`UnpicklingError` should be raised.
384

385 386 387 388
      See :ref:`pickle-persistent` for details and examples of uses.

   .. method:: find_class(module, name)

389
      Import *module* if necessary and return the object called *name* from it,
390 391 392
      where the *module* and *name* arguments are :class:`str` objects.  Note,
      unlike its name suggests, :meth:`find_class` is also used for finding
      functions.
393 394

      Subclasses may override this to gain control over what type of objects and
395 396
      how they can be loaded, potentially reducing security risks. Refer to
      :ref:`pickle-restrict` for details.
397

398

399
.. _pickle-picklable:
400 401 402 403 404 405 406 407

What can be pickled and unpickled?
----------------------------------

The following types can be pickled:

* ``None``, ``True``, and ``False``

408
* integers, floating point numbers, complex numbers
409

410
* strings, bytes, bytearrays
411 412 413

* tuples, lists, sets, and dictionaries containing only picklable objects

414 415
* functions defined at the top level of a module (using :keyword:`def`, not
  :keyword:`lambda`)
416 417 418 419 420

* built-in functions defined at the top level of a module

* classes that are defined at the top level of a module

421 422
* instances of such classes whose :attr:`~object.__dict__` or the result of
  calling :meth:`__getstate__` is picklable  (see section :ref:`pickle-inst` for
423
  details).
424 425 426

Attempts to pickle unpicklable objects will raise the :exc:`PicklingError`
exception; when this happens, an unspecified number of bytes may have already
427
been written to the underlying file.  Trying to pickle a highly recursive data
428
structure may exceed the maximum recursion depth, a :exc:`RuntimeError` will be
429
raised in this case.  You can carefully raise this limit with
430 431 432
:func:`sys.setrecursionlimit`.

Note that functions (built-in and user-defined) are pickled by "fully qualified"
433
name reference, not by value. [#]_  This means that only the function name is
434 435
pickled, along with the name of the module the function is defined in.  Neither
the function's code, nor any of its function attributes are pickled.  Thus the
436 437 438 439 440 441 442 443 444
defining module must be importable in the unpickling environment, and the module
must contain the named object, otherwise an exception will be raised. [#]_

Similarly, classes are pickled by named reference, so the same restrictions in
the unpickling environment apply.  Note that none of the class's code or data is
pickled, so in the following example the class attribute ``attr`` is not
restored in the unpickling environment::

   class Foo:
445
       attr = 'A class attribute'
446 447 448 449 450 451 452 453 454 455 456 457 458 459 460

   picklestring = pickle.dumps(Foo)

These restrictions are why picklable functions and classes must be defined in
the top level of a module.

Similarly, when class instances are pickled, their class's code and data are not
pickled along with them.  Only the instance data are pickled.  This is done on
purpose, so you can fix bugs in a class or add methods to the class and still
load objects that were created with an earlier version of the class.  If you
plan to have long-lived objects that will see many versions of a class, it may
be worthwhile to put a version number in the objects so that suitable
conversions can be made by the class's :meth:`__setstate__` method.


461
.. _pickle-inst:
462

463 464
Pickling Class Instances
------------------------
465

466 467
.. currentmodule:: None

468 469
In this section, we describe the general mechanisms available to you to define,
customize, and control how class instances are pickled and unpickled.
470

471 472 473 474 475 476
In most cases, no additional code is needed to make instances picklable.  By
default, pickle will retrieve the class and the attributes of an instance via
introspection. When a class instance is unpickled, its :meth:`__init__` method
is usually *not* invoked.  The default behaviour first creates an uninitialized
instance and then restores the saved attributes.  The following code shows an
implementation of this behaviour::
477

478 479
   def save(obj):
       return (obj.__class__, obj.__dict__)
480

481 482 483 484
   def load(cls, attributes):
       obj = cls.__new__(cls)
       obj.__dict__.update(attributes)
       return obj
485

Georg Brandl's avatar
Georg Brandl committed
486
Classes can alter the default behaviour by providing one or several special
487
methods:
Georg Brandl's avatar
Georg Brandl committed
488

489 490 491 492 493 494 495 496 497 498 499 500 501 502 503
.. method:: object.__getnewargs_ex__()

   In protocols 4 and newer, classes that implements the
   :meth:`__getnewargs_ex__` method can dictate the values passed to the
   :meth:`__new__` method upon unpickling.  The method must return a pair
   ``(args, kwargs)`` where *args* is a tuple of positional arguments
   and *kwargs* a dictionary of named arguments for constructing the
   object.  Those will be passed to the :meth:`__new__` method upon
   unpickling.

   You should implement this method if the :meth:`__new__` method of your
   class requires keyword-only arguments.  Otherwise, it is recommended for
   compatibility to implement :meth:`__getnewargs__`.


504
.. method:: object.__getnewargs__()
505

506
   This method serve a similar purpose as :meth:`__getnewargs_ex__` but
507
   for protocols 2 and newer.  It must return a tuple of arguments ``args``
508 509 510 511
   which will be passed to the :meth:`__new__` method upon unpickling.

   In protocols 4 and newer, :meth:`__getnewargs__` will not be called if
   :meth:`__getnewargs_ex__` is defined.
512 513


514
.. method:: object.__getstate__()
515

516 517 518 519
   Classes can further influence how their instances are pickled; if the class
   defines the method :meth:`__getstate__`, it is called and the returned object
   is pickled as the contents for the instance, instead of the contents of the
   instance's dictionary.  If the :meth:`__getstate__` method is absent, the
520
   instance's :attr:`~object.__dict__` is pickled as usual.
521 522 523 524 525 526 527 528 529 530 531 532 533


.. method:: object.__setstate__(state)

   Upon unpickling, if the class defines :meth:`__setstate__`, it is called with
   the unpickled state.  In that case, there is no requirement for the state
   object to be a dictionary.  Otherwise, the pickled state must be a dictionary
   and its items are assigned to the new instance's dictionary.

   .. note::

      If :meth:`__getstate__` returns a false value, the :meth:`__setstate__`
      method will not be called upon unpickling.
534 535


536 537
Refer to the section :ref:`pickle-state` for more information about how to use
the methods :meth:`__getstate__` and :meth:`__setstate__`.
538

Benjamin Peterson's avatar
Benjamin Peterson committed
539
.. note::
540

Benjamin Peterson's avatar
Benjamin Peterson committed
541 542
   At unpickling time, some methods like :meth:`__getattr__`,
   :meth:`__getattribute__`, or :meth:`__setattr__` may be called upon the
543 544 545 546
   instance.  In case those methods rely on some internal invariant being
   true, the type should implement :meth:`__getnewargs__` or
   :meth:`__getnewargs_ex__` to establish such an invariant; otherwise,
   neither :meth:`__new__` nor :meth:`__init__` will be called.
Benjamin Peterson's avatar
Benjamin Peterson committed
547

548
.. index:: pair: copy; protocol
549

550 551 552 553
As we shall see, pickle does not use directly the methods described above.  In
fact, these methods are part of the copy protocol which implements the
:meth:`__reduce__` special method.  The copy protocol provides a unified
interface for retrieving the data necessary for pickling and copying
554
objects. [#]_
555

556 557
Although powerful, implementing :meth:`__reduce__` directly in your classes is
error prone.  For this reason, class designers should use the high-level
558
interface (i.e., :meth:`__getnewargs_ex__`, :meth:`__getstate__` and
559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588
:meth:`__setstate__`) whenever possible.  We will show, however, cases where
using :meth:`__reduce__` is the only option or leads to more efficient pickling
or both.

.. method:: object.__reduce__()

   The interface is currently defined as follows.  The :meth:`__reduce__` method
   takes no argument and shall return either a string or preferably a tuple (the
   returned object is often referred to as the "reduce value").

   If a string is returned, the string should be interpreted as the name of a
   global variable.  It should be the object's local name relative to its
   module; the pickle module searches the module namespace to determine the
   object's module.  This behaviour is typically useful for singletons.

   When a tuple is returned, it must be between two and five items long.
   Optional items can either be omitted, or ``None`` can be provided as their
   value.  The semantics of each item are in order:

   .. XXX Mention __newobj__ special-case?

   * A callable object that will be called to create the initial version of the
     object.

   * A tuple of arguments for the callable object.  An empty tuple must be given
     if the callable does not accept any argument.

   * Optionally, the object's state, which will be passed to the object's
     :meth:`__setstate__` method as previously described.  If the object has no
     such method then, the value must be a dictionary and it will be added to
589
     the object's :attr:`~object.__dict__` attribute.
590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613

   * Optionally, an iterator (and not a sequence) yielding successive items.
     These items will be appended to the object either using
     ``obj.append(item)`` or, in batch, using ``obj.extend(list_of_items)``.
     This is primarily used for list subclasses, but may be used by other
     classes as long as they have :meth:`append` and :meth:`extend` methods with
     the appropriate signature.  (Whether :meth:`append` or :meth:`extend` is
     used depends on which pickle protocol version is used as well as the number
     of items to append, so both must be supported.)

   * Optionally, an iterator (not a sequence) yielding successive key-value
     pairs.  These items will be stored to the object using ``obj[key] =
     value``.  This is primarily used for dictionary subclasses, but may be used
     by other classes as long as they implement :meth:`__setitem__`.


.. method:: object.__reduce_ex__(protocol)

   Alternatively, a :meth:`__reduce_ex__` method may be defined.  The only
   difference is this method should take a single integer argument, the protocol
   version.  When defined, pickle will prefer it over the :meth:`__reduce__`
   method.  In addition, :meth:`__reduce__` automatically becomes a synonym for
   the extended version.  The main use for this method is to provide
   backwards-compatible reduce values for older Python releases.
614

615 616
.. currentmodule:: pickle

617 618
.. _pickle-persistent:

619 620
Persistence of External Objects
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
621

622 623 624 625
.. index::
   single: persistent_id (pickle protocol)
   single: persistent_load (pickle protocol)

626 627
For the benefit of object persistence, the :mod:`pickle` module supports the
notion of a reference to an object outside the pickled data stream.  Such
628 629 630
objects are referenced by a persistent ID, which should be either a string of
alphanumeric characters (for protocol 0) [#]_ or just an arbitrary object (for
any newer protocol).
631

632 633
The resolution of such persistent IDs is not defined by the :mod:`pickle`
module; it will delegate this resolution to the user defined methods on the
634 635
pickler and unpickler, :meth:`~Pickler.persistent_id` and
:meth:`~Unpickler.persistent_load` respectively.
636 637

To pickle objects that have an external persistent id, the pickler must have a
638 639 640 641 642
custom :meth:`~Pickler.persistent_id` method that takes an object as an
argument and returns either ``None`` or the persistent id for that object.
When ``None`` is returned, the pickler simply pickles the object as normal.
When a persistent ID string is returned, the pickler will pickle that object,
along with a marker so that the unpickler will recognize it as a persistent ID.
643 644

To unpickle external objects, the unpickler must have a custom
645 646
:meth:`~Unpickler.persistent_load` method that takes a persistent ID object and
returns the referenced object.
647

648 649
Here is a comprehensive example presenting how persistent ID can be used to
pickle external objects by reference.
650

651
.. literalinclude:: ../includes/dbpickle.py
652

653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690
.. _pickle-dispatch:

Dispatch Tables
^^^^^^^^^^^^^^^

If one wants to customize pickling of some classes without disturbing
any other code which depends on pickling, then one can create a
pickler with a private dispatch table.

The global dispatch table managed by the :mod:`copyreg` module is
available as :data:`copyreg.dispatch_table`.  Therefore, one may
choose to use a modified copy of :data:`copyreg.dispatch_table` as a
private dispatch table.

For example ::

   f = io.BytesIO()
   p = pickle.Pickler(f)
   p.dispatch_table = copyreg.dispatch_table.copy()
   p.dispatch_table[SomeClass] = reduce_SomeClass

creates an instance of :class:`pickle.Pickler` with a private dispatch
table which handles the ``SomeClass`` class specially.  Alternatively,
the code ::

   class MyPickler(pickle.Pickler):
       dispatch_table = copyreg.dispatch_table.copy()
       dispatch_table[SomeClass] = reduce_SomeClass
   f = io.BytesIO()
   p = MyPickler(f)

does the same, but all instances of ``MyPickler`` will by default
share the same dispatch table.  The equivalent code using the
:mod:`copyreg` module is ::

   copyreg.pickle(SomeClass, reduce_SomeClass)
   f = io.BytesIO()
   p = pickle.Pickler(f)
691

692 693 694 695 696 697 698 699 700 701 702
.. _pickle-state:

Handling Stateful Objects
^^^^^^^^^^^^^^^^^^^^^^^^^

.. index::
   single: __getstate__() (copy protocol)
   single: __setstate__() (copy protocol)

Here's an example that shows how to modify pickling behavior for a class.
The :class:`TextReader` class opens a text file, and returns the line number and
703
line contents each time its :meth:`!readline` method is called. If a
704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721
:class:`TextReader` instance is pickled, all attributes *except* the file object
member are saved. When the instance is unpickled, the file is reopened, and
reading resumes from the last location. The :meth:`__setstate__` and
:meth:`__getstate__` methods are used to implement this behavior. ::

   class TextReader:
       """Print and number lines in a text file."""

       def __init__(self, filename):
           self.filename = filename
           self.file = open(filename)
           self.lineno = 0

       def readline(self):
           self.lineno += 1
           line = self.file.readline()
           if not line:
               return None
722
           if line.endswith('\n'):
723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758
               line = line[:-1]
           return "%i: %s" % (self.lineno, line)

       def __getstate__(self):
           # Copy the object's state from self.__dict__ which contains
           # all our instance attributes. Always use the dict.copy()
           # method to avoid modifying the original state.
           state = self.__dict__.copy()
           # Remove the unpicklable entries.
           del state['file']
           return state

       def __setstate__(self, state):
           # Restore instance attributes (i.e., filename and lineno).
           self.__dict__.update(state)
           # Restore the previously opened file's state. To do so, we need to
           # reopen it and read from it until the line count is restored.
           file = open(self.filename)
           for _ in range(self.lineno):
               file.readline()
           # Finally, save the file.
           self.file = file


A sample usage might be something like this::

   >>> reader = TextReader("hello.txt")
   >>> reader.readline()
   '1: Hello world!'
   >>> reader.readline()
   '2: I am line number two.'
   >>> new_reader = pickle.loads(pickle.dumps(reader))
   >>> new_reader.readline()
   '3: Goodbye!'


759
.. _pickle-restrict:
760

761
Restricting Globals
762
-------------------
763

764
.. index::
765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781
   single: find_class() (pickle protocol)

By default, unpickling will import any class or function that it finds in the
pickle data.  For many applications, this behaviour is unacceptable as it
permits the unpickler to import and invoke arbitrary code.  Just consider what
this hand-crafted pickle data stream does when loaded::

    >>> import pickle
    >>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
    hello world
    0

In this example, the unpickler imports the :func:`os.system` function and then
apply the string argument "echo hello world".  Although this example is
inoffensive, it is not difficult to imagine one that could damage your system.

For this reason, you may want to control what gets unpickled by customizing
782 783 784 785
:meth:`Unpickler.find_class`.  Unlike its name suggests,
:meth:`Unpickler.find_class` is called whenever a global (i.e., a class or
a function) is requested.  Thus it is possible to either completely forbid
globals or restrict them to a safe subset.
786 787 788 789 790 791 792

Here is an example of an unpickler allowing only few safe classes from the
:mod:`builtins` module to be loaded::

   import builtins
   import io
   import pickle
793

794 795 796 797 798 799 800 801 802
   safe_builtins = {
       'range',
       'complex',
       'set',
       'frozenset',
       'slice',
   }

   class RestrictedUnpickler(pickle.Unpickler):
803

804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830
       def find_class(self, module, name):
           # Only allow safe classes from builtins.
           if module == "builtins" and name in safe_builtins:
               return getattr(builtins, name)
           # Forbid everything else.
           raise pickle.UnpicklingError("global '%s.%s' is forbidden" %
                                        (module, name))

   def restricted_loads(s):
       """Helper function analogous to pickle.loads()."""
       return RestrictedUnpickler(io.BytesIO(s)).load()

A sample usage of our unpickler working has intended::

    >>> restricted_loads(pickle.dumps([1, 2, range(15)]))
    [1, 2, range(0, 15)]
    >>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
    Traceback (most recent call last):
      ...
    pickle.UnpicklingError: global 'os.system' is forbidden
    >>> restricted_loads(b'cbuiltins\neval\n'
    ...                  b'(S\'getattr(__import__("os"), "system")'
    ...                  b'("echo hello world")\'\ntR.')
    Traceback (most recent call last):
      ...
    pickle.UnpicklingError: global 'builtins.eval' is forbidden

831 832

.. XXX Add note about how extension codes could evade our protection
833
   mechanism (e.g. cached classes do not invokes find_class()).
834 835 836

As our examples shows, you have to be careful with what you allow to be
unpickled.  Therefore if security is a concern, you may want to consider
837 838
alternatives such as the marshalling API in :mod:`xmlrpc.client` or
third-party solutions.
839

840

841 842 843 844 845 846 847 848
Performance
-----------

Recent versions of the pickle protocol (from protocol 2 and upwards) feature
efficient binary encodings for several common features and built-in types.
Also, the :mod:`pickle` module has a transparent optimizer written in C.


849 850
.. _pickle-example:

851 852
Examples
--------
853

854
For the simplest code, use the :func:`dump` and :func:`load` functions. ::
855 856 857

   import pickle

858 859
   # An arbitrary collection of objects supported by pickle.
   data = {
860 861 862
       'a': [1, 2.0, 3, 4+6j],
       'b': ("character string", b"byte string"),
       'c': set([None, True, False])
863
   }
864

865 866 867
   with open('data.pickle', 'wb') as f:
       # Pickle the 'data' dictionary using the highest protocol available.
       pickle.dump(data, f, pickle.HIGHEST_PROTOCOL)
868 869


870
The following example reads the resulting pickled data. ::
871

872
   import pickle
873

874 875 876 877
   with open('data.pickle', 'rb') as f:
       # The protocol version used is detected automatically, so we do not
       # have to specify it.
       data = pickle.load(f)
878 879


880 881 882 883
.. XXX: Add examples showing how to optimize pickles for size (like using
.. pickletools.optimize() or the gzip module).


884 885
.. seealso::

886
   Module :mod:`copyreg`
887 888
      Pickle interface constructor registration for extension types.

889 890 891
   Module :mod:`pickletools`
      Tools for working with and analyzing pickled data.

892 893 894 895 896 897 898 899 900 901 902 903 904 905
   Module :mod:`shelve`
      Indexed databases of objects; uses :mod:`pickle`.

   Module :mod:`copy`
      Shallow and deep object copying.

   Module :mod:`marshal`
      High-performance serialization of built-in types.


.. rubric:: Footnotes

.. [#] Don't confuse this with the :mod:`marshal` module

906 907 908
.. [#] This is why :keyword:`lambda` functions cannot be pickled:  all
    :keyword:`lambda` functions share the same name:  ``<lambda>``.

909 910 911
.. [#] The exception raised will likely be an :exc:`ImportError` or an
   :exc:`AttributeError` but it could be something else.

912 913
.. [#] The :mod:`copy` module uses this protocol for shallow and deep copying
   operations.
914

915 916 917
.. [#] The limitation on alphanumeric characters is due to the fact
   the persistent IDs, in protocol 0, are delimited by the newline
   character.  Therefore if any kind of newline characters occurs in
918
   persistent IDs, the resulting pickle will become unreadable.