index.rst 48 KB
Newer Older
1 2 3 4 5
.. highlightlang:: none

.. _install-index:

*****************************
6
  Installing Python Modules
7 8 9 10 11 12
*****************************

:Author: Greg Ward
:Release: |version|
:Date: |today|

13 14 15 16 17 18 19 20
.. TODO: Fill in XXX comments

.. The audience for this document includes people who don't know anything
   about Python and aren't about to learn the language just in order to
   install and maintain it for their users, i.e. system administrators.
   Thus, I have to be sure to explain the basics at some point:
   sys.path and PYTHONPATH at least.  Should probably give pointers to
   other docs on "import site", PYTHONSTARTUP, PYTHONHOME, etc.
21

22 23
   Finally, it might be useful to include all the material from my "Care
   and Feeding of a Python Installation" talk in here somewhere.  Yow!
24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74

.. topic:: Abstract

   This document describes the Python Distribution Utilities ("Distutils") from the
   end-user's point-of-view, describing how to extend the capabilities of a
   standard Python installation by building and installing third-party Python
   modules and extensions.


.. _inst-intro:

Introduction
============

Although Python's extensive standard library covers many programming needs,
there often comes a time when you need to add some new functionality to your
Python installation in the form of third-party modules.  This might be necessary
to support your own programming, or to support an application that you want to
use and that happens to be written in Python.

In the past, there has been little support for adding third-party modules to an
existing Python installation.  With the introduction of the Python Distribution
Utilities (Distutils for short) in Python 2.0, this changed.

This document is aimed primarily at the people who need to install third-party
Python modules: end-users and system administrators who just need to get some
Python application running, and existing Python programmers who want to add some
new goodies to their toolbox.  You don't need to know Python to read this
document; there will be some brief forays into using Python's interactive mode
to explore your installation, but that's it.  If you're looking for information
on how to distribute your own Python modules so that others may use them, see
the :ref:`distutils-index` manual.


.. _inst-trivial-install:

Best case: trivial installation
-------------------------------

In the best case, someone will have prepared a special version of the module
distribution you want to install that is targeted specifically at your platform
and is installed just like any other software on your platform.  For example,
the module developer might make an executable installer available for Windows
users, an RPM package for users of RPM-based Linux systems (Red Hat, SuSE,
Mandrake, and many others), a Debian package for users of Debian-based Linux
systems, and so forth.

In that case, you would download the installer appropriate to your platform and
do the obvious thing with it: run it if it's an executable installer, ``rpm
--install`` it if it's an RPM, etc.  You don't need to run Python or a setup
script, you don't need to compile anything---you might not even need to read any
75
instructions (although it's always a good idea to do so anyway).
76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98

Of course, things will not always be that easy.  You might be interested in a
module distribution that doesn't have an easy-to-use installer for your
platform.  In that case, you'll have to start with the source distribution
released by the module's author/maintainer.  Installing from a source
distribution is not too hard, as long as the modules are packaged in the
standard way.  The bulk of this document is about building and installing
modules from standard source distributions.


.. _inst-new-standard:

The new standard: Distutils
---------------------------

If you download a module source distribution, you can tell pretty quickly if it
was packaged and distributed in the standard way, i.e. using the Distutils.
First, the distribution's name and version number will be featured prominently
in the name of the downloaded archive, e.g. :file:`foo-1.0.tar.gz` or
:file:`widget-0.9.7.zip`.  Next, the archive will unpack into a similarly-named
directory: :file:`foo-1.0` or :file:`widget-0.9.7`.  Additionally, the
distribution will contain a setup script :file:`setup.py`, and a file named
:file:`README.txt` or possibly just :file:`README`, which should explain that
Éric Araujo's avatar
Éric Araujo committed
99 100
building and installing the module distribution is a simple matter of running
one command from a terminal::
101 102 103

   python setup.py install

104
For Windows, this command should be run from a command prompt window
Éric Araujo's avatar
Éric Araujo committed
105
(:menuselection:`Start --> Accessories`)::
Éric Araujo's avatar
Éric Araujo committed
106 107 108

   setup.py install

109 110 111 112 113 114 115 116 117 118 119 120 121
If all these things are true, then you already know how to build and install the
modules you've just downloaded:  Run the command above. Unless you need to
install things in a non-standard way or customize the build process, you don't
really need this manual.  Or rather, the above command is everything you need to
get out of this manual.


.. _inst-standard-install:

Standard Build and Install
==========================

As described in section :ref:`inst-new-standard`, building and installing a module
Éric Araujo's avatar
Éric Araujo committed
122 123
distribution using the Distutils is usually one simple command to run from a
terminal::
124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146

   python setup.py install


.. _inst-platform-variations:

Platform variations
-------------------

You should always run the setup command from the distribution root directory,
i.e. the top-level subdirectory that the module source distribution unpacks
into.  For example, if you've just downloaded a module source distribution
:file:`foo-1.0.tar.gz` onto a Unix system, the normal thing to do is::

   gunzip -c foo-1.0.tar.gz | tar xf -    # unpacks into directory foo-1.0
   cd foo-1.0
   python setup.py install

On Windows, you'd probably download :file:`foo-1.0.zip`.  If you downloaded the
archive file to :file:`C:\\Temp`, then it would unpack into
:file:`C:\\Temp\\foo-1.0`; you can use either a archive manipulator with a
graphical user interface (such as WinZip) or a command-line tool (such as
:program:`unzip` or :program:`pkunzip`) to unpack the archive.  Then, open a
Éric Araujo's avatar
Éric Araujo committed
147
command prompt window and run::
148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246

   cd c:\Temp\foo-1.0
   python setup.py install


.. _inst-splitting-up:

Splitting the job up
--------------------

Running ``setup.py install`` builds and installs all modules in one run.  If you
prefer to work incrementally---especially useful if you want to customize the
build process, or if things are going wrong---you can use the setup script to do
one thing at a time.  This is particularly helpful when the build and install
will be done by different users---for example, you might want to build a module
distribution and hand it off to a system administrator for installation (or do
it yourself, with super-user privileges).

For example, you can build everything in one step, and then install everything
in a second step, by invoking the setup script twice::

   python setup.py build
   python setup.py install

If you do this, you will notice that running the :command:`install` command
first runs the :command:`build` command, which---in this case---quickly notices
that it has nothing to do, since everything in the :file:`build` directory is
up-to-date.

You may not need this ability to break things down often if all you do is
install modules downloaded off the 'net, but it's very handy for more advanced
tasks.  If you get into distributing your own Python modules and extensions,
you'll run lots of individual Distutils commands on their own.


.. _inst-how-build-works:

How building works
------------------

As implied above, the :command:`build` command is responsible for putting the
files to install into a *build directory*.  By default, this is :file:`build`
under the distribution root; if you're excessively concerned with speed, or want
to keep the source tree pristine, you can change the build directory with the
:option:`--build-base` option. For example::

   python setup.py build --build-base=/tmp/pybuild/foo-1.0

(Or you could do this permanently with a directive in your system or personal
Distutils configuration file; see section :ref:`inst-config-files`.)  Normally, this
isn't necessary.

The default layout for the build tree is as follows::

   --- build/ --- lib/
   or
   --- build/ --- lib.<plat>/
                  temp.<plat>/

where ``<plat>`` expands to a brief description of the current OS/hardware
platform and Python version.  The first form, with just a :file:`lib` directory,
is used for "pure module distributions"---that is, module distributions that
include only pure Python modules.  If a module distribution contains any
extensions (modules written in C/C++), then the second form, with two ``<plat>``
directories, is used.  In that case, the :file:`temp.{plat}` directory holds
temporary files generated by the compile/link process that don't actually get
installed.  In either case, the :file:`lib` (or :file:`lib.{plat}`) directory
contains all Python modules (pure Python and extensions) that will be installed.

In the future, more directories will be added to handle Python scripts,
documentation, binary executables, and whatever else is needed to handle the job
of installing Python modules and applications.


.. _inst-how-install-works:

How installation works
----------------------

After the :command:`build` command runs (whether you run it explicitly, or the
:command:`install` command does it for you), the work of the :command:`install`
command is relatively simple: all it has to do is copy everything under
:file:`build/lib` (or :file:`build/lib.{plat}`) to your chosen installation
directory.

If you don't choose an installation directory---i.e., if you just run ``setup.py
install``\ ---then the :command:`install` command installs to the standard
location for third-party Python modules.  This location varies by platform and
by how you built/installed Python itself.  On Unix (and Mac OS X, which is also
Unix-based), it also depends on whether the module distribution being installed
is pure Python or contains extensions ("non-pure"):

+-----------------+-----------------------------------------------------+--------------------------------------------------+-------+
| Platform        | Standard installation location                      | Default value                                    | Notes |
+=================+=====================================================+==================================================+=======+
| Unix (pure)     | :file:`{prefix}/lib/python{X.Y}/site-packages`      | :file:`/usr/local/lib/python{X.Y}/site-packages` | \(1)  |
+-----------------+-----------------------------------------------------+--------------------------------------------------+-------+
| Unix (non-pure) | :file:`{exec-prefix}/lib/python{X.Y}/site-packages` | :file:`/usr/local/lib/python{X.Y}/site-packages` | \(1)  |
+-----------------+-----------------------------------------------------+--------------------------------------------------+-------+
247
| Windows         | :file:`{prefix}\\Lib\\site-packages`                | :file:`C:\\Python{XY}\\Lib\\site-packages`       | \(2)  |
248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273
+-----------------+-----------------------------------------------------+--------------------------------------------------+-------+

Notes:

(1)
   Most Linux distributions include Python as a standard part of the system, so
   :file:`{prefix}` and :file:`{exec-prefix}` are usually both :file:`/usr` on
   Linux.  If you build Python yourself on Linux (or any Unix-like system), the
   default :file:`{prefix}` and :file:`{exec-prefix}` are :file:`/usr/local`.

(2)
   The default installation directory on Windows was :file:`C:\\Program
   Files\\Python` under Python 1.6a1, 1.5.2, and earlier.

:file:`{prefix}` and :file:`{exec-prefix}` stand for the directories that Python
is installed to, and where it finds its libraries at run-time.  They are always
the same under Windows, and very often the same under Unix and Mac OS X.  You
can find out what your Python installation uses for :file:`{prefix}` and
:file:`{exec-prefix}` by running Python in interactive mode and typing a few
simple commands. Under Unix, just type ``python`` at the shell prompt.  Under
Windows, choose :menuselection:`Start --> Programs --> Python X.Y -->
Python (command line)`.   Once the interpreter is started, you type Python code
at the prompt.  For example, on my Linux system, I type the three Python
statements shown below, and get the output as shown, to find out my
:file:`{prefix}` and :file:`{exec-prefix}`::

274
   Python 2.4 (#26, Aug  7 2004, 17:19:02)
275 276 277 278 279 280 281
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import sys
   >>> sys.prefix
   '/usr'
   >>> sys.exec_prefix
   '/usr'

282 283 284 285 286 287 288 289
A few other placeholders are used in this document: :file:`{X.Y}` stands for the
version of Python, for example ``3.2``; :file:`{abiflags}` will be replaced by
the value of :data:`sys.abiflags` or the empty string for platforms which don't
define ABI flags; :file:`{distname}` will be replaced by the name of the module
distribution being installed.  Dots and capitalization are important in the
paths; for example, a value that uses ``python3.2`` on UNIX will typically use
``Python32`` on Windows.

290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317
If you don't want to install modules to the standard location, or if you don't
have permission to write there, then you need to read about alternate
installations in section :ref:`inst-alt-install`.  If you want to customize your
installation directories more heavily, see section :ref:`inst-custom-install` on
custom installations.


.. _inst-alt-install:

Alternate Installation
======================

Often, it is necessary or desirable to install modules to a location other than
the standard location for third-party Python modules.  For example, on a Unix
system you might not have permission to write to the standard third-party module
directory.  Or you might wish to try out a module before making it a standard
part of your local Python installation.  This is especially true when upgrading
a distribution already present: you want to make sure your existing base of
scripts still works with the new version before actually upgrading.

The Distutils :command:`install` command is designed to make installing module
distributions to an alternate location simple and painless.  The basic idea is
that you supply a base directory for the installation, and the
:command:`install` command picks a set of directories (called an *installation
scheme*) under this base directory in which to install files.  The details
differ across platforms, so read whichever of the following sections applies to
you.

318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369
Note that the various alternate installation schemes are mutually exclusive: you
can pass ``--user``, or ``--home``, or ``--prefix`` and ``--exec-prefix``, or
``--install-base`` and ``--install-platbase``, but you can't mix from these
groups.


.. _inst-alt-install-user:

Alternate installation: the user scheme
---------------------------------------

This scheme is designed to be the most convenient solution for users that don't
have write permission to the global site-packages directory or don't want to
install into it.  It is enabled with a simple option::

   python setup.py install --user

Files will be installed into subdirectories of :data:`site.USER_BASE` (written
as :file:`{userbase}` hereafter).  This scheme installs pure Python modules and
extension modules in the same location (also known as :data:`site.USER_SITE`).
Here are the values for UNIX, including Mac OS X:

=============== ===========================================================
Type of file    Installation directory
=============== ===========================================================
modules         :file:`{userbase}/lib/python{X.Y}/site-packages`
scripts         :file:`{userbase}/bin`
data            :file:`{userbase}`
C headers       :file:`{userbase}/include/python{X.Y}{abiflags}/{distname}`
=============== ===========================================================

And here are the values used on Windows:

=============== ===========================================================
Type of file    Installation directory
=============== ===========================================================
modules         :file:`{userbase}\\Python{XY}\\site-packages`
scripts         :file:`{userbase}\\Scripts`
data            :file:`{userbase}`
C headers       :file:`{userbase}\\Python{XY}\\Include\\{distname}`
=============== ===========================================================

The advantage of using this scheme compared to the other ones described below is
that the user site-packages directory is under normal conditions always included
in :data:`sys.path` (see :mod:`site` for more information), which means that
there is no additional step to perform after running the :file:`setup.py` script
to finalize the installation.

The :command:`build_ext` command also has a ``--user`` option to add
:file:`{userbase}/include` to the compiler search path for header files and
:file:`{userbase}/lib` to the compiler search path for libraries as well as to
the runtime search path for shared C libraries (rpath).
370

371 372

.. _inst-alt-install-home:
373 374 375 376 377 378 379 380

Alternate installation: the home scheme
---------------------------------------

The idea behind the "home scheme" is that you build and maintain a personal
stash of Python modules.  This scheme's name is derived from the idea of a
"home" directory on Unix, since it's not unusual for a Unix user to make their
home directory have a layout similar to :file:`/usr/` or :file:`/usr/local/`.
Georg Brandl's avatar
Georg Brandl committed
381 382
This scheme can be used by anyone, regardless of the operating system they
are installing for.
383 384 385 386 387 388 389 390 391 392 393

Installing a new module distribution is as simple as ::

   python setup.py install --home=<dir>

where you can supply any directory you like for the :option:`--home` option.  On
Unix, lazy typists can just type a tilde (``~``); the :command:`install` command
will expand this to your home directory::

   python setup.py install --home=~

394 395 396 397 398
To make Python find the distributions installed with this scheme, you may have
to :ref:`modify Python's search path <inst-search-path>` or edit
:mod:`sitecustomize` (see :mod:`site`) to call :func:`site.addsitedir` or edit
:data:`sys.path`.

399 400 401
The :option:`--home` option defines the installation base directory.  Files are
installed to the following directories under the installation base as follows:

402 403 404 405 406 407 408 409
=============== ===========================================================
Type of file    Installation directory
=============== ===========================================================
modules         :file:`{home}/lib/python`
scripts         :file:`{home}/bin`
data            :file:`{home}`
C headers       :file:`{home}/include/python/{distname}`
=============== ===========================================================
410

411
(Mentally replace slashes with backslashes if you're on Windows.)
412

413 414

.. _inst-alt-install-prefix-unix:
415 416 417 418 419 420 421 422

Alternate installation: Unix (the prefix scheme)
------------------------------------------------

The "prefix scheme" is useful when you wish to use one Python installation to
perform the build/install (i.e., to run the setup script), but install modules
into the third-party module directory of a different Python installation (or
something that looks like a different Python installation).  If this sounds a
423
trifle unusual, it is---that's why the user and home schemes come before.  However,
424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450
there are at least two known cases where the prefix scheme will be useful.

First, consider that many Linux distributions put Python in :file:`/usr`, rather
than the more traditional :file:`/usr/local`.  This is entirely appropriate,
since in those cases Python is part of "the system" rather than a local add-on.
However, if you are installing Python modules from source, you probably want
them to go in :file:`/usr/local/lib/python2.{X}` rather than
:file:`/usr/lib/python2.{X}`.  This can be done with ::

   /usr/bin/python setup.py install --prefix=/usr/local

Another possibility is a network filesystem where the name used to write to a
remote directory is different from the name used to read it: for example, the
Python interpreter accessed as :file:`/usr/local/bin/python` might search for
modules in :file:`/usr/local/lib/python2.{X}`, but those modules would have to
be installed to, say, :file:`/mnt/{@server}/export/lib/python2.{X}`.  This could
be done with ::

   /usr/local/bin/python setup.py install --prefix=/mnt/@server/export

In either case, the :option:`--prefix` option defines the installation base, and
the :option:`--exec-prefix` option defines the platform-specific installation
base, which is used for platform-specific files.  (Currently, this just means
non-pure module distributions, but could be expanded to C libraries, binary
executables, etc.)  If :option:`--exec-prefix` is not supplied, it defaults to
:option:`--prefix`.  Files are installed as follows:

451 452 453 454 455 456 457 458 459
================= ==========================================================
Type of file      Installation directory
================= ==========================================================
Python modules    :file:`{prefix}/lib/python{X.Y}/site-packages`
extension modules :file:`{exec-prefix}/lib/python{X.Y}/site-packages`
scripts           :file:`{prefix}/bin`
data              :file:`{prefix}`
C headers         :file:`{prefix}/include/python{X.Y}{abiflags}/{distname}`
================= ==========================================================
460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483

There is no requirement that :option:`--prefix` or :option:`--exec-prefix`
actually point to an alternate Python installation; if the directories listed
above do not already exist, they are created at installation time.

Incidentally, the real reason the prefix scheme is important is simply that a
standard Unix installation uses the prefix scheme, but with :option:`--prefix`
and :option:`--exec-prefix` supplied by Python itself as ``sys.prefix`` and
``sys.exec_prefix``.  Thus, you might think you'll never use the prefix scheme,
but every time you run ``python setup.py install`` without any other options,
you're using it.

Note that installing extensions to an alternate Python installation has no
effect on how those extensions are built: in particular, the Python header files
(:file:`Python.h` and friends) installed with the Python interpreter used to run
the setup script will be used in compiling extensions.  It is your
responsibility to ensure that the interpreter used to run extensions installed
in this way is compatible with the interpreter used to build them.  The best way
to do this is to ensure that the two interpreters are the same version of Python
(possibly different builds, or possibly copies of the same build).  (Of course,
if your :option:`--prefix` and :option:`--exec-prefix` don't even point to an
alternate Python installation, this is immaterial.)


484
.. _inst-alt-install-prefix-windows:
485 486 487 488 489 490 491 492 493 494 495 496 497 498

Alternate installation: Windows (the prefix scheme)
---------------------------------------------------

Windows has no concept of a user's home directory, and since the standard Python
installation under Windows is simpler than under Unix, the :option:`--prefix`
option has traditionally been used to install additional packages in separate
locations on Windows. ::

   python setup.py install --prefix="\Temp\Python"

to install modules to the :file:`\\Temp\\Python` directory on the current drive.

The installation base is defined by the :option:`--prefix` option; the
499 500 501 502 503 504 505 506 507 508 509 510
:option:`--exec-prefix` option is not supported under Windows, which means that
pure Python modules and extension modules are installed into the same location.
Files are installed as follows:

=============== ==========================================================
Type of file    Installation directory
=============== ==========================================================
modules         :file:`{prefix}\\Lib\\site-packages`
scripts         :file:`{prefix}\\Scripts`
data            :file:`{prefix}`
C headers       :file:`{prefix}\\Include\\{distname}`
=============== ==========================================================
511 512 513 514 515 516 517 518 519 520 521 522 523


.. _inst-custom-install:

Custom Installation
===================

Sometimes, the alternate installation schemes described in section
:ref:`inst-alt-install` just don't do what you want.  You might want to tweak just
one or two directories while keeping everything under the same base directory,
or you might want to completely redefine the installation scheme.  In either
case, you're creating a *custom installation scheme*.

524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539
To create a custom installation scheme, you start with one of the alternate
schemes and override some of the installation directories used for the various
types of files, using these options:

====================== =======================
Type of file           Override option
====================== =======================
Python modules         ``--install-purelib``
extension modules      ``--install-platlib``
all modules            ``--install-lib``
scripts                ``--install-scripts``
data                   ``--install-data``
C headers              ``--install-headers``
====================== =======================

These override options can be relative, absolute,
540 541 542
or explicitly defined in terms of one of the installation base directories.
(There are two installation base directories, and they are normally the same---
they only differ when you use the Unix "prefix scheme" and supply different
543 544 545 546
``--prefix`` and ``--exec-prefix`` options; using ``--install-lib`` will
override values computed or given for ``--install-purelib`` and
``--install-platlib``, and is recommended for schemes that don't make a
difference between Python and extension modules.)
547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572

For example, say you're installing a module distribution to your home directory
under Unix---but you want scripts to go in :file:`~/scripts` rather than
:file:`~/bin`. As you might expect, you can override this directory with the
:option:`--install-scripts` option; in this case, it makes most sense to supply
a relative path, which will be interpreted relative to the installation base
directory (your home directory, in this case)::

   python setup.py install --home=~ --install-scripts=scripts

Another Unix example: suppose your Python installation was built and installed
with a prefix of :file:`/usr/local/python`, so under a standard  installation
scripts will wind up in :file:`/usr/local/python/bin`.  If you want them in
:file:`/usr/local/bin` instead, you would supply this absolute directory for the
:option:`--install-scripts` option::

   python setup.py install --install-scripts=/usr/local/bin

(This performs an installation using the "prefix scheme," where the prefix is
whatever your Python interpreter was installed with--- :file:`/usr/local/python`
in this case.)

If you maintain Python on Windows, you might want third-party modules to live in
a subdirectory of :file:`{prefix}`, rather than right in :file:`{prefix}`
itself.  This is almost as easy as customizing the script installation directory
---you just have to remember that there are two types of modules to worry about,
573 574
Python and extension modules, which can conveniently be both controlled by one
option::
575

576
   python setup.py install --install-lib=Site
577

578 579 580 581 582
The specified installation directory is relative to :file:`{prefix}`.  Of
course, you also have to ensure that this directory is in Python's module
search path, such as by putting a :file:`.pth` file in a site directory (see
:mod:`site`).  See section :ref:`inst-search-path` to find out how to modify
Python's search path.
583 584 585 586 587 588 589 590 591 592 593 594 595 596

If you want to define an entire installation scheme, you just have to supply all
of the installation directory options.  The recommended way to do this is to
supply relative paths; for example, if you want to maintain all Python
module-related files under :file:`python` in your home directory, and you want a
separate directory for each platform that you use your home directory from, you
might define the following installation scheme::

   python setup.py install --home=~ \
                           --install-purelib=python/lib \
                           --install-platlib=python/lib.$PLAT \
                           --install-scripts=python/scripts
                           --install-data=python/data

597
or, equivalently, ::
598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633

   python setup.py install --home=~/python \
                           --install-purelib=lib \
                           --install-platlib='lib.$PLAT' \
                           --install-scripts=scripts
                           --install-data=data

``$PLAT`` is not (necessarily) an environment variable---it will be expanded by
the Distutils as it parses your command line options, just as it does when
parsing your configuration file(s).

Obviously, specifying the entire installation scheme every time you install a
new module distribution would be very tedious.  Thus, you can put these options
into your Distutils config file (see section :ref:`inst-config-files`)::

   [install]
   install-base=$HOME
   install-purelib=python/lib
   install-platlib=python/lib.$PLAT
   install-scripts=python/scripts
   install-data=python/data

or, equivalently, ::

   [install]
   install-base=$HOME/python
   install-purelib=lib
   install-platlib=lib.$PLAT
   install-scripts=scripts
   install-data=data

Note that these two are *not* equivalent if you supply a different installation
base directory when you run the setup script.  For example, ::

   python setup.py install --install-base=/tmp

634 635
would install pure modules to :file:`/tmp/python/lib` in the first case, and
to :file:`/tmp/lib` in the second case.  (For the second case, you probably
636 637 638 639 640 641 642 643 644 645 646 647
want to supply an installation base of :file:`/tmp/python`.)

You probably noticed the use of ``$HOME`` and ``$PLAT`` in the sample
configuration file input.  These are Distutils configuration variables, which
bear a strong resemblance to environment variables. In fact, you can use
environment variables in config files on platforms that have such a notion but
the Distutils additionally define a few extra variables that may not be in your
environment, such as ``$PLAT``.  (And of course, on systems that don't have
environment variables, such as Mac OS 9, the configuration variables supplied by
the Distutils are the only ones you can use.) See section :ref:`inst-config-files`
for details.

648 649
.. XXX need some Windows examples---when would custom installation schemes be
   needed on those platforms?
650 651


652
.. XXX Move this to Doc/using
653 654 655 656 657 658 659 660 661 662 663 664 665 666 667

.. _inst-search-path:

Modifying Python's Search Path
------------------------------

When the Python interpreter executes an :keyword:`import` statement, it searches
for both Python code and extension modules along a search path.  A default value
for the path is configured into the Python binary when the interpreter is built.
You can determine the path by importing the :mod:`sys` module and printing the
value of ``sys.path``.   ::

   $ python
   Python 2.2 (#11, Oct  3 2002, 13:31:27)
   [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-112)] on linux2
668
   Type "help", "copyright", "credits" or "license" for more information.
669 670
   >>> import sys
   >>> sys.path
671 672
   ['', '/usr/local/lib/python2.3', '/usr/local/lib/python2.3/plat-linux2',
    '/usr/local/lib/python2.3/lib-tk', '/usr/local/lib/python2.3/lib-dynload',
673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694
    '/usr/local/lib/python2.3/site-packages']
   >>>

The null string in ``sys.path`` represents the current working directory.

The expected convention for locally installed packages is to put them in the
:file:`{...}/site-packages/` directory, but you may want to install Python
modules into some arbitrary directory.  For example, your site may have a
convention of keeping all software related to the web server under :file:`/www`.
Add-on Python modules might then belong in :file:`/www/python`, and in order to
import them, this directory must be added to ``sys.path``.  There are several
different ways to add the directory.

The most convenient way is to add a path configuration file to a directory
that's already on Python's path, usually to the :file:`.../site-packages/`
directory.  Path configuration files have an extension of :file:`.pth`, and each
line must contain a single path that will be appended to ``sys.path``.  (Because
the new paths are appended to ``sys.path``, modules in the added directories
will not override standard modules.  This means you can't use this mechanism for
installing fixed versions of standard modules.)

Paths can be absolute or relative, in which case they're relative to the
695
directory containing the :file:`.pth` file.  See the documentation of
696
the :mod:`site` module for more information.
697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774

A slightly less convenient way is to edit the :file:`site.py` file in Python's
standard library, and modify ``sys.path``.  :file:`site.py` is automatically
imported when the Python interpreter is executed, unless the :option:`-S` switch
is supplied to suppress this behaviour.  So you could simply edit
:file:`site.py` and add two lines to it::

   import sys
   sys.path.append('/www/python/')

However, if you reinstall the same major version of Python (perhaps when
upgrading from 2.2 to 2.2.2, for example) :file:`site.py` will be overwritten by
the stock version.  You'd have to remember that it was modified and save a copy
before doing the installation.

There are two environment variables that can modify ``sys.path``.
:envvar:`PYTHONHOME` sets an alternate value for the prefix of the Python
installation.  For example, if :envvar:`PYTHONHOME` is set to ``/www/python``,
the search path will be set to ``['', '/www/python/lib/pythonX.Y/',
'/www/python/lib/pythonX.Y/plat-linux2', ...]``.

The :envvar:`PYTHONPATH` variable can be set to a list of paths that will be
added to the beginning of ``sys.path``.  For example, if :envvar:`PYTHONPATH` is
set to ``/www/python:/opt/py``, the search path will begin with
``['/www/python', '/opt/py']``.  (Note that directories must exist in order to
be added to ``sys.path``; the :mod:`site` module removes paths that don't
exist.)

Finally, ``sys.path`` is just a regular Python list, so any Python application
can modify it by adding or removing entries.


.. _inst-config-files:

Distutils Configuration Files
=============================

As mentioned above, you can use Distutils configuration files to record personal
or site preferences for any Distutils options.  That is, any option to any
command can be stored in one of two or three (depending on your platform)
configuration files, which will be consulted before the command-line is parsed.
This means that configuration files will override default values, and the
command-line will in turn override configuration files.  Furthermore, if
multiple configuration files apply, values from "earlier" files are overridden
by "later" files.


.. _inst-config-filenames:

Location and names of config files
----------------------------------

The names and locations of the configuration files vary slightly across
platforms.  On Unix and Mac OS X, the three configuration files (in the order
they are processed) are:

+--------------+----------------------------------------------------------+-------+
| Type of file | Location and filename                                    | Notes |
+==============+==========================================================+=======+
| system       | :file:`{prefix}/lib/python{ver}/distutils/distutils.cfg` | \(1)  |
+--------------+----------------------------------------------------------+-------+
| personal     | :file:`$HOME/.pydistutils.cfg`                           | \(2)  |
+--------------+----------------------------------------------------------+-------+
| local        | :file:`setup.cfg`                                        | \(3)  |
+--------------+----------------------------------------------------------+-------+

And on Windows, the configuration files are:

+--------------+-------------------------------------------------+-------+
| Type of file | Location and filename                           | Notes |
+==============+=================================================+=======+
| system       | :file:`{prefix}\\Lib\\distutils\\distutils.cfg` | \(4)  |
+--------------+-------------------------------------------------+-------+
| personal     | :file:`%HOME%\\pydistutils.cfg`                 | \(5)  |
+--------------+-------------------------------------------------+-------+
| local        | :file:`setup.cfg`                               | \(3)  |
+--------------+-------------------------------------------------+-------+

775 776 777
On all platforms, the "personal" file can be temporarily disabled by
passing the `--no-user-cfg` option.

778 779 780 781 782 783 784 785 786 787 788 789
Notes:

(1)
   Strictly speaking, the system-wide configuration file lives in the directory
   where the Distutils are installed; under Python 1.6 and later on Unix, this is
   as shown. For Python 1.5.2, the Distutils will normally be installed to
   :file:`{prefix}/lib/python1.5/site-packages/distutils`, so the system
   configuration file should be put there under Python 1.5.2.

(2)
   On Unix, if the :envvar:`HOME` environment variable is not defined, the user's
   home directory will be determined with the :func:`getpwuid` function from the
790 791
   standard :mod:`pwd` module. This is done by the :func:`os.path.expanduser`
   function used by Distutils.
792 793 794 795 796 797 798 799 800 801 802 803 804 805

(3)
   I.e., in the current directory (usually the location of the setup script).

(4)
   (See also note (1).)  Under Python 1.6 and later, Python's default "installation
   prefix" is :file:`C:\\Python`, so the system configuration file is normally
   :file:`C:\\Python\\Lib\\distutils\\distutils.cfg`. Under Python 1.5.2, the
   default prefix was :file:`C:\\Program Files\\Python`, and the Distutils were not
   part of the standard library---so the system configuration file would be
   :file:`C:\\Program Files\\Python\\distutils\\distutils.cfg` in a standard Python
   1.5.2 installation under Windows.

(5)
806 807 808 809
   On Windows, if the :envvar:`HOME` environment variable is not defined,
   :envvar:`USERPROFILE` then :envvar:`HOMEDRIVE` and :envvar:`HOMEPATH` will
   be tried. This is done by the :func:`os.path.expanduser` function used
   by Distutils.
810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957


.. _inst-config-syntax:

Syntax of config files
----------------------

The Distutils configuration files all have the same syntax.  The config files
are grouped into sections.  There is one section for each Distutils command,
plus a ``global`` section for global options that affect every command.  Each
section consists of one option per line, specified as ``option=value``.

For example, the following is a complete config file that just forces all
commands to run quietly by default::

   [global]
   verbose=0

If this is installed as the system config file, it will affect all processing of
any Python module distribution by any user on the current system.  If it is
installed as your personal config file (on systems that support them), it will
affect only module distributions processed by you.  And if it is used as the
:file:`setup.cfg` for a particular module distribution, it affects only that
distribution.

You could override the default "build base" directory and make the
:command:`build\*` commands always forcibly rebuild all files with the
following::

   [build]
   build-base=blib
   force=1

which corresponds to the command-line arguments ::

   python setup.py build --build-base=blib --force

except that including the :command:`build` command on the command-line means
that command will be run.  Including a particular command in config files has no
such implication; it only means that if the command is run, the options in the
config file will apply.  (Or if other commands that derive values from it are
run, they will use the values in the config file.)

You can find out the complete list of options for any command using the
:option:`--help` option, e.g.::

   python setup.py build --help

and you can find out the complete list of global options by using
:option:`--help` without a command::

   python setup.py --help

See also the "Reference" section of the "Distributing Python Modules" manual.


.. _inst-building-ext:

Building Extensions: Tips and Tricks
====================================

Whenever possible, the Distutils try to use the configuration information made
available by the Python interpreter used to run the :file:`setup.py` script.
For example, the same compiler and linker flags used to compile Python will also
be used for compiling extensions.  Usually this will work well, but in
complicated situations this might be inappropriate.  This section discusses how
to override the usual Distutils behaviour.


.. _inst-tweak-flags:

Tweaking compiler/linker flags
------------------------------

Compiling a Python extension written in C or C++ will sometimes require
specifying custom flags for the compiler and linker in order to use a particular
library or produce a special kind of object code. This is especially true if the
extension hasn't been tested on your platform, or if you're trying to
cross-compile Python.

In the most general case, the extension author might have foreseen that
compiling the extensions would be complicated, and provided a :file:`Setup` file
for you to edit.  This will likely only be done if the module distribution
contains many separate extension modules, or if they often require elaborate
sets of compiler flags in order to work.

A :file:`Setup` file, if present, is parsed in order to get a list of extensions
to build.  Each line in a :file:`Setup` describes a single module.  Lines have
the following structure::

   module ... [sourcefile ...] [cpparg ...] [library ...]


Let's examine each of the fields in turn.

* *module* is the name of the extension module to be built, and should be a
  valid Python identifier.  You can't just change this in order to rename a module
  (edits to the source code would also be needed), so this should be left alone.

* *sourcefile* is anything that's likely to be a source code file, at least
  judging by the filename.  Filenames ending in :file:`.c` are assumed to be
  written in C, filenames ending in :file:`.C`, :file:`.cc`, and :file:`.c++` are
  assumed to be C++, and filenames ending in :file:`.m` or :file:`.mm` are assumed
  to be in Objective C.

* *cpparg* is an argument for the C preprocessor,  and is anything starting with
  :option:`-I`, :option:`-D`, :option:`-U` or :option:`-C`.

* *library* is anything ending in :file:`.a` or beginning with :option:`-l` or
  :option:`-L`.

If a particular platform requires a special library on your platform, you can
add it by editing the :file:`Setup` file and running ``python setup.py build``.
For example, if the module defined by the line ::

   foo foomodule.c

must be linked with the math library :file:`libm.a` on your platform, simply add
:option:`-lm` to the line::

   foo foomodule.c -lm

Arbitrary switches intended for the compiler or the linker can be supplied with
the :option:`-Xcompiler` *arg* and :option:`-Xlinker` *arg* options::

   foo foomodule.c -Xcompiler -o32 -Xlinker -shared -lm

The next option after :option:`-Xcompiler` and :option:`-Xlinker` will be
appended to the proper command line, so in the above example the compiler will
be passed the :option:`-o32` option, and the linker will be passed
:option:`-shared`.  If a compiler option requires an argument, you'll have to
supply multiple :option:`-Xcompiler` options; for example, to pass ``-x c++``
the :file:`Setup` file would have to contain ``-Xcompiler -x -Xcompiler c++``.

Compiler flags can also be supplied through setting the :envvar:`CFLAGS`
environment variable.  If set, the contents of :envvar:`CFLAGS` will be added to
the compiler flags specified in the  :file:`Setup` file.


.. _inst-non-ms-compilers:

Using non-Microsoft compilers on Windows
----------------------------------------

.. sectionauthor:: Rene Liebscher <R.Liebscher@gmx.de>



Christian Heimes's avatar
Christian Heimes committed
958 959
Borland/CodeGear C++
^^^^^^^^^^^^^^^^^^^^
960

Christian Heimes's avatar
Christian Heimes committed
961
This subsection describes the necessary steps to use Distutils with the Borland
962 963 964 965 966 967 968
C++ compiler version 5.5.  First you have to know that Borland's object file
format (OMF) is different from the format used by the Python version you can
download from the Python or ActiveState Web site.  (Python is built with
Microsoft Visual C++, which uses COFF as the object file format.) For this
reason you have to convert Python's library :file:`python25.lib` into the
Borland format.  You can do this as follows:

969 970
.. Should we mention that users have to create cfg-files for the compiler?
.. see also http://community.borland.com/article/0,1410,21205,00.html
971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000

::

   coff2omf python25.lib python25_bcpp.lib

The :file:`coff2omf` program comes with the Borland compiler.  The file
:file:`python25.lib` is in the :file:`Libs` directory of your Python
installation.  If your extension uses other libraries (zlib, ...) you have to
convert them too.

The converted files have to reside in the same directories as the normal
libraries.

How does Distutils manage to use these libraries with their changed names?  If
the extension needs a library (eg. :file:`foo`) Distutils checks first if it
finds a library with suffix :file:`_bcpp` (eg. :file:`foo_bcpp.lib`) and then
uses this library.  In the case it doesn't find such a special library it uses
the default name (:file:`foo.lib`.) [#]_

To let Distutils compile your extension with Borland C++ you now have to type::

   python setup.py build --compiler=bcpp

If you want to use the Borland C++ compiler as the default, you could specify
this in your personal or system-wide configuration file for Distutils (see
section :ref:`inst-config-files`.)


.. seealso::

Christian Heimes's avatar
Christian Heimes committed
1001
   `C++Builder Compiler <http://www.codegear.com/downloads/free/cppbuilder>`_
1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017
      Information about the free C++ compiler from Borland, including links to the
      download pages.

   `Creating Python Extensions Using Borland's Free Compiler <http://www.cyberus.ca/~g_will/pyExtenDL.shtml>`_
      Document describing how to use Borland's free command-line C++ compiler to build
      Python.


GNU C / Cygwin / MinGW
^^^^^^^^^^^^^^^^^^^^^^

This section describes the necessary steps to use Distutils with the GNU C/C++
compilers in their Cygwin and MinGW distributions. [#]_ For a Python interpreter
that was built with Cygwin, everything should work without any of these
following steps.

1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040
Not all extensions can be built with MinGW or Cygwin, but many can.  Extensions
most likely to not work are those that use C++ or depend on Microsoft Visual C
extensions.

To let Distutils compile your extension with Cygwin you have to type::

   python setup.py build --compiler=cygwin

and for Cygwin in no-cygwin mode [#]_ or for MinGW type::

   python setup.py build --compiler=mingw32

If you want to use any of these options/compilers as default, you should
consider writing it in your personal or system-wide configuration file for
Distutils (see section :ref:`inst-config-files`.)

Older Versions of Python and MinGW
""""""""""""""""""""""""""""""""""
The following instructions only apply if you're using a version of Python
inferior to 2.4.1 with a MinGW inferior to 3.0.0 (with
binutils-2.13.90-20030111-1).

These compilers require some special libraries.  This task is more complex than
1041 1042
for Borland's C++, because there is no program to convert the library.  First
you have to create a list of symbols which the Python DLL exports. (You can find
Georg Brandl's avatar
Georg Brandl committed
1043 1044
a good program for this task at
http://www.emmestech.com/software/pexports-0.43/download_pexports.html).
1045

1046 1047
.. I don't understand what the next line means. --amk
.. (inclusive the references on data structures.)
1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086

::

   pexports python25.dll >python25.def

The location of an installed :file:`python25.dll` will depend on the
installation options and the version and language of Windows.  In a "just for
me" installation, it will appear in the root of the installation directory.  In
a shared installation, it will be located in the system directory.

Then you can create from these information an import library for gcc. ::

   /cygwin/bin/dlltool --dllname python25.dll --def python25.def --output-lib libpython25.a

The resulting library has to be placed in the same directory as
:file:`python25.lib`. (Should be the :file:`libs` directory under your Python
installation directory.)

If your extension uses other libraries (zlib,...) you might  have to convert
them too. The converted files have to reside in the same directories as the
normal libraries do.


.. seealso::

   `Building Python modules on MS Windows platform with MinGW <http://www.zope.org/Members/als/tips/win32_mingw_modules>`_
      Information about building the required libraries for the MinGW environment.


.. rubric:: Footnotes

.. [#] This also means you could replace all existing COFF-libraries with OMF-libraries
   of the same name.

.. [#] Check http://sources.redhat.com/cygwin/ and http://www.mingw.org/ for more
   information

.. [#] Then you have no POSIX emulation available, but you also don't need
   :file:`cygwin1.dll`.