gettext.rst 24.9 KB
Newer Older
1 2 3 4 5
:mod:`gettext` --- Multilingual internationalization services
=============================================================

.. module:: gettext
   :synopsis: Multilingual internationalization services.
6

Andrew Kuchling's avatar
Andrew Kuchling committed
7 8
.. moduleauthor:: Barry A. Warsaw <barry@python.org>
.. sectionauthor:: Barry A. Warsaw <barry@python.org>
9

Raymond Hettinger's avatar
Raymond Hettinger committed
10 11 12
**Source code:** :source:`Lib/gettext.py`

--------------
13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

The :mod:`gettext` module provides internationalization (I18N) and localization
(L10N) services for your Python modules and applications. It supports both the
GNU ``gettext`` message catalog API and a higher level, class-based API that may
be more appropriate for Python files.  The interface described below allows you
to write your module and application messages in one natural language, and
provide a catalog of translated messages for running under different natural
languages.

Some hints on localizing your Python modules and applications are also given.


GNU :program:`gettext` API
--------------------------

The :mod:`gettext` module defines the following API, which is very similar to
the GNU :program:`gettext` API.  If you use this API you will affect the
translation of your entire application globally.  Often this is what you want if
your application is monolingual, with the choice of language dependent on the
locale of your user.  If you are localizing a Python module, or if your
application needs to switch languages on the fly, you probably want to use the
class-based API instead.


37
.. function:: bindtextdomain(domain, localedir=None)
38 39 40 41 42 43 44 45 46 47 48

   Bind the *domain* to the locale directory *localedir*.  More concretely,
   :mod:`gettext` will look for binary :file:`.mo` files for the given domain using
   the path (on Unix): :file:`localedir/language/LC_MESSAGES/domain.mo`, where
   *languages* is searched for in the environment variables :envvar:`LANGUAGE`,
   :envvar:`LC_ALL`, :envvar:`LC_MESSAGES`, and :envvar:`LANG` respectively.

   If *localedir* is omitted or ``None``, then the current binding for *domain* is
   returned. [#]_


49
.. function:: bind_textdomain_codeset(domain, codeset=None)
50 51 52 53 54 55

   Bind the *domain* to *codeset*, changing the encoding of strings returned by the
   :func:`gettext` family of functions. If *codeset* is omitted, then the current
   binding is returned.


56
.. function:: textdomain(domain=None)
57 58 59 60 61 62 63 64 65 66 67 68 69 70 71

   Change or query the current global domain.  If *domain* is ``None``, then the
   current global domain is returned, otherwise the global domain is set to
   *domain*, which is returned.


.. function:: gettext(message)

   Return the localized translation of *message*, based on the current global
   domain, language, and locale directory.  This function is usually aliased as
   :func:`_` in the local namespace (see examples below).


.. function:: lgettext(message)

72 73
   Equivalent to :func:`gettext`, but the translation is returned in the
   preferred system encoding, if no other encoding was explicitly set with
74 75 76 77 78 79 80 81 82 83
   :func:`bind_textdomain_codeset`.


.. function:: dgettext(domain, message)

   Like :func:`gettext`, but look the message up in the specified *domain*.


.. function:: ldgettext(domain, message)

84 85
   Equivalent to :func:`dgettext`, but the translation is returned in the
   preferred system encoding, if no other encoding was explicitly set with
86 87 88 89 90 91 92 93 94 95 96 97
   :func:`bind_textdomain_codeset`.


.. function:: ngettext(singular, plural, n)

   Like :func:`gettext`, but consider plural forms. If a translation is found,
   apply the plural formula to *n*, and return the resulting message (some
   languages have more than two plural forms). If no translation is found, return
   *singular* if *n* is 1; return *plural* otherwise.

   The Plural formula is taken from the catalog header. It is a C or Python
   expression that has a free variable *n*; the expression evaluates to the index
98 99 100 101
   of the plural in the catalog. See
   `the GNU gettext documentation <https://www.gnu.org/software/gettext/manual/gettext.html>`__
   for the precise syntax to be used in :file:`.po` files and the
   formulas for a variety of languages.
102 103 104 105


.. function:: lngettext(singular, plural, n)

106 107
   Equivalent to :func:`ngettext`, but the translation is returned in the
   preferred system encoding, if no other encoding was explicitly set with
108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132
   :func:`bind_textdomain_codeset`.


.. function:: dngettext(domain, singular, plural, n)

   Like :func:`ngettext`, but look the message up in the specified *domain*.


.. function:: ldngettext(domain, singular, plural, n)

   Equivalent to :func:`dngettext`, but the translation is returned in the
   preferred system encoding, if no other encoding was explicitly set with
   :func:`bind_textdomain_codeset`.


Note that GNU :program:`gettext` also defines a :func:`dcgettext` method, but
this was deemed not useful and so it is currently unimplemented.

Here's an example of typical usage for this API::

   import gettext
   gettext.bindtextdomain('myapplication', '/path/to/my/language/directory')
   gettext.textdomain('myapplication')
   _ = gettext.gettext
   # ...
133
   print(_('This is a translatable string.'))
134 135 136 137 138 139 140 141 142


Class-based API
---------------

The class-based API of the :mod:`gettext` module gives you more flexibility and
greater convenience than the GNU :program:`gettext` API.  It is the recommended
way of localizing your Python applications and modules.  :mod:`gettext` defines
a "translations" class which implements the parsing of GNU :file:`.mo` format
143 144 145
files, and has methods for returning strings. Instances of this "translations"
class can also install themselves in the built-in namespace as the function
:func:`_`.
146 147


148
.. function:: find(domain, localedir=None, languages=None, all=False)
149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165

   This function implements the standard :file:`.mo` file search algorithm.  It
   takes a *domain*, identical to what :func:`textdomain` takes.  Optional
   *localedir* is as in :func:`bindtextdomain`  Optional *languages* is a list of
   strings, where each string is a language code.

   If *localedir* is not given, then the default system locale directory is used.
   [#]_  If *languages* is not given, then the following environment variables are
   searched: :envvar:`LANGUAGE`, :envvar:`LC_ALL`, :envvar:`LC_MESSAGES`, and
   :envvar:`LANG`.  The first one returning a non-empty value is used for the
   *languages* variable. The environment variables should contain a colon separated
   list of languages, which will be split on the colon to produce the expected list
   of language code strings.

   :func:`find` then expands and normalizes the languages, and then iterates
   through them, searching for an existing file built of these components:

166
   :file:`{localedir}/{language}/LC_MESSAGES/{domain}.mo`
167 168 169 170 171 172 173

   The first such file name that exists is returned by :func:`find`. If no such
   file is found, then ``None`` is returned. If *all* is given, it returns a list
   of all file names, in the order in which they appear in the languages list or
   the environment variables.


174
.. function:: translation(domain, localedir=None, languages=None, class_=None, fallback=False, codeset=None)
175

176 177
   Return a :class:`Translations` instance based on the *domain*, *localedir*,
   and *languages*, which are first passed to :func:`find` to get a list of the
178
   associated :file:`.mo` file paths.  Instances with identical :file:`.mo` file
179 180
   names are cached.  The actual class instantiated is either *class_* if
   provided, otherwise :class:`GNUTranslations`.  The class's constructor must
181 182
   take a single :term:`file object` argument.  If provided, *codeset* will change
   the charset used to encode translated strings in the :meth:`lgettext` and
183
   :meth:`lngettext` methods.
184 185 186 187 188 189

   If multiple files are found, later files are used as fallbacks for earlier ones.
   To allow setting the fallback, :func:`copy.copy` is used to clone each
   translation object from the cache; the actual instance data is still shared with
   the cache.

190
   If no :file:`.mo` file is found, this function raises :exc:`OSError` if
191 192 193
   *fallback* is false (which is the default), and returns a
   :class:`NullTranslations` instance if *fallback* is true.

194 195 196
   .. versionchanged:: 3.3
      :exc:`IOError` used to be raised instead of :exc:`OSError`.

197

198
.. function:: install(domain, localedir=None, codeset=None, names=None)
199

200
   This installs the function :func:`_` in Python's builtins namespace, based on
201
   *domain*, *localedir*, and *codeset* which are passed to the function
202
   :func:`translation`.
203 204

   For the *names* parameter, please see the description of the translation
205
   object's :meth:`~NullTranslations.install` method.
206 207 208 209 210

   As seen below, you usually mark the strings in your application that are
   candidates for translation, by wrapping them in a call to the :func:`_`
   function, like this::

211
      print(_('This string will be translated.'))
212 213

   For convenience, you want the :func:`_` function to be installed in Python's
214
   builtins namespace, so it is easily accessible in all modules of your
215 216 217 218 219 220 221 222 223 224 225 226 227
   application.


The :class:`NullTranslations` class
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Translation classes are what actually implement the translation of original
source file message strings to translated message strings. The base class used
by all translation classes is :class:`NullTranslations`; this provides the basic
interface you can use to write your own specialized translation classes.  Here
are the methods of :class:`NullTranslations`:


228
.. class:: NullTranslations(fp=None)
229

230
   Takes an optional :term:`file object` *fp*, which is ignored by the base class.
231 232 233 234 235
   Initializes "protected" instance variables *_info* and *_charset* which are set
   by derived classes, as well as *_fallback*, which is set through
   :meth:`add_fallback`.  It then calls ``self._parse(fp)`` if *fp* is not
   ``None``.

236
   .. method:: _parse(fp)
237

238 239 240 241
      No-op'd in the base class, this method takes file object *fp*, and reads
      the data from the file, initializing its message catalog.  If you have an
      unsupported message catalog file format, you should override this method
      to parse your format.
242 243


244
   .. method:: add_fallback(fallback)
245

246 247 248
      Add *fallback* as the fallback object for the current translation object.
      A translation object should consult the fallback if it cannot provide a
      translation for a given message.
249 250


251
   .. method:: gettext(message)
252

253 254
      If a fallback has been set, forward :meth:`gettext` to the fallback.
      Otherwise, return the translated message.  Overridden in derived classes.
255 256


257
   .. method:: lgettext(message)
258

259 260
      If a fallback has been set, forward :meth:`lgettext` to the fallback.
      Otherwise, return the translated message.  Overridden in derived classes.
261 262


263
   .. method:: ngettext(singular, plural, n)
264

265 266
      If a fallback has been set, forward :meth:`ngettext` to the fallback.
      Otherwise, return the translated message.  Overridden in derived classes.
267 268


269
   .. method:: lngettext(singular, plural, n)
270

271
      If a fallback has been set, forward :meth:`lngettext` to the fallback.
272
      Otherwise, return the translated message.  Overridden in derived classes.
273 274


275
   .. method:: info()
276

277
      Return the "protected" :attr:`_info` variable.
278 279


280
   .. method:: charset()
281

282 283
      Return the "protected" :attr:`_charset` variable, which is the encoding of
      the message catalog file.
284 285


286
   .. method:: output_charset()
287

288 289 290
      Return the "protected" :attr:`_output_charset` variable, which defines the
      encoding used to return translated messages in :meth:`lgettext` and
      :meth:`lngettext`.
291 292


293
   .. method:: set_output_charset(charset)
294

295 296
      Change the "protected" :attr:`_output_charset` variable, which defines the
      encoding used to return translated messages.
297 298


299
   .. method:: install(names=None)
300

301 302
      This method installs :meth:`self.gettext` into the built-in namespace,
      binding it to ``_``.
303

304
      If the *names* parameter is given, it must be a sequence containing the
305
      names of functions you want to install in the builtins namespace in
306 307 308
      addition to :func:`_`.  Supported names are ``'gettext'`` (bound to
      :meth:`self.gettext`), ``'ngettext'`` (bound to :meth:`self.ngettext`),
      ``'lgettext'`` and ``'lngettext'``.
309

310 311 312 313 314
      Note that this is only one way, albeit the most convenient way, to make
      the :func:`_` function available to your application.  Because it affects
      the entire application globally, and specifically the built-in namespace,
      localized modules should never install :func:`_`. Instead, they should use
      this code to make :func:`_` available to their module::
315

316 317 318
         import gettext
         t = gettext.translation('mymodule', ...)
         _ = t.gettext
319

320 321
      This puts :func:`_` only in the module's global namespace and so only
      affects calls within this module.
322 323 324 325 326 327 328 329


The :class:`GNUTranslations` class
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The :mod:`gettext` module provides one additional class derived from
:class:`NullTranslations`: :class:`GNUTranslations`.  This class overrides
:meth:`_parse` to enable reading GNU :program:`gettext` format :file:`.mo` files
330
in both big-endian and little-endian format.
331 332 333 334 335 336 337 338 339

:class:`GNUTranslations` parses optional meta-data out of the translation
catalog.  It is convention with GNU :program:`gettext` to include meta-data as
the translation for the empty string.  This meta-data is in :rfc:`822`\ -style
``key: value`` pairs, and should contain the ``Project-Id-Version`` key.  If the
key ``Content-Type`` is found, then the ``charset`` property is used to
initialize the "protected" :attr:`_charset` instance variable, defaulting to
``None`` if not found.  If the charset encoding is specified, then all message
ids and message strings read from the catalog are converted to Unicode using
340 341 342 343
this encoding, else ASCII encoding is assumed.

Since message ids are read as Unicode strings too, all :meth:`*gettext` methods
will assume message ids as Unicode strings, not byte strings.
344 345 346 347

The entire set of key/value pairs are placed into a dictionary and set as the
"protected" :attr:`_info` instance variable.

348 349 350
If the :file:`.mo` file's magic number is invalid, the major version number is
unexpected, or if other problems occur while reading the file, instantiating a
:class:`GNUTranslations` class can raise :exc:`OSError`.
351 352 353 354 355 356 357

The following methods are overridden from the base class implementation:


.. method:: GNUTranslations.gettext(message)

   Look up the *message* id in the catalog and return the corresponding message
358 359 360
   string, as a Unicode string.  If there is no entry in the catalog for the
   *message* id, and a fallback has been set, the look up is forwarded to the
   fallback's :meth:`gettext` method.  Otherwise, the *message* id is returned.
361 362 363 364


.. method:: GNUTranslations.lgettext(message)

365 366 367
   Equivalent to :meth:`gettext`, but the translation is returned as a
   bytestring encoded in the selected output charset, or in the preferred system
   encoding if no encoding was explicitly set with :meth:`set_output_charset`.
368 369 370 371 372 373


.. method:: GNUTranslations.ngettext(singular, plural, n)

   Do a plural-forms lookup of a message id.  *singular* is used as the message id
   for purposes of lookup in the catalog, while *n* is used to determine which
374
   plural form to use.  The returned message string is a Unicode string.
375 376 377 378

   If the message id is not found in the catalog, and a fallback is specified, the
   request is forwarded to the fallback's :meth:`ngettext` method.  Otherwise, when
   *n* is 1 *singular* is returned, and *plural* is returned in all other cases.
379

380 381 382 383
   Here is an example::

      n = len(os.listdir('.'))
      cat = GNUTranslations(somefile)
384
      message = cat.ngettext(
385 386 387 388 389
          'There is %(num)d file in this directory',
          'There are %(num)d files in this directory',
          n) % {'num': n}


390 391
.. method:: GNUTranslations.lngettext(singular, plural, n)

392 393 394
   Equivalent to :meth:`gettext`, but the translation is returned as a
   bytestring encoded in the selected output charset, or in the preferred system
   encoding if no encoding was explicitly set with :meth:`set_output_charset`.
395 396


397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415
Solaris message catalog support
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The Solaris operating system defines its own binary :file:`.mo` file format, but
since no documentation can be found on this format, it is not supported at this
time.


The Catalog constructor
^^^^^^^^^^^^^^^^^^^^^^^

.. index:: single: GNOME

GNOME uses a version of the :mod:`gettext` module by James Henstridge, but this
version has a slightly different API.  Its documented usage was::

   import gettext
   cat = gettext.Catalog(domain, localedir)
   _ = cat.gettext
416
   print(_('hello world'))
417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455

For compatibility with this older module, the function :func:`Catalog` is an
alias for the :func:`translation` function described above.

One difference between this module and Henstridge's: his catalog objects
supported access through a mapping API, but this appears to be unused and so is
not currently supported.


Internationalizing your programs and modules
--------------------------------------------

Internationalization (I18N) refers to the operation by which a program is made
aware of multiple languages.  Localization (L10N) refers to the adaptation of
your program, once internationalized, to the local language and cultural habits.
In order to provide multilingual messages for your Python programs, you need to
take the following steps:

#. prepare your program or module by specially marking translatable strings

#. run a suite of tools over your marked files to generate raw messages catalogs

#. create language specific translations of the message catalogs

#. use the :mod:`gettext` module so that message strings are properly translated

In order to prepare your code for I18N, you need to look at all the strings in
your files.  Any string that needs to be translated should be marked by wrapping
it in ``_('...')`` --- that is, a call to the function :func:`_`.  For example::

   filename = 'mylog.txt'
   message = _('writing a log message')
   fp = open(filename, 'w')
   fp.write(message)
   fp.close()

In this example, the string ``'writing a log message'`` is marked as a candidate
for translation, while the strings ``'mylog.txt'`` and ``'w'`` are not.

456 457 458 459 460 461 462 463
There are a few tools to extract the strings meant for translation.
The original GNU :program:`gettext` only supported C or C++ source
code but its extended version :program:`xgettext` scans code written
in a number of languages, including Python, to find strings marked as
translatable.  `Babel <http://babel.pocoo.org/>`__ is a Python
internationalization library that includes a :file:`pybabel` script to
extract and compile message catalogs.  François Pinard's program
called :program:`xpot` does a similar job and is available as part of
464
his `po-utils package <https://github.com/pinard/po-utils>`__.
465 466 467 468 469 470 471 472 473 474 475 476 477 478 479

(Python also includes pure-Python versions of these programs, called
:program:`pygettext.py` and :program:`msgfmt.py`; some Python distributions
will install them for you.  :program:`pygettext.py` is similar to
:program:`xgettext`, but only understands Python source code and
cannot handle other programming languages such as C or C++.
:program:`pygettext.py` supports a command-line interface similar to
:program:`xgettext`; for details on its use, run ``pygettext.py
--help``.  :program:`msgfmt.py` is binary compatible with GNU
:program:`msgfmt`.  With these two programs, you may not need the GNU
:program:`gettext` package to internationalize your Python
applications.)

:program:`xgettext`, :program:`pygettext`, and similar tools generate
:file:`.po` files that are message catalogs.  They are structured
480 481 482
human-readable files that contain every marked string in the source
code, along with a placeholder for the translated versions of these
strings.
483 484 485 486 487 488 489 490 491

Copies of these :file:`.po` files are then handed over to the
individual human translators who write translations for every
supported natural language.  They send back the completed
language-specific versions as a :file:`<language-name>.po` file that's
compiled into a machine-readable :file:`.mo` binary catalog file using
the :program:`msgfmt` program.  The :file:`.mo` files are used by the
:mod:`gettext` module for the actual translation processing at
run-time.
492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528

How you use the :mod:`gettext` module in your code depends on whether you are
internationalizing a single module or your entire application. The next two
sections will discuss each case.


Localizing your module
^^^^^^^^^^^^^^^^^^^^^^

If you are localizing your module, you must take care not to make global
changes, e.g. to the built-in namespace.  You should not use the GNU ``gettext``
API but instead the class-based API.

Let's say your module is called "spam" and the module's various natural language
translation :file:`.mo` files reside in :file:`/usr/share/locale` in GNU
:program:`gettext` format.  Here's what you would put at the top of your
module::

   import gettext
   t = gettext.translation('spam', '/usr/share/locale')
   _ = t.lgettext


Localizing your application
^^^^^^^^^^^^^^^^^^^^^^^^^^^

If you are localizing your application, you can install the :func:`_` function
globally into the built-in namespace, usually in the main driver file of your
application.  This will let all your application-specific files just use
``_('...')`` without having to explicitly install it in each file.

In the simple case then, you need only add the following bit of code to the main
driver file of your application::

   import gettext
   gettext.install('myapplication')

529
If you need to set the locale directory, you can pass it into the
530
:func:`install` function::
531 532

   import gettext
533
   gettext.install('myapplication', '/usr/share/locale')
534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567


Changing languages on the fly
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If your program needs to support many languages at the same time, you may want
to create multiple translation instances and then switch between them
explicitly, like so::

   import gettext

   lang1 = gettext.translation('myapplication', languages=['en'])
   lang2 = gettext.translation('myapplication', languages=['fr'])
   lang3 = gettext.translation('myapplication', languages=['de'])

   # start by using language1
   lang1.install()

   # ... time goes by, user selects language 2
   lang2.install()

   # ... more time goes by, user selects language 3
   lang3.install()


Deferred translations
^^^^^^^^^^^^^^^^^^^^^

In most coding situations, strings are translated where they are coded.
Occasionally however, you need to mark strings for translation, but defer actual
translation until later.  A classic example is::

   animals = ['mollusk',
              'albatross',
568 569 570
              'rat',
              'penguin',
              'python', ]
571 572
   # ...
   for a in animals:
573
       print(a)
574 575 576 577 578 579 580 581 582 583 584

Here, you want to mark the strings in the ``animals`` list as being
translatable, but you don't actually want to translate them until they are
printed.

Here is one way you can handle this situation::

   def _(message): return message

   animals = [_('mollusk'),
              _('albatross'),
585 586 587
              _('rat'),
              _('penguin'),
              _('python'), ]
588 589 590 591 592

   del _

   # ...
   for a in animals:
593
       print(_(a))
594 595 596 597 598 599 600 601

This works because the dummy definition of :func:`_` simply returns the string
unchanged.  And this dummy definition will temporarily override any definition
of :func:`_` in the built-in namespace (until the :keyword:`del` command). Take
care, though if you have a previous definition of :func:`_` in the local
namespace.

Note that the second use of :func:`_` will not identify "a" as being
602 603
translatable to the :program:`gettext` program, because the parameter
is not a string literal.
604 605 606 607 608 609 610

Another way to handle this is with the following example::

   def N_(message): return message

   animals = [N_('mollusk'),
              N_('albatross'),
611 612 613
              N_('rat'),
              N_('penguin'),
              N_('python'), ]
614 615 616

   # ...
   for a in animals:
617
       print(_(a))
618

619 620 621 622 623 624 625 626
In this case, you are marking translatable strings with the function
:func:`N_`, which won't conflict with any definition of :func:`_`.
However, you will need to teach your message extraction program to
look for translatable strings marked with :func:`N_`. :program:`xgettext`,
:program:`pygettext`, ``pybabel extract``, and :program:`xpot` all
support this through the use of the :option:`-k` command-line switch.
The choice of :func:`N_` here is totally arbitrary; it could have just
as easily been :func:`MarkThisStringForTranslation`.
627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660


Acknowledgements
----------------

The following people contributed code, feedback, design suggestions, previous
implementations, and valuable experience to the creation of this module:

* Peter Funk

* James Henstridge

* Juan David Ibáñez Palomar

* Marc-André Lemburg

* Martin von Löwis

* François Pinard

* Barry Warsaw

* Gustavo Niemeyer

.. rubric:: Footnotes

.. [#] The default locale directory is system dependent; for example, on RedHat Linux
   it is :file:`/usr/share/locale`, but on Solaris it is :file:`/usr/lib/locale`.
   The :mod:`gettext` module does not try to support these system dependent
   defaults; instead its default is :file:`sys.prefix/share/locale`. For this
   reason, it is always best to call :func:`bindtextdomain` with an explicit
   absolute path at the start of your application.

.. [#] See the footnote for :func:`bindtextdomain` above.