html.parser.rst 11 KB
Newer Older
1 2
:mod:`html.parser` --- Simple HTML and XHTML parser
===================================================
3

4
.. module:: html.parser
5 6
   :synopsis: A simple parser that can handle HTML and XHTML.

7
**Source code:** :source:`Lib/html/parser.py`
8

9 10 11
.. index::
   single: HTML
   single: XHTML
12

Raymond Hettinger's avatar
Raymond Hettinger committed
13 14
--------------

15 16 17
This module defines a class :class:`HTMLParser` which serves as the basis for
parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML.

18
.. class:: HTMLParser(*, convert_charrefs=True)
19

20
   Create a parser instance able to parse invalid markup.
21

22
   If *convert_charrefs* is ``True`` (the default), all character
23 24 25
   references (except the ones in ``script``/``style`` elements) are
   automatically converted to the corresponding Unicode characters.

26 27 28 29
   An :class:`.HTMLParser` instance is fed HTML data and calls handler methods
   when start tags, end tags, text, comments, and other markup elements are
   encountered.  The user should subclass :class:`.HTMLParser` and override its
   methods to implement the desired behavior.
30

31 32
   This parser does not check that end tags match start tags or call the end-tag
   handler for elements which are closed implicitly by closing an outer element.
33

34 35 36
   .. versionchanged:: 3.4
      *convert_charrefs* keyword argument added.

37 38 39
   .. versionchanged:: 3.5
      The default value for argument *convert_charrefs* is now ``True``.

40

41 42
Example HTML Parser Application
-------------------------------
43

44 45 46
As a basic example, below is a simple HTML parser that uses the
:class:`HTMLParser` class to print out start tags, end tags, and data
as they are encountered::
47

48 49 50 51 52
   from html.parser import HTMLParser

   class MyHTMLParser(HTMLParser):
       def handle_starttag(self, tag, attrs):
           print("Encountered a start tag:", tag)
53

54 55
       def handle_endtag(self, tag):
           print("Encountered an end tag :", tag)
56

57 58 59
       def handle_data(self, data):
           print("Encountered some data  :", data)

60
   parser = MyHTMLParser()
61 62 63
   parser.feed('<html><head><title>Test</title></head>'
               '<body><h1>Parse me!</h1></body></html>')

64 65 66
The output will then be:

.. code-block:: none
67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85

   Encountered a start tag: html
   Encountered a start tag: head
   Encountered a start tag: title
   Encountered some data  : Test
   Encountered an end tag : title
   Encountered an end tag : head
   Encountered a start tag: body
   Encountered a start tag: h1
   Encountered some data  : Parse me!
   Encountered an end tag : h1
   Encountered an end tag : body
   Encountered an end tag : html


:class:`.HTMLParser` Methods
----------------------------

:class:`HTMLParser` instances have the following methods:
86 87 88 89 90 91


.. method:: HTMLParser.feed(data)

   Feed some text to the parser.  It is processed insofar as it consists of
   complete elements; incomplete data is buffered until more data is fed or
92
   :meth:`close` is called.  *data* must be :class:`str`.
93 94 95 96 97 98 99 100 101 102


.. method:: HTMLParser.close()

   Force processing of all buffered data as if it were followed by an end-of-file
   mark.  This method may be redefined by a derived class to define additional
   processing at the end of the input, but the redefined version should always call
   the :class:`HTMLParser` base class method :meth:`close`.


103 104 105 106 107 108
.. method:: HTMLParser.reset()

   Reset the instance.  Loses all unprocessed data.  This is called implicitly at
   instantiation time.


109 110 111 112 113 114 115 116 117 118 119 120 121
.. method:: HTMLParser.getpos()

   Return current line number and offset.


.. method:: HTMLParser.get_starttag_text()

   Return the text of the most recently opened start tag.  This should not normally
   be needed for structured processing, but may be useful in dealing with HTML "as
   deployed" or for re-generating input with minimal changes (whitespace between
   attributes can be preserved, etc.).


122 123 124 125 126
The following methods are called when data or markup elements are encountered
and they are meant to be overridden in a subclass.  The base class
implementations do nothing (except for :meth:`~HTMLParser.handle_startendtag`):


127 128
.. method:: HTMLParser.handle_starttag(tag, attrs)

129
   This method is called to handle the start of a tag (e.g. ``<div id="main">``).
130 131 132 133 134

   The *tag* argument is the name of the tag converted to lower case. The *attrs*
   argument is a list of ``(name, value)`` pairs containing the attributes found
   inside the tag's ``<>`` brackets.  The *name* will be translated to lower case,
   and quotes in the *value* have been removed, and character and entity references
135 136
   have been replaced.

137 138
   For instance, for the tag ``<A HREF="https://www.cwi.nl/">``, this method
   would be called as ``handle_starttag('a', [('href', 'https://www.cwi.nl/')])``.
139

140 141
   All entity references from :mod:`html.entities` are replaced in the attribute
   values.
142 143


144 145 146 147 148 149 150
.. method:: HTMLParser.handle_endtag(tag)

   This method is called to handle the end tag of an element (e.g. ``</div>``).

   The *tag* argument is the name of the tag converted to lower case.


151 152 153
.. method:: HTMLParser.handle_startendtag(tag, attrs)

   Similar to :meth:`handle_starttag`, but called when the parser encounters an
154
   XHTML-style empty tag (``<img ... />``).  This method may be overridden by
155
   subclasses which require this particular lexical information; the default
156
   implementation simply calls :meth:`handle_starttag` and :meth:`handle_endtag`.
157 158 159 160


.. method:: HTMLParser.handle_data(data)

161 162
   This method is called to process arbitrary data (e.g. text nodes and the
   content of ``<script>...</script>`` and ``<style>...</style>``).
163 164


165
.. method:: HTMLParser.handle_entityref(name)
166

167 168
   This method is called to process a named character reference of the form
   ``&name;`` (e.g. ``&gt;``), where *name* is a general entity reference
169 170
   (e.g. ``'gt'``).  This method is never called if *convert_charrefs* is
   ``True``.
171 172


173
.. method:: HTMLParser.handle_charref(name)
174

175 176 177
   This method is called to process decimal and hexadecimal numeric character
   references of the form ``&#NNN;`` and ``&#xNNN;``.  For example, the decimal
   equivalent for ``&gt;`` is ``&#62;``, whereas the hexadecimal is ``&#x3E;``;
178 179
   in this case the method will receive ``'62'`` or ``'x3E'``.  This method
   is never called if *convert_charrefs* is ``True``.
180 181 182 183


.. method:: HTMLParser.handle_comment(data)

184
   This method is called when a comment is encountered (e.g. ``<!--comment-->``).
185

186 187
   For example, the comment ``<!-- comment -->`` will cause this method to be
   called with the argument ``' comment '``.
188

189 190
   The content of Internet Explorer conditional comments (condcoms) will also be
   sent to this method, so, for ``<!--[if IE 9]>IE9-specific content<![endif]-->``,
191
   this method will receive ``'[if IE 9]>IE9-specific content<![endif]'``.
192

193

194
.. method:: HTMLParser.handle_decl(decl)
195

196 197
   This method is called to handle an HTML doctype declaration (e.g.
   ``<!DOCTYPE html>``).
198

199 200
   The *decl* parameter will be the entire contents of the declaration inside
   the ``<!...>`` markup (e.g. ``'DOCTYPE html'``).
201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217


.. method:: HTMLParser.handle_pi(data)

   Method called when a processing instruction is encountered.  The *data*
   parameter will contain the entire processing instruction. For example, for the
   processing instruction ``<?proc color='red'>``, this method would be called as
   ``handle_pi("proc color='red'")``.  It is intended to be overridden by a derived
   class; the base class implementation does nothing.

   .. note::

      The :class:`HTMLParser` class uses the SGML syntactic rules for processing
      instructions.  An XHTML processing instruction using the trailing ``'?'`` will
      cause the ``'?'`` to be included in *data*.


218
.. method:: HTMLParser.unknown_decl(data)
219

220 221 222 223
   This method is called when an unrecognized declaration is read by the parser.

   The *data* parameter will be the entire contents of the declaration inside
   the ``<![...]>`` markup.  It is sometimes useful to be overridden by a
224
   derived class.  The base class implementation does nothing.
225

226 227 228 229 230 231 232 233

.. _htmlparser-examples:

Examples
--------

The following class implements a parser that will be used to illustrate more
examples::
234 235

   from html.parser import HTMLParser
236
   from html.entities import name2codepoint
237 238 239

   class MyHTMLParser(HTMLParser):
       def handle_starttag(self, tag, attrs):
240 241 242
           print("Start tag:", tag)
           for attr in attrs:
               print("     attr:", attr)
243

244
       def handle_endtag(self, tag):
245
           print("End tag  :", tag)
246

247
       def handle_data(self, data):
248
           print("Data     :", data)
249

250 251
       def handle_comment(self, data):
           print("Comment  :", data)
252

253 254 255
       def handle_entityref(self, name):
           c = chr(name2codepoint[name])
           print("Named ent:", c)
256

257 258 259 260 261 262
       def handle_charref(self, name):
           if name.startswith('x'):
               c = chr(int(name[1:], 16))
           else:
               c = chr(int(name))
           print("Num ent  :", c)
263

264 265 266
       def handle_decl(self, data):
           print("Decl     :", data)

267
   parser = MyHTMLParser()
268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294

Parsing a doctype::

   >>> parser.feed('<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" '
   ...             '"http://www.w3.org/TR/html4/strict.dtd">')
   Decl     : DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"

Parsing an element with a few attributes and a title::

   >>> parser.feed('<img src="python-logo.png" alt="The Python logo">')
   Start tag: img
        attr: ('src', 'python-logo.png')
        attr: ('alt', 'The Python logo')
   >>>
   >>> parser.feed('<h1>Python</h1>')
   Start tag: h1
   Data     : Python
   End tag  : h1

The content of ``script`` and ``style`` elements is returned as is, without
further parsing::

   >>> parser.feed('<style type="text/css">#python { color: green }</style>')
   Start tag: style
        attr: ('type', 'text/css')
   Data     : #python { color: green }
   End tag  : style
295

296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318
   >>> parser.feed('<script type="text/javascript">'
   ...             'alert("<strong>hello!</strong>");</script>')
   Start tag: script
        attr: ('type', 'text/javascript')
   Data     : alert("<strong>hello!</strong>");
   End tag  : script

Parsing comments::

   >>> parser.feed('<!-- a comment -->'
   ...             '<!--[if IE 9]>IE-specific content<![endif]-->')
   Comment  :  a comment
   Comment  : [if IE 9]>IE-specific content<![endif]

Parsing named and numeric character references and converting them to the
correct char (note: these 3 references are all equivalent to ``'>'``)::

   >>> parser.feed('&gt;&#62;&#x3E;')
   Named ent: >
   Num ent  : >
   Num ent  : >

Feeding incomplete chunks to :meth:`~HTMLParser.feed` works, but
319 320
:meth:`~HTMLParser.handle_data` might be called more than once
(unless *convert_charrefs* is set to ``True``)::
321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340

   >>> for chunk in ['<sp', 'an>buff', 'ered ', 'text</s', 'pan>']:
   ...     parser.feed(chunk)
   ...
   Start tag: span
   Data     : buff
   Data     : ered
   Data     : text
   End tag  : span

Parsing invalid HTML (e.g. unquoted attributes) also works::

   >>> parser.feed('<p><a class=link href=#main>tag soup</p ></a>')
   Start tag: p
   Start tag: a
        attr: ('class', 'link')
        attr: ('href', '#main')
   Data     : tag soup
   End tag  : p
   End tag  : a