Create _ast module.

Cleanup Python-ast.c generation.

Doc/lib/lib.tex

@@ -416,6 +416,7 @@ and how to embed it in other applications.
 \input{distutils}
 
 \input{compiler}                % compiler package
+\input{libast}
 
 \input{libmisc}                 % Miscellaneous Services
 \input{libformatter}

Doc/lib/libast.tex

+% XXX Label can't be _ast?
+% XXX Where should this section/chapter go?
+\chapter{Abstract Syntax Trees\label{ast}}
+
+\sectionauthor{Martin v. L\"owis}{martin@v.loewis.de}
+
+The \code{_ast} module helps Python applications to process
+trees of the Python abstract syntax grammar. The Python compiler
+currently provides read-only access to such trees, meaning that
+applications can only create a tree for a given piece of Python
+source code; generating byte code from a (potentially modified)
+tree is not supported. The abstract syntax itself might change with
+each Python release; this module helps to find out programmatically
+what the current grammar looks like.
+
+An abstract syntax tree can be generated by passing \code{_ast.PyCF_ONLY_AST}
+as a flag to the \function{compile} builtin function. The result will be a tree
+of objects whose classes all inherit from \code{_ast.AST}.
+
+The actual classes are derived from the \code{Parser/Python.asdl} file,
+which is reproduced below. There is one class defined for each left-hand
+side symbol in the abstract grammar (for example, \code{_ast.stmt} or \code{_ast.expr}).
+In addition, there is one class defined for each constructor on the
+right-hand side; these classes inherit from the classes for the left-hand
+side trees. For example, \code{_ast.BinOp} inherits from \code{_ast.expr}.
+For production rules with alternatives (aka "sums"), the left-hand side
+class is abstract: only instances of specific constructor nodes are ever
+created.
+
+Each concrete class has an attribute \code{_fields} which gives the
+names of all child nodes.
+
+Each instance of a concrete class has one attribute for each child node,
+of the type as defined in the grammar. For example, \code{_ast.BinOp}
+instances have an attribute \code{left} of type \code{_ast.expr}.
+
+If these attributes are marked as optional in the grammar (using a
+question mark), the value might be \code{None}. If the attributes
+can have zero-or-more values (marked with an asterisk), the
+values are represented as Python lists.
+
+\subsection{Abstract Grammar}
+
+The abstract grammar is currently defined as follows:
+
+\verbatiminput{../../Parser/Python.asdl}
\ No newline at end of file

Misc/NEWS

@@ -76,7 +76,7 @@ Core and builtins
 
 - A new AST parser implementation was completed. The abstract
   syntax tree is available for read-only (non-compile) access
-  to Python code.
+  to Python code; an _ast module was added.
 
 - SF bug #1167751: fix incorrect code being for generator expressions.
   The following code now raises a SyntaxError:  foo(a = i for i in range(10))

Modules/config.c.in

@@ -23,6 +23,7 @@ redistribution of this file, and for a DISCLAIMER OF ALL WARRANTIES.
 extern void PyMarshal_Init(void);
 extern void initimp(void);
 extern void initgc(void);
+extern void init_ast(void);
 
 struct _inittab _PyImport_Inittab[] = {
 
@@ -34,6 +35,9 @@ struct _inittab _PyImport_Inittab[] = {
 	/* This lives in import.c */
 	{"imp", initimp},
 
+	/* This lives in Python/Python-ast.c */
+	{"_ast", init_ast},
+
 	/* These entries are here for sys.builtin_module_names */
 	{"__main__", NULL},
 	{"__builtin__", NULL},

PC/config.c

@@ -67,6 +67,7 @@ extern void init_codecs_kr(void);
 extern void init_codecs_tw(void);
 extern void init_subprocess(void);
 extern void init_lsprof(void);
+extern void init_ast(void);
 
 /* tools/freeze/makeconfig.py marker for additional "extern" */
 /* -- ADDMODULE MARKER 1 -- */
@@ -77,6 +78,7 @@ extern void initimp(void);
 struct _inittab _PyImport_Inittab[] = {
 
         {"array", initarray},
+	{"_ast", init_ast},
 #ifdef MS_WINDOWS
 #ifndef MS_WIN64
         {"audioop", initaudioop},