# Building Mac Python from source

This document explains how to build MacPython from source. This is necessary if you want to write extension modules for 68K Python, and currently also probably the easiest way to build PPC extension modules. Building Python is not something to be undertaken lightly, the process is not very streamlined so you need a reasonable working knowledge of the CodeWarrior development environment, a good net connection and probably quite some time too.

The information density in this file is high, so you should probably print it and read it at your leasure. Most things are explained only once (and probably in the wrong place:-).

I am very interested in feedback on this document, contact me at <jack@cwi.nl> or send your comments to the Mac Python Special Interest Group.

## What you need.

The following things you definitely need:

- You need a MacPython source distribution, of course. You can obtain one from ftp://ftp.cwi.nl/pub/jack/python/mac or from the companion webpage at http://www.cwi.nl/~jack/macpython.html (which has up-to-date links to the other packages needed too) and possibly also from the standard python.org ftp site. Everything you need is also included in the standard Python source distribution, but the organization is different. Look in directory `Mac/mwerks/projects` for the project files and related stuff.
- You need MetroWerks CodeWarrior. The current distribution has been built with version 11 of CodeWarrior. Ordering information is available on the MetroWerks homepage. You might still be able to build Python with MPW or Think/Symantec C but you are basically on your own.
- You need GUSI, the Grand Unified Socket Interface, by Matthias Neeracher. The original CWGUSI is obtainable from ftp://sunsite.cnlab-switch.ch/software/platform/macos/src, but at the moment Python is built with a rather hacked-up version of GUSI, available from Jack's Python webpage and ftp-directory referenced above. It is also possible to build a non-GUSI Python, see below.

The MacPython project files are configured to include a plethora of optional modules, and these modules need a number extra packages. To use the project files as-is you have to download these packages too. PPC and CFM68K Python have all such modules as dynamically loaded modules, so if you don't need a certain package it suffices to just refrain from builing the extension module. For static 68K Python things are a bit more complicated: you have to edit the interpreter project file to remove the reference to the module (and the libraries it uses), and edit the `Mac:mwerks:mwerks_nonshared_config.h` file to remove the `USE_...` line. Here are the locations for the various things you need:

- Tcl and Tk can be obtained from ftp://ftp.smli.com/pub/tcl/mac/. The current distributions, Tcl 7.6 and Tk 4.2 need a bit of work, see the section on building Tcl/Tk Python below. Get the "full source" distribution, which includes MoreFiles.
- Waste, a TextEdit replacement written by Marco Piovanelli, <piovanel@kagi.com>. Python was built using version 1.2, which you can obtain from <ftp://rhino.harvard.edu/pub/dan/WASTE> and various other places.
- Gdbm library for the Mac. Available from Jack's Mac software page at http://www.cwi.nl/~jack/macsoftware.html and ftp://ftp.cwi.nl/pub/jack/mac.
- JPEG library by the Independent JPEG Group. A version including Mac projects can be found at Jack's page mentioned above. The most recent JPEG library can always be obtained from ftp://ftp.uu.net/graphics/jpeg/.
- The netpbm/pbmplus, libtiff, zlib and png libraries. The netpbm distribution (which includes libtiff) is generally available on Internet ftp servers. For Python pbmplus, an older incarnation of netpbm, is functionally identical to netpbm, since Python only uses the library and not the complete applications. A distribution with correct projects and library source only is available from, you guessed it, Jack's Mac software page mentioned above.

## Setting Up

Now that you have collected everything you should start with building the various parts. If you don't want to fix access paths try to set things up as follows:

    Top-level-folder:
    	CWGUSI
    	imglibs
    		libjpeg
    		pbmplus
    		libtiff
    		zlib
    		libpng
    	gdbm
    	MoreFiles 1.4.3 (not needed by Python, only by tcl/tk)
    	Python
    	Tcl 7.6
    	Tk 4.2
    	Waste 1.2 distribution (if you want waste)

First build GUSI. If you didn't get the python-specific GUSI you have to move the files from the "CWGUSI-mods" to the right place in the CWGUSI distribution folder, and build the projects `GUSI.68K.µ`, `GUSI.CFM68K.µ` and `GUSI.PPC.µ`. Two notes:

- The projects as distributed have a slightly incorrect access path, due to which they cannot find their `GUSICWFile.???` files. Change the access path to `{project folder}::` to fix this.
- The Gusi CFM68K project is distributed without "far data": you should set this option before building.

If you have a CWGUSI version more recent than 1.8 it could be that the fixes are already included, check it. Alternatively, you can forget about the fixes and remove the one reference to GUSILoadPreferences in the Python sources. This will disable the "default file creator/type" and "Delay Sioux window" preferences in Python. Next, in `MoreFiles`, `libjpeg`, `pbmplus`, `zlib`, `libpng`, `gdbm`, and`libtiff` you build all projects. Sometimes the projects are in "mac" subfolders, sometimes they are in the main folder. Tcl/tk is a special case, see below. Of course, if you are only interested in one of static 68K, CFM68K or PPC you can skip building the other libraries.

## Building Tcl/Tk

You need to make a minor organizational change to the Tcl/Tk distribution. The current instructions are for the `tcl7.6` and `tk4.2` distribution:

- Rename the `compat` folders to `(compat)` in both the Tcl and Tk folders.
- In the Tcl folder, move `strncasecmp.c` and `tclErrno.h` from `(compat)` to the main Tcl folder.
- Fix `dnr.c` as provided by MetroWerks by inserting

       #pragma ANSI_strict off

  at the beginning. The tcl library is built with strict ANSI on, and this file uses C++ style comments.
- If you want to build `SimpleTcl` and `SimpleTk` you may have to remove the references to `libmoto` from the project, not everyone has this library.
- If you want to build for CFM68K you have to create the projects for the libraries yourself. Take the 68K library projects, set the C++ header file to "MW_TclHeaderCFM68K", the project type to "cfm68K library" and the filename to "Tcl-CFM68K.Lib" (and similarly for Tk). You may also have to modify `TclMacNotify.c` because there is an error in the Apple Universal headers (sic!). Read the comments at the beginning of `Mac:Python:macglue.c` and copy the code to `TclMacNotify.c`. If you get linker errors on `GetEvQHdr` you have not done this correctly. XXXX Is this still needed?
- Note that if you use a different release of Tcl and Tk than the ones I have used you may have to adapt the Python `tkresources.rsrc` file. This is easiest done by building `SimpleTk` and copying the TEXT, ICON and CRSR resources from it to `tkresources.rsrc`. This allows the `_tkinter` module to work without an installed Tk/Tcl on your machine.
- You have to adapt the projects to use the MSL libraries. This is important.

Build first the Tcl library, then SimpleTcl (test it by typing `ls -l` in the window you get) then the Tk library, then SimpleTk (which can again be tested with `ls -l`). If this all worked you are all set to try building Python.

## Building Waste

You do not need to build the Waste libraries, as Python includes the source modules themselves. You have to make one modification, though. In file `ICCFMGlue.c` in folder `Minimal IC APIs`, add the following lines:

>     #include
>     #include

## The organization of the Python source tree

Time for a short break, while we have a look at the organization of the Python source tree. At the top level, we find the following folders:

build.mac68k.stand
  This is where you build static 68K interpreters.
build.mac68k.shared
  This is where you build the CFM68K shared library, interpreter and applet framework.
build.macppc.shared
  This is where you build the PPC shared library, interpreter and applet framework. You can also build the fat applet framework here.
build.macppc.stand
  This is where you build a nonshared PPC interpreter (optional).
Demo
  Demo programs that are not Mac-specific. Some of these may not work, the file `README-Mac` has some details.
Extensions
  Extensions to the interpreter that are not Mac-specific. Contains only the `img` extension in this distribution. Extensions are not built here, as they are on Unix, but incorporated in the core interpreter or built as plugin modules.
Grammar
  The Python grammar. Included for reference only, you cannot build the parser on a Mac.
Include
  Machine-independent header files.
Modules
  Machine-independent optional modules. Not all of these will work on the Mac.
Objects
  Machine-independent code for various objects. Most of these are not really optional: the interpreter will not function without them.
Parser
  The Python parser (machine-independent).
PlugIns
  This is where you build the PPC and CFM68K dynamically-loaded plugin modules.
Python
  The core interpreter. Most files are machine-independent, some are unix-specific and not used on the Mac.
Tools
  Tools for python developers. Contains `modulator` which builds skeleton C extension modules and `bgen` which generates complete interface modules from information in C header files. There are some readme files, but more documentation is sorely needed.

All the mac-specific stuff lives in the `Mac` folder:

Compat
  Unix-compatability routines. Some of these are not used anymore, since CWGUSI provides a rather complete emulation, but you may need these if you are trying to build a non-GUSI python.
Demo
  Mac-specific demo programs, some of them annotated.
Include
  Mac-specific but compiler-independent include files.
Lib
  Mac-specific standard modules. The `toolbox` folder contains modules specifically needed with various MacOS toolbox interface modules.
Modules
  Mac-specific builtin modules. Theoretically these are all optional, but some are rather essential (like `macmodule`). A lot of these modules are generated with `bgen`, in which case the bgen input files are included so you can attempt to regenerate them or extend them.
MPW
  MPW-specific files. These have not been used or kept up-to-date for a long time, so use at your own risk.
mwerks
  Mwerks-specific sources and headers. Contains glue code for Pythons shared-library architecture, a replacement for `malloc` and a directory with various projects for building variations on the Python interpreter. The `mwerks_*.h` files here are the option-setting files for the various interpreters and such, comparable to the unix command-line `-D` options to the compiler. Each project uses the correct option file as its "prefix file" in the "C/C++ language" settings. Disabling optional modules (for the 68K interpreter), building non-GUSI interpreters and various other things are accomplished by modifying these files (and possibly changing the list of files included in the project window, of course).
Python
  Mac-specific parts of the core interpreter.
Resources
  Resource files needed to build the interpreter.
Scripts
  A collection of various mac-specific Python scripts. Some are essential, some are useful but few are documented, so you will have to use your imagination to work them out.
Unsupported
  Modules that are not supported any longer but may still work with a little effort.

## Building the 68K interpreter

If you have all the optional libraries mentioned above loaded buildin Python for 68K macs is a breeze: open the project in the folder `build.mac68k.stand` and build it. Do not run it yet, this will possibly result in a garbled preferences file.

First remove the `Python preferences` file from your preference folder, only if you had an older version of Python installed. (this is also what you do if you did not heed the last sentence of the preceeding paragraph). Next, move the interpreter to the main Python folder (up one level) and run it there. This will create a correct initial preferences file. You are now all set, and your tree should be completely compatible with a binary-only distribution. Read the release notes (`Relnotes-somethingorother`) and `ReadMeOrSuffer` in the `Mac` folder.

## Building the CFM68K interpreter

Building the CFM68K interpreter is as almost exactly the same as building the PPC interpreter, with the exception that you should read "CFM68K" for "PPC" every time. Continue reading with the next section.

## Building the PPC interpreter

First you build the interpreter, core library and applet skeleton in folder `build.macppc.stand`. The order to build things is the following:

PythonCorePPC
  The shared library that contains the bulk of the interpreter and its resources. It is a good idea to immedeately put an alias to this shared library in the `Extensions` folder of your system folder. Do exactly that: put an alias there, copying or moving the file will cause you grief later.
PythonPPC
  The interpreter. This is basically a routine to call out to the shared library. Do not run it yet, this will possibly result in a garbled preferences file. See the section below on rebuilding .exp files if you get funny linker errors.

PythonAppletPPC
  The applet skeleton application. Very similar to `PythonPPC`, but it calls to a different entrypoint in the core library. The `mkapplet` script will copy this complete file, and add a `'PYC '` with the module to generate an applet.

After creating the alias to `PythonCorePPC` you should move `PythonPPC` to the main Python folder. Next you remove any old `Python XXX Preferences` file from the `Preferences` folder (if you had python installed on your system before) and run the interpreter once to create the correct preferences file. You should also make an alias `PythonApplet` pointing to `PythonAppletPPC` in the main Python folder. (again: making an alias is preferrable to copying or moving the file, since this will cause the correct file to be used if you ever rebuild PythonAppletPPC).

Next, you have to build the extension modules in the `PlugIns` folder. Open each project with `.ppc` in the name and build it. After all the dynamically loaded modules are built you have to create a number of aliases: some modules live together in a single dynamic library. Run the `MkPluginAliases.py` script from `Mac:scripts` to create the aliases.

Finally, you must build the standard applets: `EditPythonPrefs`, `mkapplet`, etc. This is easiest done with the `fullbuild` script from `Mac:scripts`. Answer no to all questions except when it asks whether to build the applets.

> Actually, the `fullbuild` script can be used to build everything, but you need a fully-functional interpreter before you can use it (and one that isn't rebuilt in the process: you cannot rebuild a running program). You could copy the 68K interpreter to a different place and use that to run fullbuild, or use the standalone PPC python for this. I tend to keep a standalone interpreter in a safe place for this use only.

You are all set now, and should read the release notes and `ReadMeOrSuffer` file from the `Mac` folder.

## Rebuilding `.exp` files for PPC and CFM68K

Occasionally it may be necessary to rebuild your PythonCore `.exp` file, a file that controls which symbols are exported by your PythonCore shared library. Rebuild it if you get unexpected undefined symbols when you are building a plugin module.

Rebuilding the .exp file is done by first removing the file and removing the reference to it in the project (in the "config" section). Next, build PythonCore. This will create a new .exp file. Edit this file to remove the references to the symbols `__initialize`, `__terminate`, `setjmp`, `longjmp`, `main` and (for PPC) `__ptmf_null` or (for CFM68K) `__start` and `dummy_init_routine`. Next, add the .exp file to the project again and rebuild PythonCore.

This rather convoluted procedure is needed to ensure that plugin modules don't accidentally link with those entrypoints from PythonCore, which will not work because those routines have to be in the same code fragment as they are used from.

## Odds and ends

Some remarks that I could not fit in elsewhere:

- It may be possible to use the `PythonCore` shared library to embed Python in another program, if your program can live with using GUSI for I/O. Use PythonCore in stead of your MSL C library (or, at the very least, link it before the normal C library). Let me know whether this works.
- It is possible to build PPC extension modules without building a complete Python. Take the binary distribution, add folders `Include`, `Mac:Include` and `Mac:mwerks` from the source distribution and you should be all set. A template for a dynamic module can be found in `xx.ppc.µ` or `xx.CFM68K.µ`.
- The Python shared library architecture is a variant of the architecture described as "application with shared libraries and dropins" in the MetroWerks "Targeting MacOS" documentation. The Python Application and applet-template use the `MSL AppRuntime.Lib` runtime library (with properly set CFM initialization and termination routines). PythonCore uses `MSL Runtime.Lib`, which is really intended for standalone programs but which we fool into working by providing a dummy main program. It is linked statically into PythonCore (and exported to the applications and plugins) so we do not have to distribute yet another shared library. Plugin modules use `MSL ShlibRuntime.Lib` (not the dropin runtime: modules are never unloaded) and obtain the rest from PythonCore. PythonCore uses a non-standard initialization entry point, `__initialize_with_resources`, to be able to obtain resources from the library file later on. Plugins can do the same (_tkinter does) or use the standard `__initialize` entry point.