equivalent[*] check that doesn't produce compiler warnings about a 'x < 0'
check on an unsigned type.

[*] it's equivalent for inputs of type size_t or Py_ssize_t, or any smaller
unsigned or signed integer type.

The debug memory api now keeps track of which external API (PyMem_* or PyObject_*) was used to allocate each block and treats any API violation as an error.  Added separate _PyMem_DebugMalloc functions for the Py_Mem API instead of having it use the _PyObject_DebugMalloc functions.

was not always being done properly in some python types and extension
modules.  PyMem_MALLOC, PyMem_REALLOC, PyMem_NEW and PyMem_RESIZE have
all been updated to perform better checks and places in the code that
would previously leak memory on the error path when such an allocation
failed have been fixed.

 Added checks for integer overflows, contributed by Google. Some are
 only available if asserts are left in the code, in cases where they
 can't be triggered from Python code.

http://mail.python.org/pipermail/python-dev/2007-May/072896.html

objimpl.h, pymem.h:  Stop mapping PyMem_{Del, DEL} and PyMem_{Free, FREE}
to PyObject_{Free, FREE} in a release build.  They're aliases for the
system free() now.

_subprocess.c/sp_handle_dealloc():  Since the memory was originally
obtained via PyObject_NEW, it must be released via PyObject_FREE (or
_DEL).

pythonrun.c, tokenizer.c, parsermodule.c:  I lost count of the number of
PyObject vs PyMem mismatches in these -- it's like the specific
function called at each site was picked at random, sometimes even with
memory obtained via PyMem getting released via PyObject.  Changed most
to use PyObject uniformly, since the blobs allocated are predictably
small in most cases, and obmalloc is generally faster than system
mallocs then.

If extension modules in real life prove as sloppy as Python's front
end, we'll have to revert the objimpl.h + pymem.h part of this patch.
Note that no problems will show up in a debug build (all calls still go
thru obmalloc then). Problems will show up only in a release build, most
likely segfaults.

Thanks to Skip Montanaro and Kalle Svensson for the patches.

As threatened, PyMem_{Free, FREE} also invoke the object deallocator now
when pymalloc is enabled (well, it does when pymalloc isn't enabled too,
but in that case "the object deallocator" is plain free()).

This is maximally backward-compatible, but it leaves a bitter aftertaste.

Also massive reworking of comments.

+ Redirect PyMem_{Del, DEL} to the object allocator's free() when
  pymalloc is enabled.  Needed so old extensions can continue to
  mix PyObject_New with PyMem_DEL.

+ This implies that pgen needs to be able to see the PyObject_XYZ
  declarations too.  pgenheaders.h now includes Python.h.  An
  implication is that I expect obmalloc.o needs to get linked into
  pgen on non-Windows boxes.

+ When PYMALLOC_DEBUG is defined, *all* Py memory API functions
  now funnel through the debug allocator wrapper around pymalloc.
  This is the default in a debug build.

+ That caused compile.c to fail:  it indirectly mixed PyMem_Malloc
  with raw platform free() in one place.  This is verbotten.

PyMem_{Del, DEL} doesn't work yet (compilation problems).

pyport.h:  _PyMem_EXTRA is gone.

pmem.h:  Repaired comments.  PyMem_{Malloc, MALLOC} and
PyMem_{Realloc, REALLOC} now make the same x-platform guarantees when
asking for 0 bytes, and when passing a NULL pointer to the latter.

object.c:  PyMem_{Malloc, Realloc} just call their macro versions
now, since the latter take care of the x-platform 0 and NULL stuff
by themselves now.

pypcre.c, grow_stack():  So sue me.  On two lines, this called
PyMem_RESIZE to grow a "const" area.  It's not legit to realloc a
const area, so the compiler warned given the new expansion of
PyMem_RESIZE.  It would have gotten the same warning before if it
had used PyMem_Resize() instead; the older macro version, but not the
function version, silently cast away the constness.  IMO that was a wrong
thing to do, and the docs say the macro versions of PyMem_xyz are
deprecated anyway.  If somebody else is resizing const areas with the
macro spelling, they'll get a warning when they recompile now too.

it's enabled.

This displays stats about the # of arenas, pools, blocks and bytes, to
stderr, both used and reserved but unused.

CAUTION:  Because PYMALLOC_DEBUG is on, the debug malloc routine adds
16 bytes to each request.  This makes each block appear two size classes
higher than it would be if PYMALLOC_DEBUG weren't on.

So far, playing with this confirms the obvious:  there's a lot of activity
in the "small dict" size class, but nothing in the core makes any use of
the 8-byte or 16-byte classes.

fast, and just cluttered the code.  Get rid of it for now.  If a compelling
case can be made for it, easy to restore it later.

When WITH_PYMALLOC is defined, define PYMALLOC_DEBUG to enable the debug
allocator.  This can be done independent of build type (release or debug).
A debug build automatically defines PYMALLOC_DEBUG when pymalloc is
enabled.  It's a detected error to define PYMALLOC_DEBUG when pymalloc
isn't enabled.

Two debugging entry points defined only under PYMALLOC_DEBUG:

+ _PyMalloc_DebugCheckAddress(const void *p) can be used (e.g., from gdb)
  to sanity-check a memory block obtained from pymalloc.  It sprays
  info to stderr (see next) and dies via Py_FatalError if the block is
  detectably damaged.

+ _PyMalloc_DebugDumpAddress(const void *p) can be used to spray info
  about a debug memory block to stderr.

A tiny start at implementing "API family" checks isn't good for
anything yet.

_PyMalloc_DebugRealloc() has been optimized to do little when the new
size is <= old size.  However, if the new size is larger, it really
can't call the underlying realloc() routine without either violating its
contract, or knowing something non-trivial about how the underlying
realloc() works.  A memcpy is always done in this case.

This was a disaster for (and only) one of the std tests:  test_bufio
creates single text file lines up to a million characters long.  On
Windows, fileobject.c's get_line() uses the horridly funky
getline_via_fgets(), which keeps growing and growing a string object
hoping to find a newline.  It grew the string object 1000 bytes each
time, so for a million-character string it took approximately forever
(I gave up after a few minutes).

So, also:

fileobject.c, getline_via_fgets():  When a single line is outrageously
long, grow the string object at a mildly exponential rate, instead of
just 1000 bytes at a time.

That's enough so that a debug-build test_bufio finishes in about 5 seconds
on my Win98SE box.  I'm curious to try this on Win2K, because it has very
different memory behavior than Win9X, and test_bufio always took a factor
of 10 longer to complete on Win2K.  It *could* be that the endless
reallocs were simply killing it on Win2K even in the release build.

Windows some modules are considered (by me, and I don't care what anyone
else thinks about this <wink>) to be part of "the core" despite that they
happen to be compiled into separate DLLs (the "to DLL or not to DLL?"
question on Windows is nearly arbitrary).  Making the pymalloc entry
points available to them allows the Windows build to complete without
incident when WITH_PYMALLOC is #define'd.

Note that this isn't unprecedented.  Other "private API" functions we
export include _PySequence_IterSearch, _PyEval_SliceIndex, _PyCodec_Lookup,
_Py_ZeroStruct, _Py_TrueStruct, _PyLong_New and _PyModule_Clear.

malloc() and free() don't change.

platform realloc(p, 0) returns NULL, so MALLOC_ZERO_RETURNS_NULL can
be correctly undefined yet realloc(p, 0) can return NULL anyway.

Prevent realloc(p, 0) doing free(p) and returning NULL via a different
hack.  Would probably be better to get rid of MALLOC_ZERO_RETURNS_NULL
entirely.

Bugfix candidate.

This should match the situation in the 1.6b1 tree.

marked my*.h as obsolete

Leave the actual #define in for API compatibility.

allocator.

For more comments, read the patches@python.org archives.
For documentation read the comments in mymalloc.h and objimpl.h.

(This is not exactly what Vladimir posted to the patches list; I've
made a few changes, and Vladimir sent me a fix in private email for a
problem that only occurs in debug mode.  I'm also holding back on his
change to main.c, which seems unnecessary to me.)

#ifdef though, so if you still need these for a really backwards
compiler you know what to do.

realloc(p, 0) returns NULL.  See changes to configure script to be
checked in later.

unique set of ideosyncracies:-(