Subject: Buglet in PyLong_AsLong

From: "Tim Peters" <tim_one@email.msn.com>
To: "Guido van Rossum" <guido@CNRI.Reston.VA.US>
Date: Sat, 23 May 1998 21:45:53 -0400

Guido, the overflow checking in PyLong_AsLong is off a little:

1) If the C in use sign-extends right shifts on signed longs, there's a
spurious overflow error when converting the most-negative int:

Python 1.5.1 (#0, Apr 13 1998, 20:22:04) [MSC 32 bit (Intel)] on win32
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> x = -1L << 31
>>> x
-2147483648L
>>> int(x)
Traceback (innermost last):
  File "<stdin>", line 1, in ?
OverflowError: long int too long to convert
>
> 2) If C does not sign-extend, some genuine overflows won't be caught.
>
> The attached should repair both, and, because I installed a new disk and a C
> compiler today, it's even been compiled this time <wink>.
>
> Python 1.5.1 (#0, May 23 1998, 20:24:58) [MSC 32 bit (Intel)] on win32
> Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
> >>> x = -1L << 31
> >>> x
> -2147483648L
> >>> int(x)
> -2147483648
> >>> int(-x)
> Traceback (innermost last):
>   File "<stdin>", line 1, in ?
> OverflowError: long int too long to convert
> >>> int(-x-1)
> 2147483647
> >>> int(x-1)
> Traceback (innermost last):
>   File "<stdin>", line 1, in ?
> OverflowError: long int too long to convert

end-casing-ly y'rs  - tim

Objects/longobject.c

@@ -175,10 +175,11 @@ long
 PyLong_AsLong(vv)
 	PyObject *vv;
 {
+	/* This version by Tim Peters */
 	register PyLongObject *v;
-	long x, prev;
+	unsigned long x, prev;
 	int i, sign;
-	
+
 	if (vv == NULL || !PyLong_Check(vv)) {
 		PyErr_BadInternalCall();
 		return -1;
@@ -194,13 +195,22 @@ PyLong_AsLong(vv)
 	while (--i >= 0) {
 		prev = x;
 		x = (x << SHIFT) + v->ob_digit[i];
-		if ((x >> SHIFT) != prev) {
-			PyErr_SetString(PyExc_OverflowError,
-				"long int too long to convert");
-			return -1;
-		}
-	}
-	return x * sign;
+		if ((x >> SHIFT) != prev)
+			goto overflow;
+	}
+	/* Haven't lost any bits, but if the sign bit is set we're in
+	 * trouble *unless* this is the min negative number.  So,
+	 * trouble iff sign bit set && (positive || some bit set other
+	 * than the sign bit).
+	 */
+	if ((long)x < 0 && (sign > 0 || (x << 1) != 0))
+		goto overflow;
+	return (long)x * sign;
+
+ overflow:
+	PyErr_SetString(PyExc_OverflowError,
+			"long int too long to convert");
+	return -1;
 }
 
 /* Get a C long int from a long int object.