Issue #21141: The Windows build process no longer attempts to find Perl,

instead relying on OpenSSL source being configured and ready to build.  The
``PCbuild\build_ssl.py`` script has been re-written and re-named to
``PCbuild\prepare_ssl.py``, and takes care of configuring OpenSSL source
for both 32 and 64 bit platforms.  OpenSSL sources obtained from
svn.python.org will always be pre-configured and ready to build.

Misc/NEWS

@@ -354,6 +354,13 @@ IDLE
 Build
 -----
 
+- Issue #21141: The Windows build process no longer attempts to find Perl,
+  instead relying on OpenSSL source being configured and ready to build.  The
+  ``PCbuild\build_ssl.py`` script has been re-written and re-named to
+  ``PCbuild\prepare_ssl.py``, and takes care of configuring OpenSSL source
+  for both 32 and 64 bit platforms.  OpenSSL sources obtained from
+  svn.python.org will always be pre-configured and ready to build.
+
 - Issue #21037: Add a build option to enable AddressSanitizer support.
 
 - The Windows build now includes OpenSSL 1.0.1g

PCbuild/build_ssl.bat → PCbuild/prepare_ssl.bat

 @echo off
 if not defined HOST_PYTHON (
   if %1 EQU Debug (
+    shift
     set HOST_PYTHON=python_d.exe
     if not exist python35_d.dll exit 1
   ) ELSE (
@@ -8,5 +9,4 @@ if not defined HOST_PYTHON (
     if not exist python35.dll exit 1
   )
 )
-%HOST_PYTHON% build_ssl.py %1 %2 %3
-
+%HOST_PYTHON% prepare_ssl.py %1

PCbuild/build_ssl.py → PCbuild/prepare_ssl.py

-# Script for building the _ssl and _hashlib modules for Windows.
-# Uses Perl to setup the OpenSSL environment correctly
-# and build OpenSSL, then invokes a simple nmake session
-# for the actual _ssl.pyd and _hashlib.pyd DLLs.
+# Script for preparing OpenSSL for building on Windows.
+# Uses Perl to create nmake makefiles and otherwise prepare the way
+# for building on 32 or 64 bit platforms.
+
+# Script originally authored by Mark Hammond.
+# Major revisions by:
+#   Martin v. Löwis
+#   Christian Heimes
+#   Zachary Ware
 
 # THEORETICALLY, you can:
-# * Unpack the latest SSL release one level above your main Python source
-#   directory.  It is likely you will already find the zlib library and
-#   any other external packages there.
+# * Unpack the latest OpenSSL release where $(opensslDir) in
+#   PCbuild\pyproject.props expects it to be.
 # * Install ActivePerl and ensure it is somewhere on your path.
-# * Run this script from the PCBuild directory.
-#
-# it should configure and build SSL, then build the _ssl and _hashlib
-# Python extensions without intervention.
-
-# Modified by Christian Heimes
-# Now this script supports pre-generated makefiles and assembly files.
-# Developers don't need an installation of Perl anymore to build Python. A svn
-# checkout from our svn repository is enough.
+# * Run this script with the OpenSSL source dir as the only argument.
 #
-# In Order to create the files in the case of an update you still need Perl.
-# Run build_ssl in this order:
-# python.exe build_ssl.py Release x64
-# python.exe build_ssl.py Release Win32
+# it should configure OpenSSL such that it is ready to be built by
+# ssl.vcxproj on 32 or 64 bit platforms.
 
-import os, sys, re, shutil
+import os
+import re
+import sys
+import shutil
 
 # Find all "foo.exe" files on the PATH.
 def find_all_on_path(filename, extras = None):
@@ -63,14 +60,6 @@ def find_working_perl(perls):
     print(" Please install ActivePerl and ensure it appears on your path")
     return None
 
-# Fetch SSL directory from VC properties
-def get_ssl_dir():
-    propfile = (os.path.join(os.path.dirname(__file__), 'pyproject.props'))
-    with open(propfile) as f:
-        m = re.search('openssl-([^<]+)<', f.read())
-        return "..\..\openssl-"+m.group(1)
-
-
 def create_makefile64(makefile, m32):
     """Create and fix makefile for 64bit
 
@@ -138,24 +127,14 @@ def copy(src, dst):
         return
     shutil.copy(src, dst)
 
-def main():
-    build_all = "-a" in sys.argv
-    if sys.argv[1] == "Release":
-        debug = False
-    elif sys.argv[1] == "Debug":
-        debug = True
-    else:
-        raise ValueError(str(sys.argv))
-
-    if sys.argv[2] == "Win32":
-        arch = "x86"
+def prep(arch):
+    if arch == "x86":
         configure = "VC-WIN32"
         do_script = "ms\\do_nasm"
         makefile="ms\\nt.mak"
         m32 = makefile
         dirsuffix = "32"
-    elif sys.argv[2] == "x64":
-        arch="amd64"
+    elif arch == "amd64":
         configure = "VC-WIN64A"
         do_script = "ms\\do_win64a"
         makefile = "ms\\nt64.mak"
@@ -163,11 +142,54 @@ def main():
         dirsuffix = "64"
         #os.environ["VSEXTCOMP_USECL"] = "MS_OPTERON"
     else:
-        raise ValueError(str(sys.argv))
+        raise ValueError('Unrecognized platform: %s' % arch)
+
+    # rebuild makefile when we do the role over from 32 to 64 build
+    if arch == "amd64" and os.path.isfile(m32) and not os.path.isfile(makefile):
+        os.unlink(m32)
+
+    # If the ssl makefiles do not exist, we invoke Perl to generate them.
+    # Due to a bug in this script, the makefile sometimes ended up empty
+    # Force a regeneration if it is.
+    if not os.path.isfile(makefile) or os.path.getsize(makefile)==0:
+        print("Creating the makefiles...")
+        sys.stdout.flush()
+        run_configure(configure, do_script)
+
+        if arch == "amd64":
+            create_makefile64(makefile, m32)
+        fix_makefile(makefile)
+        copy(r"crypto\buildinf.h", r"crypto\buildinf_%s.h" % arch)
+        copy(r"crypto\opensslconf.h", r"crypto\opensslconf_%s.h" % arch)
+    else:
+        print(makefile, 'already exists!')
+
+    # If the assembler files don't exist in tmpXX, copy them there
+    if os.path.exists("asm"+dirsuffix):
+        if not os.path.exists("tmp"+dirsuffix):
+            os.mkdir("tmp"+dirsuffix)
+        for f in os.listdir("asm"+dirsuffix):
+            if not f.endswith(".asm"): continue
+            if os.path.isfile(r"tmp%s\%s" % (dirsuffix, f)): continue
+            shutil.copy(r"asm%s\%s" % (dirsuffix, f), "tmp"+dirsuffix)
+
+def main():
+    if len(sys.argv) == 1:
+        print("Not enough arguments: directory containing OpenSSL",
+              "sources must be supplied")
+        sys.exit(1)
+
+    if len(sys.argv) > 2:
+        print("Too many arguments supplied, all we need is the directory",
+              "containing OpenSSL sources")
+        sys.exit(1)
+
+    ssl_dir = sys.argv[1]
+
+    if not os.path.exists(ssl_dir) and os.path.isdir(ssl_dir):
+        print(ssl_dir, "is not an existing directory!")
+        sys.exit(1)
 
-    make_flags = ""
-    if build_all:
-        make_flags = "-a"
     # perl should be on the path, but we also look in "\perl" and "c:\\perl"
     # as "well known" locations
     perls = find_all_on_path("perl.exe", ["\\perl\\bin", "C:\\perl\\bin"])
@@ -175,79 +197,21 @@ def main():
     if perl:
         print("Found a working perl at '%s'" % (perl,))
     else:
-        print("No Perl installation was found. Existing Makefiles are used.")
-    sys.stdout.flush()
-    # Look for SSL 2 levels up from pcbuild - ie, same place zlib etc all live.
-    ssl_dir = get_ssl_dir()
-    if ssl_dir is None:
         sys.exit(1)
+    sys.stdout.flush()
+
+    # Put our working Perl at the front of our path
+    os.environ["PATH"] = os.path.dirname(perl) + \
+                                os.pathsep + \
+                                os.environ["PATH"]
 
-    old_cd = os.getcwd()
+    old_cwd = os.getcwd()
     try:
         os.chdir(ssl_dir)
-        # rebuild makefile when we do the role over from 32 to 64 build
-        if arch == "amd64" and os.path.isfile(m32) and not os.path.isfile(makefile):
-            os.unlink(m32)
-
-        # If the ssl makefiles do not exist, we invoke Perl to generate them.
-        # Due to a bug in this script, the makefile sometimes ended up empty
-        # Force a regeneration if it is.
-        if not os.path.isfile(makefile) or os.path.getsize(makefile)==0:
-            if perl is None:
-                print("Perl is required to build the makefiles!")
-                sys.exit(1)
-
-            print("Creating the makefiles...")
-            sys.stdout.flush()
-            # Put our working Perl at the front of our path
-            os.environ["PATH"] = os.path.dirname(perl) + \
-                                          os.pathsep + \
-                                          os.environ["PATH"]
-            run_configure(configure, do_script)
-            if debug:
-                print("OpenSSL debug builds aren't supported.")
-            #if arch=="x86" and debug:
-            #    # the do_masm script in openssl doesn't generate a debug
-            #    # build makefile so we generate it here:
-            #    os.system("perl util\mk1mf.pl debug "+configure+" >"+makefile)
-
-            if arch == "amd64":
-                create_makefile64(makefile, m32)
-            fix_makefile(makefile)
-            copy(r"crypto\buildinf.h", r"crypto\buildinf_%s.h" % arch)
-            copy(r"crypto\opensslconf.h", r"crypto\opensslconf_%s.h" % arch)
-
-        # If the assembler files don't exist in tmpXX, copy them there
-        if perl is None and os.path.exists("asm"+dirsuffix):
-            if not os.path.exists("tmp"+dirsuffix):
-                os.mkdir("tmp"+dirsuffix)
-            for f in os.listdir("asm"+dirsuffix):
-                if not f.endswith(".asm"): continue
-                if os.path.isfile(r"tmp%s\%s" % (dirsuffix, f)): continue
-                shutil.copy(r"asm%s\%s" % (dirsuffix, f), "tmp"+dirsuffix)
-
-        # Now run make.
-        if arch == "amd64":
-            rc = os.system("nasm -f win64 -DNEAR -Ox -g ms\\uptable.asm")
-            if rc:
-                print("nasm assembler has failed.")
-                sys.exit(rc)
-
-        copy(r"crypto\buildinf_%s.h" % arch, r"crypto\buildinf.h")
-        copy(r"crypto\opensslconf_%s.h" % arch, r"crypto\opensslconf.h")
-
-        #makeCommand = "nmake /nologo PERL=\"%s\" -f \"%s\"" %(perl, makefile)
-        makeCommand = "nmake /nologo -f \"%s\"" % makefile
-        print("Executing ssl makefiles:", makeCommand)
-        sys.stdout.flush()
-        rc = os.system(makeCommand)
-        if rc:
-            print("Executing "+makefile+" failed")
-            print(rc)
-            sys.exit(rc)
+        for arch in ['amd64', 'x86']:
+            prep(arch)
     finally:
-        os.chdir(old_cd)
-    sys.exit(rc)
+        os.chdir(old_cwd)
 
 if __name__=='__main__':
     main()

PCbuild/readme.txt

@@ -181,30 +181,19 @@ _ssl
     you should first try to update NASM and do a full rebuild of
     OpenSSL.
 
-    If you like to use the official sources instead of the files from
-    python.org's subversion repository, Perl is required to build the
-    necessary makefiles and assembly files.  ActivePerl is available
-    from
+    The ssl sub-project expects your OpenSSL sources to have already
+    been configured and be ready to build.  If you get your sources
+    from svn.python.org as suggested in the "Getting External Sources"
+    section below, the OpenSSL source will already be ready to go.  If
+    you want to build a different version, you will need to run
+
+       PCbuild\prepare_ssl.py path\to\openssl-source-dir
+
+    That script will prepare your OpenSSL sources in the same way that
+    those available on svn.python.org have been prepared.  Note that
+    Perl must be installed and available on your PATH to configure
+    OpenSSL.  ActivePerl is recommended and is available from
         http://www.activestate.com/activeperl/
-    The svn.python.org version contains pre-built makefiles and assembly
-    files.
-
-    The build process makes sure that no patented algorithms are
-    included.  For now RC5, MDC2 and IDEA are excluded from the build.
-    You may have to manually remove $(OBJ_D)\i_*.obj from ms\nt.mak if
-    using official sources; the svn.python.org-hosted version is already
-    fixed.
-
-    The ssl.vcxproj sub-project simply invokes PCbuild/build_ssl.py,
-    which locates and builds OpenSSL.
-
-    build_ssl.py attempts to catch the most common errors (such as not
-    being able to find OpenSSL sources, or not being able to find a Perl
-    that works with OpenSSL) and give a reasonable error message.  If
-    you have a problem that doesn't seem to be handled correctly (e.g.,
-    you know you have ActivePerl but we can't find it), please take a
-    peek at build_ssl.py and suggest patches.  Note that build_ssl.py
-    should be able to be run directly from the command-line.
 
     The ssl sub-project does not have the ability to clean the OpenSSL
     build; if you need to rebuild, you'll have to clean it by hand.