indygreg/python-build-standalone 20230114

on GitHub

THIS RELEASE MAY BE DELETED. ARTIFACTS MAY BE MUTATED AT ANY TIME. DO NOT USE THIS RELEASE BEYOND SHORT TERM TESTING UNLESS THIS MESSAGE IS REMOVED.

This is a major release compared to recent releases due to a number of changes in order to support Python 3.11. Extended testing of this release during upgrades is encouraged.

• Added CPython 3.11 support and distributions for all supported platforms.
• macOS and non-musl i686 and x86-64 Linux distributions now compiled with LLVM/Clang 15.0.7 instead of 14.0.3.
• 3.8 and 3.9 Linux distributions now always link against libedit (02e8695). Before, these 3.8 and 3.9 distributions had their _readline extension module linked against readline by default. The full distributions contained a compiled extension variant linked against libedit.
• The _gdbm extension module is now globally disabled (8028229). Before, it was enabled inconsistently on various distributions.
• As a consequence of the above 2 changes, all Python distributions are no longer GPL encumbered. Before, the 3.8 and 3.9 Linux and macOS distributions were GPL encumbered due to a dependency on readline and possibly gdbm.
• The definition of extension modules on Linux and macOS distributions is now centrally defined in a YAML file. All extension build instructions and metadata for PYTHON.json is now derived from this file. This teased out subtle inconsistencies with how extension modules are built. Some of these are documented below. Some changes may have eluded the project maintainer's watchful eye. Be on the lookout for subtle changes in extension modules.
• _hashlib metadata in PYTHON.json no longer advertises a dependency on libssl for some distributions.
• There is a possibility that _zlib linking behavior changed on some distributions as a result of porting extension module metadata to YAML. (89e0952)
• The _testcapi extension module is now globally disabled.
• The xxlimited extension module is now globally disabled.
• --with-dbmliborder is now explicitly passed to CPython's configure. This may have impacted with dbm library is used on some distributions. But hopefully any changes won't have a meaningful impact on functionality.
• mpdecimal is now built from source (fe9fca0) and _decimal is now linked against our built libmpdec (40ca5c7). Previously, we used the mpdecimal vendored with and built by CPython.
• expat is now built from source (735402c) and the _elementtree and pyexpat extension modules now link against our built libexpat (5887123). Previously, we used the expat library vendored with and built by CPython.
• iOS builds have been removed from CI. They weren't tested. We weren't releasing them. And the CI builds were only running on CPython 3.9. If iOS (and other Apple OS) support is important to you, please file a GitHub issue so we can discuss how to support these releases.
• PY_SQLITE_ENABLE_LOAD_EXTENSION is now explicit set on all Linux and macOS distributions.
• CI now runs verification invoking the built interpreter on Linux where it can. Previously, we only performed the static Rust-based validation on all platforms and the running interpreter tests on macOS and Windows.
• All CPython patches have been factored out of cpython-unix/build-cpython.sh into standalone .patch files.
• Rust dependencies have been updated to latest versions.
• build-windows.py now accepts a --vs argument to control which Visual Studio version to use. It defaults to 2019 (the previous implicit default). But Visual Studio 2022 is known to work.
• build-windows.py now accepts a --windows-sdk-version argument to control which Windows SDK version to use. It defaults to 10.0.20348.0 (the previous implicit default). Minimal testing has been done to validate that other SDK versions (including Windows 11 SDKs) work.
• Duplicate symbol linker warnings are suppressed on Windows static builds. This greatly cuts down on build log spam.
• Validation code now ensures that all PYTHON.json referenced object files are present in the tar archive.
• Validation code now ensures that all PYTHON.json extension module initialization functions are present in binaries.
• Fixed macOS aarch64 -> x86_64 cross-compilation (#150).

CPython 3.11 Potential Areas of Concern

Supporting a new major Python release is always a fair amount of work and there are bound to be bugs in the initial release. This section calls out where we think there may be potential for issues with the 3.11 distributions.

• Extension module configuration in general. CPython 3.11 started moving a lot of logic from its setup.py into configure. This is a good idea. But the port is not yet complete and logic still spans configure and setup.py. This refactoring was rather invasive to CPython's build system and necessitated a lot of work on our end to work in the new world. Because the port to configure is not yet complete, we devised patches to effectively disable a lot of its functionality and continue leveraging our (somewhat hacky) solution of emitting config files and makefile rules explicitly instead of going through the normal mechanisms. All this means extension module building in general is more prone to regressions and unexpected changes than prior Python versions. See 0afdfa0 and commits around it for more specifics.
• _dbm extension module behavior. See 3234e7f for more info.