gh-90905: Allow cross-compilation on macOS

[zanieb]

Adds support for cross-compiling x86_64 from arm64 macOS. This is particularly useful due to the decreasing availability of x86_64 machines to perform builds on.

The changes here are loosely based on the patch that python-build-standalone uses for cross-compiling.

I tested compiling x86_64 from an arm64 machine with the following script:

#!/usr/bin/env bash

set -ex

ROOT=$(pwd)
OUT=${ROOT}/../cpython-out

# Assume that x86_64 Homebrew is installed
ibrew() { arch -x86_64 /usr/local/bin/brew "$@"; }
export -f ibrew

mkdir -p ${OUT}

# Compile the "build" aarch64 Python

make clean
rm -rf ${OUT}/cpython-aarch64

# Exclude modules we don't need that require system dependencies
./configure \
    py_cv_module__openssl=n/a \
    py_cv_module__hashlib=n/a \
    py_cv_module__gdbm=n/a \
    --without-ensurepip \
    --prefix ${OUT}/cpython-aarch64

make -j8
make -j sharedinstall
make -j install

# Build the "host" x86_64 Python

make clean
rm -rf ${OUT}/cpython-x86_64

CFLAGS="-arch x86_64" \
LDFLAGS="-arch x86_64" \
MACOSX_DEPLOYMENT_TARGET=10.15 \
GDBM_CFLAGS="-I$(ibrew --prefix gdbm)/include" \
GDBM_LIBS="-L$(ibrew --prefix gdbm)/lib -lgdbm" \
PKG_CONFIG="$(ibrew --prefix pkg-config)/bin/pkg-config" \
./configure \
    --with-pydebug \
    --with-system-libmpdec \
    --with-openssl="$(ibrew --prefix openssl@3)" \
    --build=aarch64-apple-darwin \
    --host=x86_64-apple-darwin \
    --prefix ${OUT}/cpython-x86_64 \
    --with-build-python=${OUT}/cpython-aarch64/bin/python3 \
    --disable-ipv6 \
    ac_cv_file__dev_ptc=no \
    ac_cv_file__dev_ptmx=no \
    ac_cv_func_sendfile=no

make -j8
make -j sharedinstall
make -j install

${OUT}/cpython-x86_64/bin/python3 -c "import platform; print(platform.machine())"

• Issue: Support cross compilation on macOS #90905

[zanieb]

This seems a bit too easy. I'm curious if I'm missing something?

The justification for the configure flags are as follows:

--disable-ipv6

configure: error: You must get working getaddrinfo() function or pass the "--disable-ipv6" option to configure.

ac_cv_file__dev_ptmx=no

checking for /dev/ptmx... not set
configure: error: set ac_cv_file__dev_ptmx to yes/no in your CONFIG_SITE file when cross compiling

ac_cv_file__dev_ptc=no

checking for /dev/ptc... not set
configure: error: set ac_cv_file__dev_ptc to yes/no in your CONFIG_SITE file when cross compiling

ac_cv_func_sendfile=no

./Modules/posixmodule.c:11871:15: error: call to undeclared function 'sendfile'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    ret = sendfile(in_fd, out_fd, offset, &sbytes, &sf, flags);
            ^

It's unclear to me if it's expected that the -arch needs to be added to CFLAGS and LDFLAGS explicitly.

I also checked the Platform in the sysconfig, as I know we have an extra patch for this in python-build-standalone but it doesn't look necessary here:

❯ ../cpython-out/cpython-x86_64/bin/python3 -m sysconfig | grep Platform
Platform: "macosx-10.15-x86_64"

[corona10]

I think that @ned-deily and @ronaldoussoren have interests with this patch.

[FFY00]

The change looks correct to me.

I'll approve, but give @ned-deily some time to look it over before merging. @zanieb if this doesn't happen within a week, please ping me to move it forward. Thanks!

[ned-deily]

Thanks for the PR. The change looks OK so I'm merging it for 3.14.

But, IMO, I wouldn't recommend using this approach for most uses. There is a reason that we haven't bothered to support the "traditional" style of cross-building for macOS builds and that is because essentially the same result can be accomplished through the use of macOS universal builds, something that has been around in Python for many years and which is robust and well-tested. For example, the current universal2 flavor of universal builds produces fat builds including both arm64 and x86_64 architectures that can be built on either kind of Mac and will then run natively on both archs. The main differences are that the installed binary files are larger, likely negligible in most cases although the resultant binaries could be lipo-ed into single arch binaries, and it is necessary to build with universal builds of the third-party libraries used by the standard library that are not provided by macOS (i.e. openssl, xz, gdbm, tcl-tk, mpdecimal). AFAIK, Homebrew does not support universal builds of those libs; MacPorts does. We have had discussions about providing pre-built universal macOS and iOS binaries of at least some of these libs; that may yet happen in the coming months.

In the meantime, it might be a very useful interim experiment for someone to try to provide a script that lipo's and, as needed, install_name_tool's the Homebrew arm64 (/opt/homebrew) and x86_64 (/usr/local) versions of those few libs we need so that they could be cached and used in a GHA CI job to add macOS universal testing.

[ned-deily]

As an example, a very simple universal build rough equivalent:

make clean || true

# If you need GDBM, uncomment the next two lines for MacPorts and/or edit.
# GDBM_CFLAGS="-I$(dirname $(dirname $(which port)))/include" \
# GDBM_LIBS="-L$(dirname $(dirname $(which port)))/lib -lgdbm" \
MACOSX_DEPLOYMENT_TARGET=10.15 \
./configure \
    --with-pydebug \
    --enable-universalsdk \
    --with-universal-archs=universal2 \
    --with-system-libmpdec \
    --prefix ${OUT}/cpython-x86_64 \

make -j8
make -j install

${OUT}/cpython-x86_64/bin/python3  -c "import platform; print(platform.machine())"
${OUT}/cpython-x86_64/bin/python3-intel64  -c "import platform; print(platform.machine())"

And, the one-time commands to build and install the universal libs from MacPorts:

#!/bin/sh

set -ex

# Assume that MacPorts base is installed and that "port" is on $PATH
#   (https://www.macports.org/install.php)

# Update ports info
sudo port selfupdate

# Install Python dependencies
sudo port -N install \
    pkgconfig \
    openssl +universal \
    xz +universal \
    gdbm +universal \
    mpdecimal +universal \
    # Tk disabled at the moment due to open MacPorts issue with univeral variant:
    # https://trac.macports.org/ticket/71415
    # tk +quartz +universal

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.x has failed when building commit 6ecb620.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/345/builds/9923) and take a look at the build logs.
4. Check if the failure is related to this commit (6ecb620) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/345/builds/9923

Failed tests:

• test.test_multiprocessing_forkserver.test_misc

Failed subtests:

• test_large_pool - test.test_multiprocessing_forkserver.test_misc.MiscTestCase.test_large_pool

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 6607, in test_large_pool
    rc, out, err = script_helper.assert_python_ok(testfn)
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 182, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 167, in _assert_python
    res.fail(cmd_line)
    ~~~~~~~~^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 80, in fail
    raise AssertionError(f"Process return code is {exitcode}\n"
    ...<10 lines>...
                         f"---")
AssertionError: Process return code is 1
command line: ['/root/buildarea/3.x.angelico-debian-amd64/build/python', '-X', 'faulthandler', '-I', '@test_3507502_tmpæ']


Traceback (most recent call last):
  File "<string>", line 1, in <module>
    from multiprocessing.forkserver import main; main(10, 11, ['__main__'], **{'sys_path': ['/root/buildarea/3.x.angelico-debian-amd64/build/target/lib/python314.zip', '/root/buildarea/3.x.angelico-debian-amd64/build/Lib', '/root/buildarea/3.x.angelico-debian-amd64/build/build/lib.linux-x86_64-3.14'], 'authkey_r': 13})
                                                 ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line 324, in main
    pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/build/test_python_3517661æ/@test_3517661_tmpæ", line 4, in <module>
    with multiprocessing.Pool(200) as p:
         ~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
                context=self.get_context())
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 215, in __init__
    self._repopulate_pool()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 306, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
                                        self._processes,
                                        ^^^^^^^^^^^^^^^^
    ...<3 lines>...
                                        self._maxtasksperchild,
                                        ^^^^^^^^^^^^^^^^^^^^^^^
                                        self._wrap_exception)
                                        ^^^^^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 329, in _repopulate_pool_static
    w.start()
    ~~~~~~~^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ~~~~~~~~~~~^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line 300, in _Popen
    return Popen(process_obj)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
    ~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line 59, in _launch
    self.pid = forkserver.read_signed(self.sentinel)
               ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line 390, in read_signed
    raise EOFError('unexpected EOF')
EOFError: unexpected EOF
---


Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 6607, in test_large_pool
    rc, out, err = script_helper.assert_python_ok(testfn)
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 182, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 167, in _assert_python
    res.fail(cmd_line)
    ~~~~~~~~^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 80, in fail
    raise AssertionError(f"Process return code is {exitcode}\n"
    ...<10 lines>...
                         f"---")
AssertionError: Process return code is 1
command line: ['/root/buildarea/3.x.angelico-debian-amd64/build/python', '-X', 'faulthandler', '-I', '@test_3517661_tmpæ']


Traceback (most recent call last):
  File "<string>", line 1, in <module>
    from multiprocessing.forkserver import main; main(10, 11, ['__main__'], **{'sys_path': ['/root/buildarea/3.x.angelico-debian-amd64/build/target/lib/python314.zip', '/root/buildarea/3.x.angelico-debian-amd64/build/Lib', '/root/buildarea/3.x.angelico-debian-amd64/build/build/lib.linux-x86_64-3.14'], 'authkey_r': 13})
                                                 ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line 324, in main
    pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/build/test_python_3507502æ/@test_3507502_tmpæ", line 4, in <module>
    with multiprocessing.Pool(200) as p:
         ~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
                context=self.get_context())
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 215, in __init__
    self._repopulate_pool()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 306, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
                                        self._processes,
                                        ^^^^^^^^^^^^^^^^
    ...<3 lines>...
                                        self._maxtasksperchild,
                                        ^^^^^^^^^^^^^^^^^^^^^^^
                                        self._wrap_exception)
                                        ^^^^^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 329, in _repopulate_pool_static
    w.start()
    ~~~~~~~^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ~~~~~~~~~~~^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line 300, in _Popen
    return Popen(process_obj)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
    ~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line 59, in _launch
    self.pid = forkserver.read_signed(self.sentinel)
               ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line 390, in read_signed
    raise EOFError('unexpected EOF')
EOFError: unexpected EOF
---

[zanieb]

Thanks for the explanation @ned-deily — there is some discussion on universal builds downstream astral-sh/python-build-standalone#140