Build issue, file or directory not found

[kwmlodozeniec]

When I try to build python locally (using docker, which seems the default), I get the following error and the build fails.

FileNotFoundError: [Errno 2] No such file or directory: '/MY_HOME_DIR/python-build-standalone/build/image-gcc.linux_x86_64'

I build in WSL2, but I have also tried building on a "real" Linux host and got the same result. This used to work for me in the past, so I'm at a loss as to what has happened.

Full log:

./build-linux.py --python cpython-3.10
Collecting attrs==24.3.0 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 3))
  Using cached attrs-24.3.0-py3-none-any.whl (63 kB)
Collecting certifi==2024.12.14 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 9))
  Using cached certifi-2024.12.14-py3-none-any.whl (164 kB)
Collecting charset-normalizer==3.4.1 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 13))
  Using cached charset_normalizer-3.4.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)
Collecting docker==7.1.0 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 107))
  Using cached docker-7.1.0-py3-none-any.whl (147 kB)
Collecting idna==3.10 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 111))
  Using cached idna-3.10-py3-none-any.whl (70 kB)
Collecting jinja2==3.1.5 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 115))
  Using cached jinja2-3.1.5-py3-none-any.whl (134 kB)
Collecting jsonschema==4.23.0 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 119))
  Using cached jsonschema-4.23.0-py3-none-any.whl (88 kB)
Collecting jsonschema-specifications==2024.10.1 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 123))
  Using cached jsonschema_specifications-2024.10.1-py3-none-any.whl (18 kB)
Collecting markupsafe==3.0.2 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 127))
  Using cached MarkupSafe-3.0.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23 kB)
Collecting pyyaml==6.0.2 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 190))
  Using cached PyYAML-6.0.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (759 kB)
Collecting referencing==0.35.1 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 245))
  Using cached referencing-0.35.1-py3-none-any.whl (26 kB)
Collecting requests==2.32.3 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 251))
  Using cached requests-2.32.3-py3-none-any.whl (64 kB)
Collecting rpds-py==0.22.3 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 255))
  Using cached rpds_py-0.22.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (385 kB)
Collecting six==1.17.0 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 362))
  Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)
Collecting tomli==2.2.1 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 366))
  Using cached tomli-2.2.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (241 kB)
Collecting urllib3==2.3.0 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 400))
  Using cached urllib3-2.3.0-py3-none-any.whl (128 kB)
Collecting zstandard==0.23.0 (from -r /MY_HOME_DIR/python-build-standalone/requirements.txt (line 406))
  Using cached zstandard-0.23.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB)
Installing collected packages: zstandard, urllib3, tomli, six, rpds-py, pyyaml, markupsafe, idna, charset-normalizer, certifi, attrs, requests, referencing, jinja2, jsonschema-specifications, docker, jsonschema
Successfully installed attrs-24.3.0 certifi-2024.12.14 charset-normalizer-3.4.1 docker-7.1.0 idna-3.10 jinja2-3.1.5 jsonschema-4.23.0 jsonschema-specifications-2024.10.1 markupsafe-3.0.2 pyyaml-6.0.2 referencing-0.35.1 requests-2.32.3 rpds-py-0.22.3 six-1.17.0 tomli-2.2.1 urllib3-2.3.0 zstandard-0.23.0
TARGET_TRIPLE=x86_64-unknown-linux-gnu
HOST_PLATFORM=linux_x86_64
PACKAGE_SUFFIX=x86_64-unknown-linux-gnu-noopt
/MY_HOME_DIR/python-build-standalone/cpython-unix/build.py --host-platform linux_x86_64 --target-triple x86_64-unknown-linux-gnu --options noopt --python-source null --dest-archive /MY_HOME_DIR/python-build-standalone/build/image-build.linux_x86_64.tar  --toolchain image-build
/MY_HOME_DIR/python-build-standalone/cpython-unix/build.py --host-platform linux_x86_64 --target-triple x86_64-unknown-linux-gnu --options noopt --python-source null --dest-archive /MY_HOME_DIR/python-build-standalone/build/binutils-2.43-linux_x86_64.tar  --toolchain --docker-image gcc binutils
/MY_HOME_DIR/python-build-standalone/cpython-unix/build.py --host-platform linux_x86_64 --target-triple x86_64-unknown-linux-gnu --options noopt --python-source null --dest-archive /MY_HOME_DIR/python-build-standalone/build/llvm-20-x86_64-linux-20.1.4+20250511-linux_x86_64.tar  --toolchain clang --target-triple x86_64-unknown-linux-gnu
Traceback (most recent call last):
  File "/MY_HOME_DIR/python-build-standalone/cpython-unix/build.py", line 1313, in <module>
    sys.exit(main())
             ~~~~^^
  File "/MY_HOME_DIR/python-build-standalone/cpython-unix/build.py", line 1104, in main
    get_image(client, ROOT, BUILD, docker_image, host_platform),
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/MY_HOME_DIR/python-build-standalone/pythonbuild/docker.py", line 81, in get_image
    with image_path.open("r") as fh:
         ~~~~~~~~~~~~~~~^^^^^
  File "/home/linuxbrew/.linuxbrew/opt/python@3.13/lib/python3.13/pathlib/_local.py", line 537, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/MY_HOME_DIR/python-build-standalone/build/image-gcc.linux_x86_64'
downloading https://github.com/indygreg/toolchain-tools/releases/download/toolchain-bootstrap%2F20250511/llvm-20.1.4+20250511-gnu_only-x86_64-unknown-linux-gnu.tar.zst to /MY_HOME_DIR/python-build-standalone/build/downloads/llvm-20.1.4+20250511-gnu_only-x86_64-unknown-linux-gnu.tar.zst
make: *** [Makefile:86: /MY_HOME_DIR/python-build-standalone/build/binutils-2.43-linux_x86_64.tar] Error 1
make: *** Waiting for unfinished jobs....
image-build> Step 1/9 : FROM debian@sha256:32ad5050caffb2c7e969dac873bce2c370015c2256ff984b70c1c08b3a2816a0
image-build> ---> 32ad5050caff
image-build> Step 2/9 : MAINTAINER Gregory Szorc <gregory.szorc@gmail.com>
image-build> ---> Using cache
image-build> ---> 2744d6738034
image-build> Step 3/9 : RUN groupadd -g 1000 build &&     useradd -u 1000 -g 1000 -d /build -s /bin/bash -m build &&     mkdir /tools &&     chown -R build:build /build /tools
image-build> ---> Using cache
image-build> ---> a11ed3d05fa6
image-build> Step 4/9 : ENV HOME=/build     SHELL=/bin/bash     USER=build     LOGNAME=build     HOSTNAME=builder     DEBIAN_FRONTEND=noninteractive
image-build> ---> Using cache
image-build> ---> b583d10b1fa1
image-build> Step 5/9 : CMD ["/bin/bash", "--login"]
image-build> ---> Using cache
image-build> ---> 540f326581ae
image-build> Step 6/9 : WORKDIR '/build'
image-build> ---> Using cache
image-build> ---> 981d84816f67
image-build> Step 7/9 : RUN for s in debian_jessie debian_jessie-updates debian-security_jessie/updates; do       echo "deb [trusted=yes] http://snapshot.debian.org/archive/${s%_*}/20230322T152120Z/ ${s#*_} main";     done > /etc/apt/sources.list &&     ( echo 'quiet "true";';       echo 'APT::Get::Assume-Yes "true";';       echo 'APT::Install-Recommends "false";';       echo 'Acquire::Check-Valid-Until "false";';       echo 'Acquire::Retries "5";';     ) > /etc/apt/apt.conf.d/99cpython-portable
image-build> ---> Using cache
image-build> ---> c27573331df4
image-build> Step 8/9 : RUN ulimit -n 10000 && apt-get update
image-build> ---> Using cache
image-build> ---> fc9f30c45490
image-build> Step 9/9 : RUN ulimit -n 10000 && apt-get install     bzip2     file     libc6-dev     libffi-dev     make     patch     perl     pkg-config     tar     xz-utils     unzip     zip     zlib1g-dev
image-build> ---> Using cache
image-build> ---> fbe46a28ff5d
image-build> Successfully built fbe46a28ff5d
successfully downloaded https://github.com/indygreg/toolchain-tools/releases/download/toolchain-bootstrap%2F20250511/llvm-20.1.4+20250511-gnu_only-x86_64-unknown-linux-gnu.tar.zst

[zanieb]

This is a regression from #607 / #670

I think we're just missing a dependency somewhere..

[zanieb]

Would you mind giving #693 a try?

[kwmlodozeniec]

I applied your changes and now get a different issue:

Traceback (most recent call last):
  File "/MY_HOME_DIR/python-build-standalone/cpython-unix/build.py", line 1313, in <module>
    sys.exit(main())
             ~~~~^^
  File "/MY_HOME_DIR/python-build-standalone/cpython-unix/build.py", line 1104, in main
    get_image(client, ROOT, BUILD, docker_image, host_platform),
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/MY_HOME_DIR/python-build-standalone/pythonbuild/docker.py", line 98, in get_image
    return build_docker_image(
        client, str(source_dir).encode(), image_dir, name, host_platform
    )
  File "/MY_HOME_DIR/python-build-standalone/pythonbuild/docker.py", line 37, in build_docker_image
    return ensure_docker_image(client, io.BytesIO(image_data), image_path=image_path)
  File "/MY_HOME_DIR/python-build-standalone/pythonbuild/docker.py", line 57, in ensure_docker_image
    raise Exception("unable to determine built Docker image")
Exception: unable to determine built Docker image

[zanieb]

Huh, that means the build isn't resulting in an image id. Weird. I'll need to reproduce on a Linux box to debug.

[kwmlodozeniec]

The actual issues is the build.Dockerfile is not properly formatted

ERROR: failed to build: failed to solve: dockerfile parse error on line 1: unknown instruction: {%

I have modified the get_image function call to handle it and now the build works, but I'm not entirely convinced this is the right "fix".

def get_image(
    client, source_dir: pathlib.Path, image_dir: pathlib.Path, name, host_platform
):
    if client is None:
        return None

    image_name = f"image-{name}.{host_platform}"
    image_path = image_dir / image_name
    tar_path = image_path.with_suffix(".tar")

    try:
        with image_path.open("r") as fh:
            image_id = fh.read().strip()
    except FileNotFoundError:
        pass
    else:
        try:
            client.images.get(image_id)
            return image_id
        except docker.errors.ImageNotFound:
            if tar_path.exists():
                with tar_path.open("rb") as fh:
                    data = fh.read()
                client.images.load(data)

                return image_id

    dockerfile_actual_name = f"{name}.Dockerfile"
    dockerfile_path = image_dir / dockerfile_actual_name # Use image_dir where rendered files are

    if not dockerfile_path.exists():
        raise FileNotFoundError(
            f"Rendered Dockerfile not found at {dockerfile_path}. "
            "Ensure write_dockerfiles was called correctly and targets this directory."
        )

    with dockerfile_path.open("rb") as f:
        dockerfile_content = f.read()

    return build_docker_image(
        client, dockerfile_content, image_dir, name, host_platform
    )

[zanieb]

I wonder if the .{host_platform} addition broke the default file extension handling or something.

[geofft]

I am pretty sure #695 fixes it. Pushed to main, please try and let us know if it doesn't work!

[kwmlodozeniec]

yep, that fixed it, many thanks!