Speed regression under emulation when using Trixie.

[stumpylog]

I'll be up front, I'm unsure this is an issue with this image exactly, given the myriad of things to build under emulation (buildx, qemu).

But since looking to upgrade our image to Trixie, most steps, including Python commands, have gotten noticably slower.

I've put together a toy example here:
https://github.com/stumpylog/nltk-download-reproducer

As you can see, Bookworm under emulation is around a minute to complete. Trixie is about 7 minutes. I have no idea what could have changed here. Is there missing optimization? Does Trixie use newer instructions qemu does not support?

[tianon]

I've noticed apparent emulation performance issues with other tools too (anecdotally), so I think this is more related to QEMU and/or Trixie than it is the Python image specifically. 🤔

`hyperfine` results:

(all tests run on a Debian Bookworm amd64 host, kernel 6.1.0-37-amd64, with qemu-user-static version 1:7.2+dfsg-7+deb12u13)

Even for something as basic as apt-get moo (which is just a cute easter egg that prints a static ASCII art of a cow):

$ hyperfine -L suite 'bookworm,trixie' --command-name '{suite}' --setup 'docker pull --platform linux/arm64/v8 arm64v8/debian:{suite}-slim' 'docker run --rm --platform linux/arm64/v8 arm64v8/debian:{suite}-slim apt-get moo'
Benchmark 1: bookworm
  Time (mean ± σ):     637.0 ms ±  32.7 ms    [User: 8.6 ms, System: 5.0 ms]
  Range (min … max):   596.0 ms … 685.7 ms    10 runs
 
Benchmark 2: trixie
  Time (mean ± σ):     698.7 ms ±  28.8 ms    [User: 7.5 ms, System: 6.1 ms]
  Range (min … max):   661.0 ms … 746.0 ms    10 runs
 
Summary
  'bookworm' ran
    1.10 ± 0.07 times faster than 'trixie'

More dramatic for something bigger like curl:

$ hyperfine -L suite 'bookworm,trixie' --command-name '{suite}' --setup 'docker pull --platform linux/arm64/v8 arm64v8/buildpack-deps:{suite}-curl' 'docker run --rm --platform linux/arm64/v8 arm64v8/buildpack-deps:{suite}-curl curl https://1.1.1.1'
Benchmark 1: bookworm
  Time (mean ± σ):      1.263 s ±  0.142 s    [User: 0.008 s, System: 0.007 s]
  Range (min … max):    1.156 s …  1.649 s    10 runs
 
Benchmark 2: trixie
  Time (mean ± σ):      2.434 s ±  0.136 s    [User: 0.013 s, System: 0.005 s]
  Range (min … max):    2.355 s …  2.811 s    10 runs
 
Summary
  'bookworm' ran
    1.93 ± 0.24 times faster than 'trixie'

and even true shows a slight difference:

$ hyperfine -L suite 'bookworm,trixie' --command-name '{suite}' --setup 'docker pull --platform linux/arm64/v8 arm64v8/buildpack-deps:{suite}-curl' 'docker run --rm --platform linux/arm64/v8 arm64v8/buildpack-deps:{suite}-curl true'
Benchmark 1: bookworm
  Time (mean ± σ):     568.0 ms ±  34.6 ms    [User: 6.5 ms, System: 8.1 ms]
  Range (min … max):   514.6 ms … 616.1 ms    10 runs
 
Benchmark 2: trixie
  Time (mean ± σ):     581.8 ms ±  46.7 ms    [User: 8.1 ms, System: 6.6 ms]
  Range (min … max):   517.3 ms … 636.6 ms    10 runs
 
Summary
  'bookworm' ran
    1.02 ± 0.10 times faster than 'trixie'

There's certainly nothing (besides updating to Trixie) that we've changed in the image that might've caused this, so it's definitely not something we've caused intentionally (and unlikely something we're going to make changes to the image to fix), so I'm going to convert this to a discussion instead. 🙇

[stumpylog]

Sorry I didn't when see discussions were enabled. It's perplexing and I'm not even sure where to start investigating, let alone raise a real issue

[tianon]

I don't yet have a system handy which is running the Trixie kernel, but I do have one that's running Trixie's copy of qemu-user-binfmt (which is probably what's really relevant / matters here anyhow), and all the above benchmarks are pretty consistently faster for the Trixie versions, so that's really interesting (and less worrying).

`hyperfine` results:

$ hyperfine -L suite 'bookworm,trixie' --command-name '{suite}' --setup 'docker pull --platform linux/arm64/v8 arm64v8/debian:{suite}-slim' 'docker run --rm --platform linux/arm64/v8 arm64v8/debian:{suite}-slim apt-get moo'
Benchmark 1: bookworm
  Time (mean ± σ):     481.6 ms ±  27.5 ms    [User: 8.9 ms, System: 11.4 ms]
  Range (min … max):   434.3 ms … 519.7 ms    10 runs

Benchmark 2: trixie
  Time (mean ± σ):     480.0 ms ±  23.2 ms    [User: 7.0 ms, System: 13.1 ms]
  Range (min … max):   442.6 ms … 515.4 ms    10 runs

Summary
  trixie ran
    1.00 ± 0.08 times faster than bookworm

$ hyperfine --warmup 1 -L suite 'bookworm,trixie' --command-name '{suite}' --setup 'docker pull --platform linux/arm64/v8 arm64v8/buildpack-deps:{suite}-curl' 'docker run --rm --platform linux/arm64/v8 arm64v8/buildpack-deps:{suite}-curl curl https://1.1.1.1'
Benchmark 1: bookworm
  Time (mean ± σ):     942.1 ms ±  51.5 ms    [User: 8.4 ms, System: 12.4 ms]
  Range (min … max):   875.9 ms … 1042.1 ms    10 runs

Benchmark 2: trixie
  Time (mean ± σ):     823.1 ms ±  25.7 ms    [User: 7.6 ms, System: 13.2 ms]
  Range (min … max):   799.3 ms … 876.4 ms    10 runs

Summary
  trixie ran
    1.14 ± 0.07 times faster than bookworm

$ hyperfine --warmup 1 -L suite 'bookworm,trixie' --command-name '{suite}' --setup 'docker pull --platform linux/arm64/v8 arm64v8/buildpack-deps:{suite}-curl' 'docker run --rm --platform linux/arm64/v8 arm64v8/buildpack-deps:{suite}-curl true'
Benchmark 1: bookworm
  Time (mean ± σ):     428.1 ms ±  38.7 ms    [User: 6.3 ms, System: 13.8 ms]
  Range (min … max):   382.5 ms … 493.0 ms    10 runs

Benchmark 2: trixie
  Time (mean ± σ):     422.4 ms ±  18.5 ms    [User: 9.0 ms, System: 11.1 ms]
  Range (min … max):   391.2 ms … 460.1 ms    10 runs

Summary
  trixie ran
    1.01 ± 0.10 times faster than bookworm

[stumpylog]

So is that a newer qemu version? The only interaction I have with it is via the action to set it up. I've asked for it to be updated to the latest qemu release already

[tianon]

I'm using the https://packages.debian.org/qemu-user-binfmt package from Debian itself (so that's 7.2 in bookworm and 10.0 in trixie). It sets up all the binfmt registrations automatically, so there's no need for anything third-party (works in Ubuntu too).

[mhkarimi1383]

Hi

addition to slowness that is introduced
We are also getting

> [linux/arm64 builder-base 5/5] RUN poetry install --only main --no-root:
82.60
82.60 Command '['/usr/local/bin/python', '-EsSc', 'import platform; print(platform.python_version())']' timed out after 5.0 seconds

while installing our dependencies using poetry using python:3.13-slim-trixie

[mhkarimi1383]

IDK, But 5s is too much for this simple thing to just determine the python version from the platform
And this is happening only on ARM builds

[tianon]

I agree, but I'm not sure there's really anything we can do about it in our builds of Python -- we didn't explicitly "choose" any of the things that might be causing this (and user-mode emulation has always been a bit of an extremely clever hack, and it's a miracle any of it ever works).

[mhkarimi1383]

We have to use that to be able to have our container image builds to be multi-arch, and It's broken after the debian trixie update

We are using the qemu driver of docker buildkit builder and buildx commands to build our images in GitHub Actions

IDK if there is any alternative for that or
If here is not a good place to talk about this problem, can you guide me to know where to submit a but or issue

Thank you

[stumpylog]

I've requested the qemu image the actions use provide an update to the lasted version, which would be close to what tianon was using. That might resolve or improve the situation. I'm not sure how active the repo really is though.

See tonistiigi/binfmt#258

[mhkarimi1383]

Thanks for pointing out...