4 pack-related tests fail on s390x

[EliahKagan]

Current behavior 😯

On an s390x system running Ubuntu 22.04 LTS with git 2.48.1 installed from the git-core PPA, two tests fail when using committed fixture archives, and those tests as well as two others fail when ignoring archives.

Full test output of the most recent runs can be viewed in this gist. The same failures have been happening for a while; that gist links to a few older gists. This is not new as of git 2.48.1; it is the same problem as observed with git 2.47.0 (also installed through the git-core PPA) as mentioned in #1687 and discusses in comments in #1698, starting at #1698 (comment).

Without GIX_TEST_IGNORE_ARCHIVES cargo nextest run --workspace --no-fail-fast, the failing tests are:

gix::gix remote::fetch::blocking_and_async_io::fetch_pack
gix::gix remote::fetch::blocking_and_async_io::fetch_pack_without_local_destination

With GIX_TEST_IGNORE_ARCHIVES=1 cargo nextest run --workspace --no-fail-fast, the failing tests are:

gix::gix remote::fetch::blocking_and_async_io::fetch_pack
gix::gix remote::fetch::blocking_and_async_io::fetch_pack_without_local_destination
gix-pack-tests::pack pack::multi_index::access::general
gix-pack-tests::pack pack::multi_index::verify::integrity

Notably, all these failing tests relate to packs. But various other tests related to packs do not fail.

Rerunning the first two failing tests (those that occur even when using committed archives) by themselves and enabling a backtrace shows:

ubuntu@gull:~/repos/gitoxide (main %=)$ git restore .
ubuntu@gull:~/repos/gitoxide (main %=)$ gix clean -xd -m *generated* -e
removing gix/tests/fixtures/generated-archives/make_fetch_repos__home_ubuntu_repos_gitoxide_gix_tests_fixtures_generated-do-not-edit_make_remote_repos_1456912905-unix_base.tar
removing gix/tests/fixtures/generated-do-not-edit/ (🗑️)
ubuntu@gull:~/repos/gitoxide (main =)$ RUST_BACKTRACE=1 cargo nextest run -p gix --features blocking-network-client --no-fail-fast fetch_pack
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.20s
────────────
 Nextest run ID 94c9d23a-4da4-4f24-b4d8-4c8de9e93483 with nextest profile: default
    Starting 2 tests across 4 binaries (314 tests skipped)
        FAIL [   9.079s] gix::gix remote::fetch::blocking_and_async_io::fetch_pack_without_local_destination
──── STDOUT:             gix::gix remote::fetch::blocking_and_async_io::fetch_pack_without_local_destination

running 1 test
test remote::fetch::blocking_and_async_io::fetch_pack_without_local_destination ... FAILED

failures:

failures:
    remote::fetch::blocking_and_async_io::fetch_pack_without_local_destination

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 296 filtered out; finished in 9.05s

──── STDERR:             gix::gix remote::fetch::blocking_and_async_io::fetch_pack_without_local_destination
Archive at 'tests/fixtures/generated-archives/make_fetch_repos__home_ubuntu_repos_gitoxide_gix_tests_fixtures_generated-do-not-edit_make_remote_repos_1456912905-unix_base.tar' not found, creating fixture using script 'make_fetch_repos.sh'

thread 'remote::fetch::blocking_and_async_io::fetch_pack_without_local_destination' panicked at gix/tests/gix/remote/fetch.rs:526:21:
assertion `left == right` failed
  left: Sha1(3c86147bd5230d7476b277f3e73a9ca75765e3eb)
 right: Sha1(de303ef102bd5705a40a0c42ae2972eb1a668455)
stack backtrace:
   0: rust_begin_unwind
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5
   1: core::panicking::panic_fmt
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14
   2: core::panicking::assert_failed_inner
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:409:17
   3: core::panicking::assert_failed
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:364:5
   4: gix::remote::fetch::blocking_and_async_io::fetch_pack_without_local_destination
             at ./tests/gix/remote/fetch.rs:526:21
   5: gix::remote::fetch::blocking_and_async_io::fetch_pack_without_local_destination::{{closure}}
             at ./tests/gix/remote/fetch.rs:484:56
   6: core::ops::function::FnOnce::call_once
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5
   7: core::ops::function::FnOnce::call_once
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

        FAIL [   9.211s] gix::gix remote::fetch::blocking_and_async_io::fetch_pack
──── STDOUT:             gix::gix remote::fetch::blocking_and_async_io::fetch_pack

running 1 test
test remote::fetch::blocking_and_async_io::fetch_pack ... FAILED

failures:

failures:
    remote::fetch::blocking_and_async_io::fetch_pack

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 296 filtered out; finished in 9.19s

──── STDERR:             gix::gix remote::fetch::blocking_and_async_io::fetch_pack

thread 'remote::fetch::blocking_and_async_io::fetch_pack' panicked at gix/tests/gix/remote/fetch.rs:614:25:
assertion `left == right` failed
  left: Sha1(12c2105d54f366c7b1edc6593081646d9f788abe)
 right: Sha1(d07c527cf14e524a8494ce6d5d08e28079f5c6ea)
stack backtrace:
   0: rust_begin_unwind
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5
   1: core::panicking::panic_fmt
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14
   2: core::panicking::assert_failed_inner
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:409:17
   3: core::panicking::assert_failed
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:364:5
   4: gix::remote::fetch::blocking_and_async_io::fetch_pack
             at ./tests/gix/remote/fetch.rs:614:25
   5: gix::remote::fetch::blocking_and_async_io::fetch_pack::{{closure}}
             at ./tests/gix/remote/fetch.rs:543:30
   6: core::ops::function::FnOnce::call_once
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5
   7: core::ops::function::FnOnce::call_once
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

────────────
     Summary [   9.216s] 2 tests run: 0 passed, 2 failed, 314 skipped
        FAIL [   9.211s] gix::gix remote::fetch::blocking_and_async_io::fetch_pack
        FAIL [   9.079s] gix::gix remote::fetch::blocking_and_async_io::fetch_pack_without_local_destination
error: test run failed

Rerunning the two tests that fail only when ignoring commited archives shows:

ubuntu@gull:~/repos/gitoxide (main =)$ gix clean -xd -m *generated* -e
removing gix-pack/tests/fixtures/generated-do-not-edit/ (🗑️)
ubuntu@gull:~/repos/gitoxide (main =)$ RUST_BACKTRACE=1 GIX_TEST_IGNORE_ARCHIVES=1 cargo nextest run -p gix-pack-tests --no-fail-fast pack::multi_index::access::general pack::multi_index::verify::integrity
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.17s
────────────
 Nextest run ID 98ce247c-ced0-4681-8446-bb0ddff2c23a with nextest profile: default
    Starting 2 tests across 1 binary (44 tests skipped)
        FAIL [   5.020s] gix-pack-tests::pack pack::multi_index::access::general
──── STDOUT:             gix-pack-tests::pack pack::multi_index::access::general

running 1 test
test pack::multi_index::access::general ... FAILED

failures:

failures:
    pack::multi_index::access::general

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 45 filtered out; finished in 5.00s

──── STDERR:             gix-pack-tests::pack pack::multi_index::access::general
failed to extract 'fixtures/generated-archives/make_pack_gen_repo_multi_index.tar': Ignoring archive at 'fixtures/generated-archives/make_pack_gen_repo_multi_index.tar' as GIX_TEST_IGNORE_ARCHIVES is set.

thread 'pack::multi_index::access::general' panicked at gix-pack/tests/pack/multi_index/access.rs:69:5:
assertion `left == right` failed
  left: Sha1(d48be4a0ac4ba8b2b4301c7b014c7fb37656ad29)
 right: Sha1(39a3804d0a84de609e4fcb49e66dc1297c75ca11)
stack backtrace:
   0: rust_begin_unwind
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5
   1: core::panicking::panic_fmt
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14
   2: core::panicking::assert_failed_inner
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:409:17
   3: core::panicking::assert_failed
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:364:5
   4: pack::pack::multi_index::access::general
             at ./pack/multi_index/access.rs:69:5
   5: pack::pack::multi_index::access::general::{{closure}}
             at ./pack/multi_index/access.rs:61:13
   6: core::ops::function::FnOnce::call_once
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5
   7: core::ops::function::FnOnce::call_once
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

        FAIL [   5.603s] gix-pack-tests::pack pack::multi_index::verify::integrity
──── STDOUT:             gix-pack-tests::pack pack::multi_index::verify::integrity

running 1 test
test pack::multi_index::verify::integrity ... FAILED

failures:

failures:
    pack::multi_index::verify::integrity

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 45 filtered out; finished in 5.59s

──── STDERR:             gix-pack-tests::pack pack::multi_index::verify::integrity

thread 'pack::multi_index::verify::integrity' panicked at gix-pack/tests/pack/multi_index/verify.rs:25:5:
assertion `left == right` failed
  left: [Statistics { average: Outcome { kind: Tree, num_deltas: 1, decompressed_size: 47, compressed_size: 49, object_size: 152 }, objects_per_chain_length: {0: 326, 1: 106, 2: 326, 3: 108, 4: 2}, total_compressed_entries_size: 42781, total_decompressed_entries_size: 40919, total_object_size: 131993, pack_size: 45012, num_commits: 16, num_trees: 40, num_tags: 1, num_blobs: 811 }]
 right: [Statistics { average: Outcome { kind: Tree, num_deltas: 1, decompressed_size: 47, compressed_size: 46, object_size: 152 }, objects_per_chain_length: {0: 326, 1: 106, 2: 326, 3: 108, 4: 2}, total_compressed_entries_size: 40628, total_decompressed_entries_size: 40919, total_object_size: 131993, pack_size: 42856, num_commits: 16, num_trees: 40, num_tags: 1, num_blobs: 811 }]
stack backtrace:
   0: rust_begin_unwind
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5
   1: core::panicking::panic_fmt
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14
   2: core::panicking::assert_failed_inner
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:409:17
   3: core::panicking::assert_failed
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:364:5
   4: pack::pack::multi_index::verify::integrity
             at ./pack/multi_index/verify.rs:25:5
   5: pack::pack::multi_index::verify::integrity::{{closure}}
             at ./pack/multi_index/verify.rs:19:15
   6: core::ops::function::FnOnce::call_once
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5
   7: core::ops::function::FnOnce::call_once
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

────────────
     Summary [   5.605s] 2 tests run: 0 passed, 2 failed, 44 skipped
        FAIL [   5.020s] gix-pack-tests::pack pack::multi_index::access::general
        FAIL [   5.603s] gix-pack-tests::pack pack::multi_index::verify::integrity
error: test run failed

Expected behavior 🤔

All tests should pass. (Unless it turns out that the failures are due to entirely to bugs in git itself.)

s390x is bigendian, and to the best of my knowledge is the most prominent bigendian architecture for general-purpose use. Bugs that occur specifically on s390x suggest the possibility of unsoundness, though they could arise without any soundness bugs if they happen due to a mismatch in expectations in how data structures are written to disk on different architectures. (If the representation is not validated before being written into memory using unsafe code, then that would itself be unsound, but that would be unsound even in the absence of a mismatch in expectations.)

Git behavior

Presumably subtleties about git behavior are relevant, since two of the failures only occur when rerunning fixture scripts, but I do not know how. This need not imply that a bug in the git is the cause.

Steps to reproduce 🕹

On Ubuntu 22.04 LTS on s390x, I enabled the git-core PPA and upgraded git. I verified the git version with git version and by running apt list git to see that it is installed from the PPA.

I ran the test suite, allowing committed archives to be used, showing the first two failures:

cargo nextest run --workspace --no-fail-fast

I did a full clean, then ran the test suite again, forcing it to rerun fixture scripts, showing those failures again as well as the last two:

gix clean -xde
GIX_TEST_IGNORE_ARCHIVES=1 cargo nextest run --workspace --no-fail-fast

I restored the original archives, removed generated repositories, and ran the first two failing tests with backtraces:

git restore .
gix clean -xd -m *generated* -e
RUST_BACKTRACE=1 cargo nextest run -p gix --features blocking-network-client --no-fail-fast fetch_pack

I removed generated repositories again and ran the other two failing tests with backtraces:

gix clean -xd -m *generated* -e
RUST_BACKTRACE=1 GIX_TEST_IGNORE_ARCHIVES=1 cargo nextest run -p gix-pack-tests --no-fail-fast pack::multi_index::access::general pack::multi_index::verify::integrity

The full workspace runs without backtraces are shown in the above-linked gist, with older such runs shown in other gists that it links to. The specific runs' full output are shown above in the "Current behavior" section.

I have not been able to reproduce it with cross

We do not test s390x on CI. Running the full test suite in s390x on CI may not be justifiable, because GitHub Actions does not support s390x, and tests would run slower under QEMU emulation. However, this is feasible, including through cross, which speeds things up by doing the actual building with tools that themselves can run without emulation. (The cross Docker images for non-x86-64 targets are actually x86-64 images, which have toolchains on them that are built for x86-64 but that build the other non-x86-64 target. It is also possible to install dependencies on them for the cross-target architecture, which can run in QEMU via binfmt_misc, and sometimes they contain some such dependencies, such as the Android NDK on Android images.) More likely to be justifiable would be to run a subset of the tests, including any tests that have failed before or about which there is specific concern.

The testing described above was done on actual s390x hardware. (Specifically, it was done without containerization, on an s390x virtual machine, where the host was also s390x and hardware virtualization was used.) But I have also attempted to use cross for this. When I do, I have so far been unable to produce these failures--all s390x tests pass. This happens both with max-pure and with all default features, and includes all the tests that fail here. They pass in the container.

The version of git on the cross image is old, such that tests will fail if it is not upgraded. I upgrade it, also using the git-core PPA added as a repository in the cross image. This allows all tests to pass in the image, both with GIX_TEST_IGNORE_ARCHIVES unset and with it set to 1. (By default, cross filters most environment variables from outside the container. I account for this and verify that the environment variables are present and have their expected effects.)

The tests pass when amd64 (i.e. x86-64) builds of git and its dependencies are installed from the git-core PPA. They also pass when I instead install s390x builds (which the git-core PPA also provides) of git and s390x builds of its library dependencies and of its non-library dependencies, except that I am not able to install s390x builds of perl and liberror-perl, so I instead force s390x git to use the amd64 perl and liberror-perl by telling dpkg to allow those s390x dependencies to be unsatisfied and ensuring the amd64 packages are installed.

git is able to use perl of a different architecture because it uses it to run scripts, rather than linking to it as a shared library (and while liberror-perl is a library package, it provides a Perl module, not a shared library that git itself links to). Since I don't think recent versions of git use perl very heavily, it seems unlikely that the architecture perl is built for is what makes the difference here, but I have not ruled that out.

I have verified that the failures occur, and in the same way, on the system where I have observed them, even when I use cargo test rather than cargo nextest run. This is relevant to cross because cross does not support running nextest; when using cross, I have been using cross test, which runs cargo test in the container. Differences between nextest and cargo-test are not the cause. (This is worth checking because nextest makes greater use of multiprocessing to run multiple tests at a time, while cargo-test makes greater use of multithreading. However, in this case, the tests fail even when run individually--which I have also verified.)