[Python][Release] test_s3_options takes 11 minutes to complete on wheels verification for almalinux

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

The wheels verification on Almalinux for 24.0.0 fails consistently due to timeout:
https://github.com/apache/arrow/actions/runs/24404382517/job/71425229713?pr=49742

This is related to test_s3_options taking a really long time to complete. I've been able to reproduce locally with:

ALMALINUX=8 archery docker run -e TEST_DEFAULT=0 -e TEST_WHEELS=1 -e VERBOSE=1 -e VERIFY_RC=0 -e VERIFY_VERSION=24.0.0 -e GH_TOKEN=$GH_TOKEN almalinux-verify-rc

I've validated that given enough time the test is successful and I've validated that skipping the test makes the job successful in a reasonable amount of time. The diff applied:

diff --git a/ci/scripts/python_wheel_unix_test.sh b/ci/scripts/python_wheel_unix_test.sh
index 2b8ee7be74..b55b07401f 100755
--- a/ci/scripts/python_wheel_unix_test.sh
+++ b/ci/scripts/python_wheel_unix_test.sh
@@ -116,5 +116,5 @@ if [ "${CHECK_UNITTESTS}" == "ON" ]; then

   # Execute unittest, test dependencies must be installed
   python -c 'import pyarrow; pyarrow.create_library_symlinks()'
-  python -m pytest -r s --pyargs pyarrow
+  python -m pytest -r s --pyargs pyarrow -k "not test_s3_options"
 fi

The problem seems to be related to a credential resolution timeout on S3 and that the test runs twice (due to two test cases being exercised).

We should reduce the time the test takes on those specific conditions.

Component(s)

Python

[raulcd]

After a lot of investigation and trying to understand what is going on I've found a fix and what I understand to be the issue.

The same exact wheel on ubuntu-verify-rc is successful and the test takes seconds. On almalinux-verify-rc it takes ~26 minutes.
The wheel is the same one with the same bundled dependencies, the host OS for the docker containers is the same, the difference glibc, system dependencies and base image.

If I run the verification on almalinux specifying SSL_CERT_FILE:

[root@54b59d8e7749 /]# SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt /arrow/dev/release/verify-release-candidate.sh 24.0.0 0

The test is successful in seconds (same as Ubuntu).

The manylinux wheel bundles OpenSSL 3.6.0. On AlmaLinux 8, /etc/ssl/certs/ contains only bundle files:

# ls /etc/ssl/certs/
ca-bundle.crt  ca-bundle.trust.crt

On Ubuntu there's a bunch of certs on the same path and weird hash files (more on that below).

From what I've been reading Openssl is built with OPENSSLDIR=/etc/ssl but it seems OpenSSL tries to do a file search based on look up by the CA subject name hash value. From what I understand we are hitting this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1053882

Which also from what I understand was never ported to RHEL 8 (only RHEL 9 / AlmaLinux 9).

It seems that running:

p11-kit extract --format=pem-directory-hash --filter=ca-anchors --overwrite --purpose=server-auth /etc/ssl/certs/

Also fixes the issue.

@kou @pitrou any thoughts?

Should we just run on the almalinux-verify-rc command?

p11-kit extract --format=pem-directory-hash --filter=ca-anchors --overwrite --purpose=server-auth /etc/ssl/certs/