Test failures in test_crypto.py on CentOS Stream 9

[knivnia]

Describe the bug

Package fails to build on CentOS Stream 9 - tests in test_crypto.py are failing with AWS_ERROR_CAL_MALFORMED_ASN1_ENCOUNTERED

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

• Tests pass
• RPM builds successfully

Current Behavior

tests in test_crypto.py are failing with errors:

ERROR: test_rsa_encryption_roundtrip_pkcs8 (test.test_crypto.TestCredentials) [RSA Encryption Roundtrip using algo p] (p=<RSAEncryptionAlgorithm.PKCS1_5: 0>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/awscrt-0.27.2/test/test_crypto.py", line 212, in test_rsa_encryption_roundtrip_pkcs8
    rsa = RSA.new_private_key_from_pem_data(RSA_PRIVATE_KEY_PEM_PKCS8)
  File "/home/awscrt-0.27.2/awscrt/crypto.py", line 116, in new_private_key_from_pem_data
    return RSA(binding=_awscrt.rsa_private_key_from_pem_data(pem_data))
RuntimeError: 7172 (AWS_ERROR_CAL_MALFORMED_ASN1_ENCOUNTERED): An ASN.1 DER decoding operation failed on malformed input.

======================================================================
ERROR: test_rsa_encryption_roundtrip_pkcs8 (test.test_crypto.TestCredentials) [RSA Encryption Roundtrip using algo p] (p=<RSAEncryptionAlgorithm.OAEP_SHA256: 1>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/awscrt-0.27.2/test/test_crypto.py", line 212, in test_rsa_encryption_roundtrip_pkcs8
    rsa = RSA.new_private_key_from_pem_data(RSA_PRIVATE_KEY_PEM_PKCS8)
  File "/home/awscrt-0.27.2/awscrt/crypto.py", line 116, in new_private_key_from_pem_data
    return RSA(binding=_awscrt.rsa_private_key_from_pem_data(pem_data))
RuntimeError: 7172 (AWS_ERROR_CAL_MALFORMED_ASN1_ENCOUNTERED): An ASN.1 DER decoding operation failed on malformed input.

======================================================================
ERROR: test_rsa_encryption_roundtrip_pkcs8 (test.test_crypto.TestCredentials) [RSA Encryption Roundtrip using algo p] (p=<RSAEncryptionAlgorithm.OAEP_SHA512: 2>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/awscrt-0.27.2/test/test_crypto.py", line 212, in test_rsa_encryption_roundtrip_pkcs8
    rsa = RSA.new_private_key_from_pem_data(RSA_PRIVATE_KEY_PEM_PKCS8)
  File "/home/awscrt-0.27.2/awscrt/crypto.py", line 116, in new_private_key_from_pem_data
    return RSA(binding=_awscrt.rsa_private_key_from_pem_data(pem_data))
RuntimeError: 7172 (AWS_ERROR_CAL_MALFORMED_ASN1_ENCOUNTERED): An ASN.1 DER decoding operation failed on malformed input.

======================================================================
ERROR: test_rsa_signing_roundtrip_pkcs8 (test.test_crypto.TestCredentials) [RSA Signing Roundtrip using algo p] (p=<RSASignatureAlgorithm.PKCS1_5_SHA256: 0>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/awscrt-0.27.2/test/test_crypto.py", line 264, in test_rsa_signing_roundtrip_pkcs8
    rsa = RSA.new_private_key_from_pem_data(RSA_PRIVATE_KEY_PEM_PKCS8)
  File "/home/awscrt-0.27.2/awscrt/crypto.py", line 116, in new_private_key_from_pem_data
    return RSA(binding=_awscrt.rsa_private_key_from_pem_data(pem_data))
RuntimeError: 7172 (AWS_ERROR_CAL_MALFORMED_ASN1_ENCOUNTERED): An ASN.1 DER decoding operation failed on malformed input.

======================================================================
ERROR: test_rsa_signing_roundtrip_pkcs8 (test.test_crypto.TestCredentials) [RSA Signing Roundtrip using algo p] (p=<RSASignatureAlgorithm.PSS_SHA256: 2>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/awscrt-0.27.2/test/test_crypto.py", line 264, in test_rsa_signing_roundtrip_pkcs8
    rsa = RSA.new_private_key_from_pem_data(RSA_PRIVATE_KEY_PEM_PKCS8)
  File "/home/awscrt-0.27.2/awscrt/crypto.py", line 116, in new_private_key_from_pem_data
    return RSA(binding=_awscrt.rsa_private_key_from_pem_data(pem_data))
RuntimeError: 7172 (AWS_ERROR_CAL_MALFORMED_ASN1_ENCOUNTERED): An ASN.1 DER decoding operation failed on malformed input.

======================================================================
ERROR: test_rsa_signing_roundtrip_pkcs8 (test.test_crypto.TestCredentials)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/awscrt-0.27.2/test/__init__.py", line 56, in tearDown
    check_for_leaks(timeout_sec=TIMEOUT)
  File "/home/awscrt-0.27.2/awscrt/_test.py", line 146, in check_for_leaks
    raise RuntimeError("awscrt leak check failed. {} NativeResource objects. {} bytes native memory".format(
RuntimeError: awscrt leak check failed. 0 NativeResource objects. 40 bytes native memory

----------------------------------------------------------------------
Ran 18 tests in 30.165s
FAILED (errors=6)

Reproduction Steps

Can be reproduced in a CentOS Stream 9 container:

• Run the container
  podman run --rm -it centos:stream9 /bin/bash

• Download the sources from PyPI
  curl -LO https://files.pythonhosted.org/packages/2c/6d/570d1feba08c1006c8f47de3725e87b7b968a875e52a792b74d6598bef08/awscrt-0.27.2.tar.gz

• Apply this patch to get past SHA1 issues mentioned in related issue
  patch -p1 < skip-SHA1-in-test_crypto.patch

• Install dependencies
  dnf install gcc gcc-c++ cmake openssl-devel python3-devel python3-pip
pip install wheel

• Build the package specifying build flags from CentOS Stream 9 and enabling OpenSSL
  CFLAGS="-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection" AWS_CRT_BUILD_USE_SYSTEM_LIBCRYPTO=1 python3 -m pip wheel --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --no-clean --progress-bar off --verbose .

• Run the tests
  PYTHONPATH=$(pwd)/build/lib.linux-x86_64-3.9 python3 -m unittest test/test_crypto.py

Possible Solution

No response

Additional Information/Context

The same issue can be observed on Fedora, but only on i686, on CentOS Stream 9 it happens on all - x86_64, aarch64, and ppc64le

aws-crt-python version used

0.27.2

Python version used

3.9

Operating System and version

CentOS Stream 9

[knivnia]

there is a PR with the fix opened in aws-c-cal

[DmitriyMusatkin]

That PR is to fix parsing on big-endian machines, but you mentioned that you are observing failures on little-endian machines.

I dont know specifically what about CentOS Stream 9 is making it fail, but I will be looking into why its failing.

[DmitriyMusatkin]

small update: im working on a fix for this, but so far it seems that extra compiler flags you are using are somehow breaking the code. my guess is we have a ub somewhere in asn1 parser that im tracking.

[tollsimy]

@DmitriyMusatkin these specific failures were related to big endianess and have already been fixed by recent PRs (despite not all of them being merged yet, e.g. awslabs/aws-c-common#1218), however another different issue has been reported. In particular, this has been fixed in #679 and the patch is already merged in Fedora repos thanks to @knivnia , confirming it is working as intended.