Unable to load any oqs-provider generated PEM files

[andybrucenet]

Summary:

Can someone provide guidance on how to use BouncyCastle to read post-quantum PEM files generated by openssl / liboqs / oqs-provider?

Recently was tasked to implement post-quantum crypto in both C++ native libs as well as .net code. We use openssl for C++; thus, gravitated to openssl 3.2.1 / liboqs / oqs-provider which integrates ml-kem (aka Kyber), falcon, sphincs+, etc. directly into openssl APIs. Used OQS_ALGS_ENABLED=STD when building liboqs to get only the NIST-approved list of algorithms.

Created PEM outputs for all supported algorithms (no passwords).

However - latest BouncyCastle.Cryptography 2.3.0 is unable to read any of the PEM files generated by the C++ app.

Attachment contains READMEs, full source code and screenshots showing output.

Notes:

1. Created a trivial test C++ app which enumerates all included OQS algorithms and creates a PEM using EVP_PKEY_keygen / PEM_write_bio_PrivateKey_ex. In that same C++ app verified the created PEM by using PEM_read_bio_PrivateKey_ex.

2. Created trivial C# app which reads each of the generated PEM files and uses PemReader.ReadObject() to load the file. However, receive the same exception for all:

   private key: Org.BouncyCastle.Security.SecurityUtilityException: algorithm identifier in private key not recognised
      at Org.BouncyCastle.Security.PrivateKeyFactory.CreateKey(PrivateKeyInfo keyInfo)
      at Org.BouncyCastle.OpenSsl.PemReader.ReadPrivateKey(PemObject pemObject
   

3. The C# app uses latest BouncyCastle.Cryptography 2.3.0 - in fact, that is the only nuget package installed for the app.

4. Attached source code and examples of several PEM files as well as screenshots of both PEM.

5. Openssl client has oqs-provider installed and can read the PEM files generated by the C++ app.

Source

The C++ source code is pretty simple; given an input algorithm name, create a PKEY and export PEM-encoded to /tmp:

  fprintf(stdout, "GENERATE: %s ", algorithm.c_str());
  keyctx = EVP_PKEY_CTX_new_from_name(libctx, algorithm.c_str(), nullptr);
  if(!keyctx) {
    fprintf(stderr,
            "`EVP_PKEY_CTX_new_from_name` failed\n");
    rc = 1;
    funcResult = false;
    goto err;
  }
  
  // initialize context
  ret = EVP_PKEY_keygen_init(keyctx);
  if(ret != 1) {
    fprintf(stderr,
            "`EVP_PKEY_keygen_init` failed: %d\n",
            ret);
    rc = 1;
    funcResult = false;
    goto err;
  }
  
  // create key with default parameters
  ret = EVP_PKEY_keygen(keyctx, &pkeyOut);
  if(ret != 1) {
    fprintf(stderr,
            "`EVP_PKEY_keygen` failed: %d\n",
            ret);
    rc = 1;
    funcResult = false;
    goto err;
  }
  
  // convert to pem
  memset(buff, 0, sizeof(buff));
  snprintf(buff, sizeof(buff), "/tmp/foobar-%s.pem", algorithm.c_str());
  bioOut = BIO_new_file(buff, "wb");
  ret = PEM_write_bio_PrivateKey_ex(bioOut, pkeyOut, nullptr, nullptr, 0, nullptr, nullptr, libctx, nullptr);
  if(ret != 1) {
    fprintf(stderr,
            "`PEM_write_bio_PrivateKey` failed: %d\n",
            ret);
    rc = 1;
    funcResult = false;
    goto err;
  }
  ret = BIO_flush(bioOut);
  if(ret != 1) {
    fprintf(stderr,
            "`BIO_flush` failed: %d\n",
            ret);
    rc = 1;
    funcResult = false;
    goto err;
  }
  
  // read from PEM to verify bi-directional
  bioIn = BIO_new_file(buff, "rb");
  pkeyIn = PEM_read_bio_PrivateKey_ex(bioIn, nullptr, nullptr, nullptr, libctx, nullptr);
  if(!pkeyIn) {
    fprintf(stderr,
            "`PEM_read_bio_PrivateKey` failed\n");
    rc = 1;
    funcResult = false;
    goto err;
  }

The C# source code is even simpler: Read all /tmp/foobar-*.pem and attempt to read using BouncyCastle:

using System.IO;
using Org.BouncyCastle.Crypto;
using Org.BouncyCastle.OpenSsl;

// read PEM files created using openssl / oqs-provider using BouncyCastle
class Program
{
    static void Main(string[] args)
    {
        var pemFiles = Directory.GetFiles("/tmp", "foobar-*.pem");
        foreach (var pemFile in pemFiles)
        {
            Console.Write($"{pemFile}: ");
            var theKeyStream = File.OpenText(pemFile);
            using (theKeyStream)
            {
                var theKeyPemReader = new PemReader(theKeyStream);
                try
                {
                    var theKey = (AsymmetricCipherKeyPair)theKeyPemReader.ReadObject();
                    if (!(theKey is null))
                    {
                        Console.Write(theKey.Public.ToString());
                    }
                    else
                    {
                        Console.Write("[null]");
                    }
                }
                catch (Exception ex)
                {
                    Console.Write(ex.Message);
                }
            }
            Console.WriteLine();
        }
    }
}

The attachment contains the full projects including static libraries (at least for macOS which is what I'm doing my testing on).

[cipherboy]

@andybrucenet It looks like you might be short an attachment, so the below is pure speculation on my part based on what others have encountered in the past... :-)

What version of OQS are you using?

open-quantum-safe/liboqs#1626 merged two weeks ago (:tada:), adding the new draft ML-DSA &co specs that NIST has published, but it doesn't look like they've published a new release yet according to https://github.com/open-quantum-safe/liboqs/tags.

Before that PR, bc-csharp supported the latest NIST drafts but OQS didn't yet, resulting in an algorithm mismatch many were seeing trying to interop with BC. We've also updated to the newest draft OIDs while removing the older draft OIDs, which also may cause problems for interop (as the older OIDs point to older draft versions of Kyber &co which we no longer implement).

Let us know if that helps,

• A

[andybrucenet]

pq-test.zip

@cipherboy Thanks for the quick reply. I just put the attachment up - in the attachment README I wanted the link to the discussion so I had to create the discussion first :)

And - yes - I have read quite a few of your posts specifically including the note about the NIST drafts. In my case I suspect this to be the problem which is why I try creating all of the NIST-blessed algorithms. But my capabilities to dive into either openssl or BC are pretty limited, so I hope that my example app will demonstrate what I'm doing and - better yet - what I'm doing wrong.

The attachment has all of my files - I'm using the latest of everything. Latest macOS, latest Xcode, latest VS2022, latest BouncyCastle (2.3.0), latest downloaded-n-built openssl (3.2.1), latest liboqs (0.10.0-dev, just sync'ed to my fork today), and latest oqs-provider (0.5.4-dev, sync'ed to my fork today). The incredible @baentsch has been busy and just added padding for falcon so I'm truly on the very edge :)

[cipherboy]

Many thanks for this detailed reproducer @andybrucenet! I should've been more patient but I wanted to make sure you saw the missing attachment -- I didn't know that discussions could have attachments too, so was worried you thought it attached but it failed. :-)

So... I have some good news, some bad news, and a potential workaround.

Bad news is, I think I'll need to discuss this with @peterdettman... :-)

As I understand it, the issue is that BC C# is attempting to be helpful and parse your final private key into a concrete typed key for you. The issue is we have a classical/PQC factory split, wherein PrivateKeyFactory.CreateKey can't handle PQCs. This is what is called in the OpenSSL ReadPrivateKey(...) helper we have, and that gives your stack trace about unknown OIDs, even though your OIDs match our OIDs (\o/!).

I think you can workaround this with code like:

Pardon my messy code:

namespace foobar;

using System.IO;
using Org.BouncyCastle.Asn1;
using Org.BouncyCastle.Asn1.Cms;
using Org.BouncyCastle.Pqc.Crypto;
using Org.BouncyCastle.Pqc.Crypto.Utilities;

// read PEM files created using openssl / oqs-provider using BouncyCastle
class Program
{
    static void Main(string[] args)
    {
        var pemFiles = Directory.GetFiles("/tmp", "foobar-*.pem");
        foreach (var pemFile in pemFiles)
        {
            Console.Write($"{pemFile}: ");
            var theKeyStream = File.OpenText(pemFile);
            using (theKeyStream)
            {
                var theKeyPemReader = new Org.BouncyCastle.Utilities.IO.Pem.PemReader(theKeyStream);
                try
                {
                    var content = theKeyPemReader.ReadPemObject().Content;

                    var theKey = Org.BouncyCastle.Pqc.Crypto.Utilities.PqcPrivateKeyFactory.CreateKey(content);
                    if (!(theKey is null))
                    {
                        var kyberKey = theKey as Org.BouncyCastle.Pqc.Crypto.Crystals.Kyber.KyberPrivateKeyParameters;
                        var dilithiumKey = theKey as Org.BouncyCastle.Pqc.Crypto.Crystals.Dilithium.DilithiumPrivateKeyParameters;
                        if (kyberKey != null)
                        {
                            Console.Write("kyber public key: " + Org.BouncyCastle.Utilities.Encoders.Hex.ToHexString(kyberKey.GetPublicKey()));
                        }
                        else if (dilithiumKey != null)
                        {
                            Console.Write("dilithium public key: " + Org.BouncyCastle.Utilities.Encoders.Hex.ToHexString(dilithiumKey.GetPublicKey()));
                        }
                        else
                        {   
                            Console.Write("unknown key type: " + theKey.GetType());
                        }
                    }
                    else
                    {   
                        Console.Write("[null]");
                    }
                }
                catch (Exception ex)
                {   
                    Console.Write(ex.Message);
                }
            }
            Console.WriteLine();
            Console.WriteLine();
            Console.WriteLine();
        }
    }
}

Console output:

/tmp/foobar-p256_sphincssha2128ssimple.pem: Object reference not set to an instance of an object.


/tmp/foobar-dilithium2.pem: algorithm identifier in private key not recognised


/tmp/foobar-sphincssha2128fsimple.pem: private key encoding does not match parameters


/tmp/foobar-p384_sphincssha2192fsimple.pem: Object reference not set to an instance of an object.


/tmp/foobar-p256_falconpadded512.pem: algorithm identifier in private key not recognised


/tmp/foobar-mldsa87.pem: dilithium public key: c89c6022e9813d9cee65438e16870a98d3a3f3196362e3157d1ebbd30fa9fa58


/tmp/foobar-p384_mldsa65.pem: algorithm identifier in private key not recognised


/tmp/foobar-sphincssha2128ssimple.pem: private key encoding does not match parameters


/tmp/foobar-p384_dilithium3.pem: algorithm identifier in private key not recognised


/tmp/foobar-p256_mldsa44.pem: algorithm identifier in private key not recognised


/tmp/foobar-rsa3072_falcon512.pem: algorithm identifier in private key not recognised


/tmp/foobar-rsa3072_mldsa44.pem: algorithm identifier in private key not recognised


/tmp/foobar-rsa3072_falconpadded512.pem: algorithm identifier in private key not recognised


/tmp/foobar-sphincsshake128fsimple.pem: private key encoding does not match parameters


/tmp/foobar-sphincssha2192fsimple.pem: private key encoding does not match parameters


/tmp/foobar-p256_falcon512.pem: algorithm identifier in private key not recognised


/tmp/foobar-rsa3072_sphincsshake128fsimple.pem: Object reference not set to an instance of an object.


/tmp/foobar-p256_sphincsshake128fsimple.pem: Object reference not set to an instance of an object.


/tmp/foobar-mldsa65.pem: dilithium public key: d62490aa3be52a0913b6b790934ce5074a0895d4233d6ac19882be22e0ecf969


/tmp/foobar-dilithium5.pem: algorithm identifier in private key not recognised


/tmp/foobar-falconpadded512.pem: algorithm identifier in private key not recognised


/tmp/foobar-falconpadded1024.pem: algorithm identifier in private key not recognised


/tmp/foobar-p521_falconpadded1024.pem: algorithm identifier in private key not recognised


/tmp/foobar-p521_mldsa87.pem: algorithm identifier in private key not recognised


/tmp/foobar-rsa3072_sphincssha2128ssimple.pem: Object reference not set to an instance of an object.


/tmp/foobar-mldsa44.pem: dilithium public key: 89502a20a57c9efc592192cc83d5e95e761324efaedd758f23d2356b4e37f4c7


/tmp/foobar-p256_sphincssha2128fsimple.pem: Object reference not set to an instance of an object.


/tmp/foobar-p521_falcon1024.pem: algorithm identifier in private key not recognised


/tmp/foobar-falcon1024.pem: algorithm identifier in private key not recognised


/tmp/foobar-dilithium3.pem: algorithm identifier in private key not recognised


/tmp/foobar-p256_dilithium2.pem: algorithm identifier in private key not recognised


/tmp/foobar-p521_dilithium5.pem: algorithm identifier in private key not recognised


/tmp/foobar-rsa3072_dilithium2.pem: algorithm identifier in private key not recognised


/tmp/foobar-falcon512.pem: algorithm identifier in private key not recognised


/tmp/foobar-rsa3072_sphincssha2128fsimple.pem: Object reference not set to an instance of an object.

Note that I didn't build limiting to STD algorithms, but we don't support hybrid keys either, so I think these results are mostly expected. Not sure why I thought I saw mlkem, but on second look-through, I didn't, so 🤷!

p.s., everything built fine on Linux, untested on MacOS :-)

Here, we parse the private key semi-manually from the embedded PrivateKeyInfo, and then you'll have to deal with the difference between an AsymmetricCipherKeyPair and the AsymmetricKeyParameter conversion of your choice (either to there or to AsymmetricCipherKeyPair if you can derive the symmetric key -- note that in the ML-DSA case you can, but in the ML-KEM case, you can't if the public key info is withheld from the public struct (which I think may be possible based on encoding method?).

Now, the good news is, Java doesn't have this issue :-)

https://github.com/bcgit/bc-java/blob/main/pkix/src/main/java/org/bouncycastle/openssl/PEMParser.java#L572

It stops just short of being too helpful, and returns the PrivateKeyInfo ASN.1 structure, to let you parse it with the key factory of your choice (classical vs PQC)...

So, the question for @peterdettman is, would changes to support returning a raw PrivateKeyInfo (obviously, via a new method) be friendly?

HTH!

[baentsch]

The incredible @baentsch has been busy and just added padding for falcon

Nope -- this has been the incredible @SWilson4.