[ruby/openssl] pkey: restore support for decoding "openssl ecparam -genkey" output

Scan through the input for a private key, then fallback to generic
decoder.

OpenSSL 3.0's OSSL_DECODER supports encoded key parameters. The PEM
header "-----BEGIN EC PARAMETERS-----" is used by one of such encoding
formats. While this is useful for OpenSSL::PKey::PKey, an edge case has
been discovered.

The openssl CLI command line "openssl ecparam -genkey" prints two PEM
blocks in a row, one for EC parameters and another for the private key.
Feeding the whole output into OSSL_DECODER results in only the first PEM
block, the key parameters, being decoded. Previously, ruby/openssl did
not support decoding key parameters and it would decode the private key
PEM block instead.

While the new behavior is technically correct, "openssl ecparam -genkey"
is so widely used that ruby/openssl does not want to break existing
applications.

Fixes ruby/openssl#535

ruby/openssl@d486c82833

Author: rhenium

ext/openssl/ossl_pkey.c

@@ -104,6 +104,42 @@ ossl_pkey_read_generic(BIO *bio, VALUE pass)
     /* Then check PEM; multiple OSSL_DECODER_from_bio() calls may be needed */
     if (OSSL_DECODER_CTX_set_input_type(dctx, "PEM") != 1)
         goto out;
+    /*
+     * First check for private key formats. This is to keep compatibility with
+     * ruby/openssl < 3.0 which decoded the following as a private key.
+     *
+     *     $ openssl ecparam -name prime256v1 -genkey -outform PEM
+     *     -----BEGIN EC PARAMETERS-----
+     *     BggqhkjOPQMBBw==
+     *     -----END EC PARAMETERS-----
+     *     -----BEGIN EC PRIVATE KEY-----
+     *     MHcCAQEEIAG8ugBbA5MHkqnZ9ujQF93OyUfL9tk8sxqM5Wv5tKg5oAoGCCqGSM49
+     *     AwEHoUQDQgAEVcjhJfkwqh5C7kGuhAf8XaAjVuG5ADwb5ayg/cJijCgs+GcXeedj
+     *     86avKpGH84DXUlB23C/kPt+6fXYlitUmXQ==
+     *     -----END EC PRIVATE KEY-----
+     *
+     * While the first PEM block is a proper encoding of ECParameters, thus
+     * OSSL_DECODER_from_bio() would pick it up, ruby/openssl used to return
+     * the latter instead. Existing applications expect this behavior.
+     *
+     * Note that normally, the input is supposed to contain a single decodable
+     * PEM block only, so this special handling should not create a new problem.
+     */
+    OSSL_DECODER_CTX_set_selection(dctx, EVP_PKEY_KEYPAIR);
+    while (1) {
+        if (OSSL_DECODER_from_bio(dctx, bio) == 1)
+            goto out;
+        if (BIO_eof(bio))
+            break;
+        pos2 = BIO_tell(bio);
+        if (pos2 < 0 || pos2 <= pos)
+            break;
+        ossl_clear_error();
+        pos = pos2;
+    }
+
+    OSSL_BIO_reset(bio);
+    OSSL_DECODER_CTX_set_selection(dctx, 0);
     while (1) {
         if (OSSL_DECODER_from_bio(dctx, bio) == 1)
             goto out;

test/openssl/test_pkey_ec.rb

@@ -199,6 +199,29 @@ def test_ECPrivateKey
     assert_equal pem, p256.export
   end
 
+  def test_ECPrivateKey_with_parameters
+    p256 = Fixtures.pkey("p256")
+
+    # The format used by "openssl ecparam -name prime256v1 -genkey -outform PEM"
+    #
+    # "EC PARAMETERS" block should be ignored if it is followed by an
+    # "EC PRIVATE KEY" block
+    in_pem = <<~EOF
+    -----BEGIN EC PARAMETERS-----
+    BggqhkjOPQMBBw==
+    -----END EC PARAMETERS-----
+    -----BEGIN EC PRIVATE KEY-----
+    MHcCAQEEIID49FDqcf1O1eO8saTgG70UbXQw9Fqwseliit2aWhH1oAoGCCqGSM49
+    AwEHoUQDQgAEFglk2c+oVUIKQ64eZG9bhLNPWB7lSZ/ArK41eGy5wAzU/0G51Xtt
+    CeBUl+MahZtn9fO1JKdF4qJmS39dXnpENg==
+    -----END EC PRIVATE KEY-----
+    EOF
+
+    key = OpenSSL::PKey::EC.new(in_pem)
+    assert_same_ec p256, key
+    assert_equal p256.to_der, key.to_der
+  end
+
   def test_ECPrivateKey_encrypted
     p256 = Fixtures.pkey("p256")
     # key = abcdef