Error authenticating with Kerberos and arcfour-hmac

[rafaelmnatali]

Hello,

I'm trying to configure the Redpanda Console to authenticate with my Kafka Broker but, getting some encryption errors.

I'm using keytabs to authenticate to the Brokers. Currently, I'm using keytabs for Kafka Connect, Kstreams, and ksql.

All the applications that currently work are Java. Redpanda is the first Go app I'm trying to integrate with Kerberos.

Using the same krb5.conf that I use for other applications I have the following error:

{"level":"info","ts":"2022-12-19T09:02:34.332Z","msg":"started Redpanda Console","version":"v2.1.1","built_at":"1669902350"}
{"level":"info","ts":"2022-12-19T09:02:34.334Z","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata"}
{"level":"error","ts":"2022-12-19T09:02:34.347Z","msg":"unable to initialize sasl","source":"kafka_client","broker":"seed 0","err":"could not get valid TGT for client's realm: [Root cause: Encrypting_Error] KRBMessage_Handling_Error: AS Exchange Error: failed setting AS_REQ PAData for pre-authentication required < Encrypting_Error: error getting key from credentials: matching key not found in keytab. Looking for \"<redacted_username>\" realm: <redacted_realm> kvno: 0 etype: 18"}
{"level":"warn","ts":"2022-12-19T09:02:34.347Z","msg":"Failed to test Kafka connection, going to retry in 1s","remaining_retries":5}

krb5.conf

   [logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

    [libdefaults]
    dns_lookup_realm = false
    ticket_lifetime = 10h
    renew_lifetime = 7d
    forwardable = true
    rdns = false
    default_realm = <redacted>
    default_ccache_name = FILE:/tmp/krb5cc_%{uid}

    [realms]
    <redacted> = {
      kdc = <redacted>
      kdc = <redacted>
      kdc = <redacted>
      kdc = <redacted>
      admin_server = <redacted>
    }

    [domain_realm]
    .<redacted>

    # ignore_k5login = true : Never look for a .k5login file in the user's home directory. Instead, only check that the Kerberos principal maps to the local account name.
    [appdefaults]
    pam = {
      <realm> = {
        ignore_k5login = true
      }
    }

Troubleshooting

I added the Kerberos client to the image and run some kerberos commands to see if the keytab was ok in the Pod:

/app $ ls
console
/app $ klist -ket /keytabs/<redacted>.keytab
Keytab name: FILE:/keytabs/<redacted>.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 03/08/21 11:00:58 <redacted_username>@<redacted_realm> (DEPRECATED:arcfour-hmac)

/app $ kinit -fV -k -t /keytabs/<redacted>.keytab <redacted_username>
Using default cache: /tmp/krb5cc_99
Using principal: <redacted_username>@<redacted_realm>
Using keytab: /keytabs/<redacted>.keytab
Authenticated to Kerberos v5

after reading other issues here, I try to add preferred_preauth_types = 23, default_tkt_enctypes = arcfour-hmac, and default_tgs_enctypes = arcfour-hmac to the `krb5.conf. Now, I have the following error:

{"level":"error","ts":"2022-12-19T11:04:27.318Z","msg":"unable to initialize sasl","source":"kafka_client","broker":"seed 0","err":"could not get valid TGT for client's realm: [Root cause: Decrypting_Error] KRBMessage_Handling_Error: AS Exchange Error: AS_REP is not valid or client password/keytab incorrect < Decrypting_Error: error decrypting EncPart of AS_REP < Decrypting_Error: error decrypting AS_REP encrypted part: matching key not found in keytab. Looking for [<redacted_username>] realm: <redacted_realm> kvno: 8 etype: 23"}

I see that the kvno for my keytab is 1 and the kvno when using type=23 is 8. Not sure if this has any relation.

If there is any comments/suggestion in how to proceed from here I really appreciate.

Thanks!

[MikhailMS]

@rafaelmnatali I got a PR where arcfour-hmac issue is resolved - you can try to apply it to your code and let me know it it works for you :)
I hope it would convince @jcmturner that PR is something worth including into next release

Regarding kvno - while it is not checked in other implementations (ie Java), because RFCs of Kerberos protocol don't really enforces you to do so, this package does check that, so you would need to regenerate your keytab with expected kvno value

[rafaelmnatali]

Thanks @MikhailMS for your feedback. I'm seeing this error in a third-party tool I'm trying to use. Will reach out to them with this fix.

Meanwhile, I think we can close this ticket.