[bitnami/zookeeper] zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/users/ after update 3.1.2 to 3.6.0

[flashick]

Name and Version

bitnami/kafka:3.6.0

What architecture are you using?

amd64

What steps will reproduce the bug?

3 nodes with this docker-compose:

version: "2"
  
services:
  kafka:
    restart: unless-stopped
    image: "bitnami/kafka:3.6.0"
    container_name: kafka
    environment:
          - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-1:2181,zookeeper-2:2181,zookeeper-january-3:2181
          - KAFKA_BROKER_ID=3          
          - KAFKA_CFG_LOG_DIRS=/bitnami/kafka/data
          - KAFKA_CFG_LISTENERS=INTERNAL://:9092,CLIENT://:9093
          - KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka-3:9092,CLIENT://kafka-3:9093
          - KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
          - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:SASL_PLAINTEXT,CLIENT:SASL_PLAINTEXT
          - KAFKA_CFG_SECURITY_PROTOCOL=SASL_PLAINTEXT
          - KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=PLAIN
          - KAFKA_CLIENT_USERS=user_kafka
          - KAFKA_CLIENT_PASSWORDS=user_kafka_password
          - KAFKA_ZOOKEEPER_USER=zoo-user
          - KAFKA_ZOOKEEPER_PASSWORD=zoo-pass
          - KAFKA_ZOOKEEPER_PROTOCOL=SASL
    volumes:
      - /data/kafka:/bitnami/kafka
    ports:
      - "9092:9092"
      - "9093:9093"

Docker-compose for zookeeper (3 nodes too):

version: '3'
services:
  zookeeper:
    restart: unless-stopped
    image: "bitnami/zookeeper:3.8.1"
    container_name: zookeeper
    environment:
      - ZOO_SERVER_ID=3
      - ZOO_SERVERS=zookeeper-1:2888:3888,zookeeper-2:2888:3888,0.0.0.0:2888:3888
      - ZOO_ENABLE_AUTH=yes
      - ZOO_SERVER_USERS=zoo-user,user_kafka
      - ZOO_SERVER_PASSWORDS=zoo-pass,user_kafka_password
      - ZOO_CLIENT_USERS=user_kafka
      - ZOO_CLIENT_PASSWORDS=user_kafka_password
      - ZOO_4LW_COMMANDS_WHITELIST=*
      - KAFKA_OPTS= -Dzookeeper.4lw.commands.whitelist=*
    volumes:
      - /data/zookeeper/:/bitnami/zookeeper:rw
    ports:
      - "2181:2181"
      - "2888:2888"
      - "3888:3888"

With that config kafka starts normally.

Now I change the version to 3.6.0 or 3.5.1 in docker-compose.yml and do:

docker-compose down
docker compose up -d

What is the expected behavior?

New version kafka launches

What do you see instead?

Container restarts with this error:

[2023-12-04 10:32:39,743] ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/users/user_kafka

Additional information

I want to upgrade to a new version (one broker at a time) to stop using the zookeeper.
The bitnami documentation (https://github.com/bitnami/containers/tree/main/bitnami/kafka) says to just use a new image, but kafka documentation(https://kafka.apache.org/36/documentation/streams/upgrade-guide) says additional steps need to be taken.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[aoterolorenzo]

Hi @flashick,

Let me check the issue locally

[aoterolorenzo]

As mentioned in the README, there was several breaking changes introduced in version 3.5.1, so by definition a image upgrade is not enough for jump to 3.5.1 in above. Let me check with some colleagues and see if we can figure out a solution that could fit your scenario. I will reach back to you.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[tecris]

Hi @aoterolorenzo

First of all, a great thank you to bitnami team for this great work 🙇

In regards to this issue, maybe following could help:

1. As mentioned in the README, The kafka_jaas.conf will no longer be generated, although it will continue being loaded if mounted
2. From Kafka documentation Note that ZooKeeper JAAS config may only be configured using static JAAS configuration.

I could be wrong, even if we mount jaas config file (/opt/bitnami/kafka/config/kafka_jaas.conf) problem still persists.
If we compare a version from Jan 2023 kafka_create_sasl_scram_zookeeper_users with Mar 2024 kafka_create_sasl_scram_zookeeper_users we see line
export KAFKA_OPTS="-Djava.security.auth.login.config=${KAFKA_CONF_DIR}/kafka_jaas.conf" was removed.

thanks

[aacoliveira]

For those who are in this situation there is a possible workaround for this error. Probably not the optimal but it works.

Before the update process add the flag "-Dzookeeper.skipACL=yes" to the JVM of Zookeeper.

After that, simple restart zookeeper followed by kafka's broker image update and restart.

This way Kafka can create the user and the startup process works fine. Hope it helps !