[bitnami/ksql] problem connecting to broker

[msillence]

Name and Version

bitnami/ksql:7.8.0-debian-12-r0

What architecture are you using?

amd64

What steps will reproduce the bug?

podman run -it --rm bitnami/ksql:7.8.0-debian-12-r0

I also have a docker compose file that I'm using to test but the missing jar files seem to be causing the failure and this is the simplest example

What is the expected behavior?

problem connecting to broker

What do you see instead?

a stack trace:

Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
        at io.confluent.ksql.rest.server.KsqlServerMain.<clinit>(KsqlServerMain.java:56)
Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
        ... 1 more

Additional information

adding in the slf4j-api-2.0.9.jar then subsequently fails with:

Caused by: java.lang.ClassNotFoundException: com.google.common.collect.ImmutableSet
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)

[msillence]

adding in guava for the com.google.common.collect.ImmutableSet failure then fails with:

Caused by: java.lang.ClassNotFoundException: org.apache.kafka.connect.data.Schema
       at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
       at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
       at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)

this one is present in /opt/bitnami/ksql/share/java/kafka/ the connect-api jar
adding /opt/bitnami/ksql/share/java/kafka/* to the classpath then fails with:

Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Priority
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)

[msillence]

I finally got it to run with the following additional jar files:

• slf4j-api-2.0.9.jar (org.slf4j:slf4j-api:2.0.9)
• guava-32.0.1-jre.jar (com.google.guava:guava:32.0.1-jre)
• reload4j-1.2.25.jar (ch.qos.reload4j:reload4j:1.2.25)

I also added the following class path (which is probably overkill):

KSQL_CLASSPATH=/opt/bitnami/ksql/share/java/acl/*:/opt/bitnami/ksql/share/java/confluent-common/*:/opt/bitnami/ksql/share/java/confluent-metadata-service/*:/opt/bitnami/ksql/share/java/confluent-security/*:/opt/bitnami/ksql/share/java/confluent-telemetry/*:/opt/bitnami/ksql/share/java/kafka/*:/opt/bitnami/ksql/share/java/ksqldb/*

I note that the run script /opt/bitnami/ksql/bin/ksql-run-class has some logic for adding in exta paths non of which exist in this image

for library in "confluent-common" "confluent-telemetry" "ksqldb-examples" "rest-utils" "ksqldb-engine" "ksqldb-rest-app" "ksqldb-cli" "ksqldb-functional-tests" "ksqldb" "monitoring-interceptors" "confluent-security/ksql" "ksqldb-tools"; do
  DIR="$base_dir/share/java/$library"
  if [ -d "$DIR" ]; then
    KSQL_CLASSPATH="$DIR/*:$KSQL_CLASSPATH"
  fi
done

we could in the same way add all the folders under /opt/bitnami/ksql/share/java/

KSQL_CLASSPATH=/opt/bitnami/ksql/share/java/acl/*:/opt/bitnami/ksql/share/java/confluent-common/*:/opt/bitnami/ksql/share/java/confluent-control-center/*:/opt/bitnami/ksql/share/java/confluent-hub-client/*:/opt/bitnami/ksql/share/java/confluent-metadata-service/*:/opt/bitnami/ksql/share/java/confluent-security/*:/opt/bitnami/ksql/share/java/confluent-telemetry/*:/opt/bitnami/ksql/share/java/kafka/*:/opt/bitnami/ksql/share/java/ksqldb/*:/opt/bitnami/ksql/share/java/rest-utils/*

the dockerfile https://github.com/bitnami/containers/blob/main/bitnami/ksql/7.8/debian-12/Dockerfile
I'm gussing that the interesting bit is the download of ksql tar file line 34
for me that just returns 404 curl -SsLf https://downloads.bitnami.com/files/stacksmith/ksql-7.8.0-0-linux-amd64-debian-12.tag.gz
I'm not sure where the repo is go generate the tar file though?

[msillence]

I think log4j 1.2 should really be removed as logging is two of the issues here it might be worth including that requires removing:

• jul-to-slf4j*.jar
• confluent-log4j-1.2*.jar
• kafka-log4j-appender*.jar

and then I add the following (for completeness):

• org.apache.logging.log4j:log4j-jul
• org.apache.logging.log4j:log4j-jcl
• org.apache.logging.log4j:log4j-api
• org.apache.logging.log4j:log4j-core
• org.apache.logging.log4j:log4j-slf4j2-impl
• org.apache.logging.log4j:log4j-1.2-api
• org.apache.logging.log4j:log4j-layout-template-json

it also needs the log4j.properties replacing with a log4j2 file

[msillence]

I would look at adding a PR if someone could point me to the thing that creates the tar file

[sillencem]

OK I've looked again and it seems the only thing needed is the classpath fix

KSQL_CLASSPATH=/opt/bitnami/ksql/share/java/acl/*:/opt/bitnami/ksql/share/java/confluent-common/*:/opt/bitnami/ksql/share/java/confluent-control-center/*:/opt/bitnami/ksql/share/java/confluent-hub-client/*:/opt/bitnami/ksql/share/java/confluent-metadata-service/*:/opt/bitnami/ksql/share/java/confluent-security/*:/opt/bitnami/ksql/share/java/confluent-telemetry/*:/opt/bitnami/ksql/share/java/kafka/*:/opt/bitnami/ksql/share/java/ksqldb/*:/opt/bitnami/ksql/share/java/rest-utils/*

the folder /opt/bitnami/ksql/share/java/acl contains a shaded jar with all the dependencies but the startup sequence of scripts aren't including it

it's a shame it's all in one jar as I really don't want to use log4j 1.2 but that's another issue

[alvneiayu]

hi @sillencem

Then, finally you have found a solution for your problem no? Can I help you with something else?

Thanks

Álvaro

[sillencem]

I think the start scripts need to be modified as the image doesn't work as it stands

I can probably create a PR, I think there is a path issue.

Do you know where the tar file is generated I'd be interested in reviewing that as well?

[alvneiayu]

@sillencem it would be wonderful to receive a PR with this. I invite you to do it.

Thanks a lot

Álvaro

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[sillencem]

I think this does need fixing
Also please could you point me to the repo that builds the zip file?