java.lang.NoSuchMethodError: 'void com.amazonaws.util.IOUtils.closeQuietly

[umesh2611]

Describe the bug

Describe the bug
Hi,

I am following the link : https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html to send a kafka message to the MSK cluster ( provisioned / IAM )

When i am sending the message i get the following error :

2023-01-10T14:05:04.035+05:30

Copy
2023-01-10 08:35:04,033 - ERROR - kafka-producer-network-thread | producer-1 - org.apache.kafka.common.utils.KafkaThread - Uncaught exception in thread 'kafka-producer-network-thread | producer-1':

2023-01-10T14:05:04.035+05:30 java.lang.NoSuchMethodError: 'void com.amazonaws.util.IOUtils.closeQuietly(java.io.Closeable, org.apache.commons.logging.Log)'
at software.amazon.msk.auth.iam.internals.UserAgentUtils.getLibraryVersion(UserAgentUtils.java:73)
at software.amazon.msk.auth.iam.internals.UserAgentUtils.getLibraryVersion(UserAgentUtils.java:73)
at software.amazon.msk.auth.iam.internals.UserAgentUtils.(UserAgentUtils.java:43)
at software.amazon.msk.auth.iam.internals.UserAgentUtils.(UserAgentUtils.java:43)

complete log :

https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logsV2:log-groups/log-group/$252Faws$252Fecs$252FTmf628NormalizationUPALIWAL/log-events/ecs$252Ftmf628NormalizationContainer2$252F798843bb106f4e5a8fe41e423673e0a5

What i analyzed, the IOUtils pacakged in com.amazonaws.aws-java-sdk-bundle-1.12.262.jar contains a method with the following signature :

public static void com.amazonaws.util.IOUtils.closeQuietly(java.io.Closeable,com.amazonaws.thirdparty.apache.logging.Log)

but "aws-msk-iam-auth" is expecting a method with the following signature :
'void com.amazonaws.util.IOUtils.closeQuietly(java.io.Closeable, org.apache.commons.logging.Log)'

on analysis, I found that there are 2 IOUtils class from core and bundle jar.
the signature on the core jar is crrect but the bundle contains public static void com.amazonaws.util.IOUtils.closeQuietly(java.io.Closeable,com.amazonaws.thirdparty.apache.logging.Log)

Can you pease help me on moving ahead ?

/Umesh Paliwal

Expected Behavior

NA

Current Behavior

NA

Reproduction Steps

NA

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.17.295

JDK version used

11

Operating System and version

linux

[debora-ito]

That's because the aws-java-sdk-bundle includes all AWS services, their dependent jars and third-party libraries in one standalone jar. For this to work, dependencies had to be shaded, that's why you see the Log class in a different package name.

Instead of using the bundled jar, we recommend to add as a dependency only the services you are going to use in you project. For more info see - https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-project-maven.html#configuring-maven

[umesh2611]

Thanks for your reply.

Just to give more insight, we are using AWS SDK Java 2 .

But when we add AWS Hadoop library with the version 3.3.4, the SDK Bundle of aws sdk java 1 gets added.

(dependency tree )

[info] | +-org.apache.hadoop:hadoop-aws:3.3.4
[info] | | +-com.amazonaws:aws-java-sdk-bundle:1.12.262
[info] | | +-org.wildfly.openssl:wildfly-openssl:1.0.7.Final

Can you help us on this

[debora-ito]

Since you can't change the libs in your dependency list, you need to make sure that the version of the Java SDK dependencies match the version expected by the version of aws-msk-iam-auth you are using, and that the not-bundled IOUtils class is in your classpath at runtime.

[sangeethsasidharan]

@umesh2611 have you able to make any resolution here , I am facing the same issue when trying to connect to spark application to msk cluster

[safwank]

I managed to resolve this by

• excluding com.amazonaws:aws-java-sdk-bundle from org.apache.hadoop:hadoop-aws
• explicitly adding com.amazonaws:aws-java-sdk-s3 which is all that's required by hadoop-aws apparently, at least in my case

[debora-ito]

@umesh2611 I'm going to move this to the Discussions tab, since it's a usage question and not really a bug.

[13janderson]

I encountered this same error when using the latest version of the JDBC connector to communicate with Athena via python Glue. Using v3.50 of the connector: https://docs.aws.amazon.com/athena/latest/ug/jdbc-v3-driver.html as an included jar in my glue job with glue version 5. I had the exact same namespace clash and resolved this by shading the namespace software.amazon.awssdk for com.example.athena.shaded.software.amazon.awssdk. The .jar was distributed with its dependencies, and this was one of them, shading replaces all references of this namespace with the new alias that we specify. Thus, allowing all existing bundled up code that used anything in software.amazon.awssdk to still use it. Crucially though, when glue v5 includes anything else in the classpath with the same namespaces/methods as this, there are no clashes any more which was causing the issue in the first place.

[13janderson]

Following a request for the exact instructions I followed to achieve this, I'm putting them here for anyone else who may find them useful:

The following steps work for athena-jdbc-3.5.0-with-dependencies.jar.

To get v3 of the driver to work, I had to perform "shading" on the provided .jar file.
What shading achieves:
Shading packages dependency classes into a single JAR under a new namespace (e.g., software.amazon.awssdk to com.example.athena.shaded.software.amazon.awssdk).
This isolates dependencies, preventing conflicts with other versions of the same libraries in the classpath.

Why shading is needed here:
The athena-jdbc JAR includes an AWS SDK version with IoUtils.closeQuietly, but AWS Glue v5 uses a different AWS SDK version, which causes a namespace clash for this metho
There are then two versions of the same method with the same name. We only want to use the one included in the SDK within Glue already so we "shade" out the one
in the athena-jdbc-3.5.0-with-dependencies.jar file.

More context around shading:
so, all things previously under the namespace software.amazon.awssdk are now under the namespace com.example.athena.shaded.software.amazon.awssdk.
shading changes all existing references in other namspaces/classes in the original athena-jdbc-3.5.0-with-dependencies.jar file to now point to the
new namespace (com.example.athena.shaded.software.amazon.awssdk) so that all previous function calls still work.
but crucially, there is no namespace clash when our job is actually run and so we do not run into the previous issue described above.

Steps to reproduce this:

1. Install athena-jdbc-3.5.0-with-dependencies.jar to local Maven repository
   Command: mvn install:install-file -Dfile=libs/athena-jdbc-3.5.0-with-dependencies.jar -DgroupId=com.simba.athena -DartifactId=athena-jdbc -Dversion=3.5.0 -Dpackaging=jar

2. Create/update POM file (pom.xml) with:

• Standard dependency: com.simba.athena:athena-jdbc:3.5.0
• Maven Shade Plugin (version 3.2.4) to shade classes
• Relocate software.amazon.awssdk to com.example.athena.shaded.software.amazon.awssdk
• Exclude META-INF signature files (*.SF, *.DSA, *.RSA)
• Set Java 8 compatibility (source/target 1.8)

3. Build the shaded JAR

• Command: mvn clean package
• Output: target/athena-shaded-1.0-SNAPSHOT.jar

3a. Optionally, verify shaded JAR contains relocated classes

• Command: jar tf target/athena-shaded-1.0-SNAPSHOT.jar
• Check for: com/example/athena/shaded/software/amazon/awssdk/utils/IoUtils.class

4. Upload this new "shaded" jar file to s3 and copy the s3 URI of that jar file in the Depdendency jars of your glue job

The options specified in my Python Glue job were then as follows:

.option("driver", "com.amazon.athena.jdbc.AthenaDriver")
.option("CredentialsProvider", "DefaultChain")
.option("url", "jdbc:athena://athena.eu-west-1.amazonaws.com:443")
# Fully qualified path to view or table. Something like: Catalog.workspace.view_name
# NOTE: We need the single quotes surrounding this string - without this the
# internal string interpolation it does with this argument seemed to fail.
.option("dbtable", '"AwsDataCatalog"."".""')
# Need to provide S3 bucket for table/view to be read in to.
.option("S3OutputLocation","")

For the shading, I relied on the help of LLMs to get this done as this was an unfamiliar process to me. For anyone trying to replicate this, I recommend you do the same.

[andcau]

thank you very much