[SPARK-53927][BUILD][DSTREAM][4.0] Upgrade kinesis client and fix kinesis integration tests

[vrozov]

What changes were proposed in this pull request?

Backport of #52630 to 4.0

Why are the changes needed?

branch-4.0 is subject to the same error as master. The integration with kinesis does not work due to dependencies conflict.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Using AWS credentials and ENABLE_KINESIS_TESTS=1 build/mvn test -Pkinesis-asl -pl connector/kinesis-asl

Was this patch authored or co-authored using generative AI tooling?

No

[vrozov]

@sarutak Please review.

[sarutak]

@vrozov

WithAggregationKinesisBackedBlockRDDSuite doesn't seem pass.

[info] WithAggregationKinesisBackedBlockRDDSuite:
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "kpl-daemon-0000" java.lang.NoSuchFieldError: AWS_ACCOUNT_ID
	at software.amazon.awssdk.auth.credentials.internal.SystemSettingsCredentialsProvider.resolveCredentials(SystemSettingsCredentialsProvider.java:52)
	at software.amazon.awssdk.auth.credentials.AwsCredentialsProvider.resolveIdentity(AwsCredentialsProvider.java:54)
	at software.amazon.awssdk.identity.spi.IdentityProvider.resolveIdentity(IdentityProvider.java:60)
	at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:109)
	at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
	at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:129)
	at software.amazon.kinesis.producer.Daemon.makeSetCredentialsMessage(Daemon.java:563)
	at software.amazon.kinesis.producer.Daemon.startChildProcess(Daemon.java:432)
	at software.amazon.kinesis.producer.Daemon.access$100(Daemon.java:60)
	at software.amazon.kinesis.producer.Daemon$1.run(Daemon.java:129)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Also, please add the target branch to the title like [4.0] for backporting PRs next time.

[dongjoon-hyun]

Sorry but we don't expect this kind of dramatic dependency changes on the release branches, @vrozov .

[vrozov]

@dongjoon-hyun This is test only dependency and the old dependency does not work on 4.x.

[sarutak]

@dongjoon-hyun
Do you have any concern even though amazon-kinesis-producer is a test dependency and affects only kinesis-asl?

[vrozov]

@dongjoon-hyun @sarutak Note that the entire impact of the change is limited to KPLBasedKinesisTestUtils.scala and it is necessary due to other Spark dependencies being upgraded making it incompatible with the existing Kinesis producer library version.

[dongjoon-hyun]

We had better focus on delivering Apache Spark 4.1.0

• https://spark.apache.org/versioning-policy.html
  • November 1st 2025 Code freeze. Release branch cut.
  • November 15th 2025 QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged.
  • November 23th 2025 Release candidates (RC), voting, etc. until final release passes

[vrozov]

@dongjoon-hyun The problem is that Kinesis integration is broken on branch-4.0 and my testing also shows that it is broken on branch-3.5 as well (double checking). The issue impacts not only tests. Running tests against actual cluster or demo application do not work.

Sorry, I don't fully understand how fixing this regression bug on branch-4.0 impacts Spark 4.1 release.

[vrozov]

@sarutak Please take a look. Spark SBT build required a workaround.

[sarutak]

Why do we need to add these dependency for 4.0 even though they are not needed for 4.1?

[vrozov]

@sarutak @dongjoon-hyun The issue is caused by 4.0 dependency on hadoop-aws:3.4.1 that was upgraded to 3.4.2 in 4.1. hadoop-aws:3.4.1 has transitive dependency on AWS Java SDK v2 2.24.6 that is not compatible with Kinesis Producer transitive dependency on AWS Java SDK v2 2.29.24. In general, Spark should be enforcing specific version of AWS Java SDK v2 (the one specified for aws.java.sdk.v2.version) instead of relying on transitive dependencies that may not be compatible with each other.

Note that this is not an issue for maven build as it correctly handles transitive dependencies, and causes the issue that you noticed in SBT.