KAFKA-19471: Enable acknowledgement for a record which could not be deserialized #20148

DL1231 · 2025-07-11T01:20:22Z

This patch mainly includes two improvements:

Update currentFetch when pollForFetches() throws an exception.
Add an override KafkaShareConsumer.acknowledge(String topic, int partition, long offset, AcknowledgeType type) .

Reviewers: Andrew Schofield [email protected]

…eserialized

AndrewJSchofield

Thanks for the PR. I've only taken a very cursory look so far and have one comment. I'll review more deeply soon.

AndrewJSchofield · 2025-07-11T10:48:39Z

test-common/test-common-runtime/src/main/java/org/apache/kafka/common/test/ClusterInstance.java

+        return new KafkaShareConsumer<>(configs, null, null);
+    }
+
+    default <K, V> ShareConsumer<K, V> shareConsumer(Map<String, Object> configs, Deserializer<K> keyDeserializer, Deserializer<V> valueDeserializer) {
        Map<String, Object> props = new HashMap<>(configs);
        props.putIfAbsent(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());


If you're providing instances of the deserializers, I don't think you should be setting the class names in the config.

kafka/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java

Lines 743 to 744 in 2346c0e

if (keyDeserializer != null)

newConfigs.put(KEY_DESERIALIZER_CLASS_CONFIG, keyDeserializer.getClass());

Thanks for your review. During initialization, if the keyDeserializer is set, it will override the class name in the config.
Of course, the explicit check here makes the logic clearer. I'll fix it later.

AndrewJSchofield

Thanks for the PR. I think it's along the right lines, but the validation of the parameters is too loose as it stands. I suggest introducing some kind of exceptional batch that contains a single record, and then validating the parameters against this.

AndrewJSchofield · 2025-07-14T17:34:24Z

clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareFetch.java

+        for (Map.Entry<TopicIdPartition, ShareInFlightBatch<K, V>> tipBatch : batches.entrySet()) {
+            TopicIdPartition tip = tipBatch.getKey();
+            if (tip.topic().equals(topic) && (tip.partition() == partition)) {
+                tipBatch.getValue().addAcknowledgement(offset, type);


The checking here is not tight enough. If the user calls acknowledge(ConsumerRecord, AcknowledgeType), the code makes sure that the in-flight records include the offset (see ShareInFlightBatch.acknowledge(ConsumerRecord, AcknowledgeType)). However, in this case, there is no validation of the offset. The handling of an exceptional batch needs to be a bit more sophisticated I think. It should not be possible to use ShareConsumer.acknowledge(ConsumerRecord, AcknowledgeType) with an exceptional batch. It should only be possible to use ShareConsumer.acknowlege(String, int, long, AcknowledgeType) and only for the specific parameters that were retrieved from the RecordDeserializationException.

Good catch! Appreciate the suggestion.

AndrewJSchofield

Thanks for the updates. That's looking good now.

clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareFetch.java

clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareFetchException.java

AndrewJSchofield · 2025-07-18T15:45:06Z

clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareFetch.java

+            TopicIdPartition tip = tipBatch.getKey();
+            KafkaException shareException = tipBatch.getValue().getException();
+            if (tip.topic().equals(topic) && (tip.partition() == partition) &&
+                shareException instanceof RecordDeserializationException &&


I don't like the instanceof and then the cast. One way around this I suppose would be to ShareInFlightBatch have an exception which has a known type, ShareFetchException I guess. Then ShareFetchException could have methods which extract the topic, partition and offset from the RecordDeserialization exception, and then you just need shareException.offset() == offset). wdyt?

ShareFetchException was actually created to address the issue mentioned in the comments of KAFKA-19471 preventing exceptions from being swallowed. Therefore, the cause could be either a SerializationException or a CorruptRecordException. Consequently, if we use ShareFetchException, we might still need to perform instanceof checks followed by casting.

Yes, I was thinking about this case last night and you're absolutely correct. But still, ShareFetchException could perhaps be able to return an offset if appropriate rather than using instanceof and casting. There's got to be a more elegant, OO way of achieving this.

clients/src/main/java/org/apache/kafka/clients/consumer/KafkaShareConsumer.java

AndrewJSchofield · 2025-07-18T15:49:17Z

...nts-integration-tests/src/test/java/org/apache/kafka/clients/consumer/ShareConsumerTest.java

+
+            // Reject this record
+            shareConsumer.acknowledge(rde.topicPartition().topic(), rde.topicPartition().partition(), rde.offset(), AcknowledgeType.REJECT);
+            shareConsumer.commitSync();


And what happens if you repeat the acknowledge call after the commit?

Good catch! I have updated the patch.

AndrewJSchofield

Thanks for the PR. Looks good to me.

…eserialized (apache#20148) This patch mainly includes two improvements: 1. Update currentFetch when `pollForFetches()` throws an exception. 2. Add an override `KafkaShareConsumer.acknowledge(String topic, int partition, long offset, AcknowledgeType type)` . Reviewers: Andrew Schofield <[email protected]>

KAFKA-19471: Enable acknowledgement for a record which could not be d…

9bcb3d2

…eserialized

github-actions bot added triage PRs from the community core Kafka Broker consumer clients labels Jul 11, 2025

AndrewJSchofield added KIP-932 Queues for Kafka ci-approved and removed triage PRs from the community labels Jul 11, 2025

AndrewJSchofield requested changes Jul 11, 2025

View reviewed changes

address comment

4945615

AndrewJSchofield requested changes Jul 14, 2025

View reviewed changes

Enhance parameter validation.

e6bd21e

DL1231 requested a review from AndrewJSchofield July 17, 2025 00:46

AndrewJSchofield requested changes Jul 18, 2025

View reviewed changes

DL1231 added 2 commits July 19, 2025 11:17

address comment

2c46a3a

avoid using instanceof and casting

2cfecad

AndrewJSchofield approved these changes Jul 27, 2025

View reviewed changes

AndrewJSchofield merged commit abbb6b3 into apache:trunk Jul 27, 2025
23 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

KAFKA-19471: Enable acknowledgement for a record which could not be deserialized #20148

KAFKA-19471: Enable acknowledgement for a record which could not be deserialized #20148

Uh oh!

DL1231 commented Jul 11, 2025 •

edited by github-actions bot

Loading

Uh oh!

AndrewJSchofield left a comment

Uh oh!

AndrewJSchofield Jul 11, 2025

Uh oh!

DL1231 Jul 11, 2025

Uh oh!

AndrewJSchofield left a comment

Uh oh!

AndrewJSchofield Jul 14, 2025

Uh oh!

DL1231 Jul 15, 2025

Uh oh!

AndrewJSchofield left a comment

Uh oh!

Uh oh!

Uh oh!

AndrewJSchofield Jul 18, 2025

Uh oh!

DL1231 Jul 19, 2025

Uh oh!

AndrewJSchofield Jul 19, 2025

Uh oh!

Uh oh!

AndrewJSchofield Jul 18, 2025

Uh oh!

DL1231 Jul 19, 2025

Uh oh!

AndrewJSchofield left a comment

Uh oh!

Uh oh!

Uh oh!

	if (keyDeserializer != null)
	newConfigs.put(KEY_DESERIALIZER_CLASS_CONFIG, keyDeserializer.getClass());

KAFKA-19471: Enable acknowledgement for a record which could not be deserialized #20148

KAFKA-19471: Enable acknowledgement for a record which could not be deserialized #20148

Uh oh!

Conversation

DL1231 commented Jul 11, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AndrewJSchofield left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndrewJSchofield left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndrewJSchofield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndrewJSchofield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

DL1231 commented Jul 11, 2025 •

edited by github-actions bot

Loading