Does createPartitioner property supports custom methods in confluent-kafka-javascript client ?

[vikas-goswami]

Hi team,

We would like to understand whether, createPartitioner supports custom methods in confluent-kafka-javascript client.
We are migrating to Javascript client from KafkaJS and using APIs under:

const { Kafka } = require("@confluentinc/kafka-javascript").KafkaJS;

From the migration documentation present here: reference to doc , I understood that customPartitioners are not allowed. By customPartitioner, I meant passing a method like this to the createPartitioner

producer = kafka().producer({ createPartitioner: customPartitioner })

where method is defined like this:

const customPartitioner = () => {
  const topicList = { ...... }
  return ({ topic, partitionMetadata, message }) => {
    if (partitionMetadata && partitionMetadata.length === 1) {
      return partitionMetadata[0].partitionId
    }
    if (topicList[topic] && (message.partitionNumber !== undefined && message.partitionNumber !== null)) {
      return message.partitionNumber
    }
    return 0
  }
}

But, while working on migration, I saw an error:

    KafkaJSError: The 'createPartitioner' property seems to be a KafkaJS property in the main config block. It must be moved to the kafkaJS block.
    Before: 
        const kafka = new Kafka({ ... });
        const producer = kafka.producer({ createPartitioner: <value>, ... });
    After: 
        const kafka = new Kafka({ ... });
        const producer = kafka.producer({ kafkaJS: { createPartitioner: <value>, ... }, ... });

      22 |       messages.push(message)
      23 |     }
    > 24 |     kafkaUtil.sendResolvedErrorsCountToKafka(messages, (err) => {
         |               ^
      25 |       if (err) reject(err)
      26 |       resolve()
      27 |     })

      at Kafka.producer (node_modules/@confluentinc/kafka-javascript/lib/kafkajs/_kafka.js:71:13)

which signifies that createPartitioner might be supported outside kafkaJS block. So, I have raised this query to confirm whether we can pass a custom method like above to the createPartitioner property.

We currently maintain a fork of kafkajs where we have implemented and rely on custom partitioning logic. However, we're planning to migrate to the official confluent-kafka-javascript client to align with Confluent's maintained ecosystem and benefit from the performance and feature set of librdkafka. This feature is important to us because it allows better control over partition assignment.

Thank you.

[milindl]

No, custom partitioners are not supported. There isn't a plan at the moment to support them - because of how they need to be called from within C, it would require some significant code changes.

You can, of course, specify the partition that you want to send the message to while using send(), directly, though that's not the same thing as a custom partitioner.

[vikas-goswami]

Got it. Thanks