Is there some way to get current lag of the current consumer group?

[jiangjinggit]

Is there some way to get current lag of the current consumer group?I want to get the lag of each partition under the consumption group.Our project relies heavily on this.Confluent Kafka go supports it.I think it is a very normal demand,thank you.

[achille-roussel]

Hello @jiangjinggit

We usually recommend using an out-of-band system like Burrow to report consumer group lag.

Would this be an acceptable solution to your problem? Or are there reasons why you would prefer to have partition lag reported in the consumer application?

[Jrmy2402]

Hello, I think it could be great to have the possibility of having the lag with the lib Kafka-go so that we can monitor the lag without another system. Because now for making this, we have to need to use another lib like Sarama or confluent-Kafka-go

[achille-roussel]

Hello @Jrmy2402, thanks for contributing to the discussion.

You can retrieve the list of committed consumer group offsets, and partition offsets through these two API calls:

• https://pkg.go.dev/github.com/segmentio/kafka-go#Client.OffsetFetch
• https://pkg.go.dev/github.com/segmentio/kafka-go#Client.ListOffsets

These are the building blocks upon which lag computation can be done, being the delta between the committed offset of a partition and its last offset.

I know this is not a fully packaged solution, I just wanted to point out the existing building blocks that are available to solve this problem.

If you think kafka-go should provide more support to solve these issues, would you have a suggestion of what that API would ideally look like to best suit your applications?

[jiangjinggit]

Thank you for your answer,I've used Sarama to get the lag,Thanks again!

[achille-roussel]

Is this solution satisfying to you? In which case I will close the ticket, otherwise please let me know what alternative you would prefer.

[ys93239]

Thank you for your answer,I've used Sarama to get the lag,Thanks again!

@jiangjinggit Did you have to move over to Sarama entirely or were you able to use it with segmentio somehow?

[jiangjinggit]

I just use Sarama for lag acquisition, and I still use segmentio for other functions

[achille-roussel]

Hello @jiangjinggit,

would you be able to share more details about how you're using Sarama for lag computation?

[prakharrke]

Upon calling these API, I get an error "fmt.WrapError" Unexpected EOF.
Could you please help me understand that better?

[sansmoraxz]

You can retrieve the list of committed consumer group offsets, and partition offsets through these two API calls:

• https://pkg.go.dev/github.com/segmentio/kafka-go#Client.OffsetFetch
• https://pkg.go.dev/github.com/segmentio/kafka-go#Client.ListOffsets

These are the building blocks upon which lag computation can be done, being the delta between the committed offset of a partition and its last offset.

I know this is not a fully packaged solution, I just wanted to point out the existing building blocks that are available to solve this problem.

If you think kafka-go should provide more support to solve these issues, would you have a suggestion of what that API would ideally look like to best suit your applications?

It would be nice if reader had a direct implementation of this. Like as a map for each topic and partition when backed by a consumer group.

[zouyaowen]

Is there any API that can get consumer-group through topic?The API Client.ListGroups just get all consumer group,but I don't know the consumer relation between consumer group and topic?Any suggestions?