Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use offset from WAL to resume processing after replay #382

Open
xdevxy opened this issue Nov 22, 2022 · 0 comments
Open

Use offset from WAL to resume processing after replay #382

xdevxy opened this issue Nov 22, 2022 · 0 comments
Labels
area/reduce Reduce operations like GroupByKey enhancement New feature or request

Comments

@xdevxy
Copy link
Contributor

xdevxy commented Nov 22, 2022

Summary

As we have persist message offset information in PBQ WAL, it would be a clean design to resume the processing from the last offset that we replayed in case there are processed message that are not persisted yet before pod failover.

Use Cases

Reducer pod failover.

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.
@xdevxy xdevxy added enhancement New feature or request area/reduce Reduce operations like GroupByKey labels Nov 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/reduce Reduce operations like GroupByKey enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@xdevxy