Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[controller] Make Helix rebalance preferences and capacity keys configurable for the controller #1475

Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open

[controller] Make Helix rebalance preferences and capacity keys configurable for the controller #1475

wants to merge 18 commits into from

Conversation

kvargha
Copy link
Contributor

@kvargha kvargha commented Jan 27, 2025 •
edited
Loading

[controller] Make Helix rebalance preferences and capacity keys configurable for the controller

We are migrating from SEMI_AUTO to FULL_AUTO Waged. After thorough testing, we discovered a config that achieves overall even distribution for resource assignment.

  1. Made the Helix rebalance preference configurable. This is helpful when using Waged, as we can configure it to priortize evenness.
  2. Made instance capacity keys configurable, allowing Helix to consider the current top-state when calculating assignments.
  3. To accommodate these configurations, I updated the logic of updateClusterConfigs to support more complex configurations beyond simple fields. This logic was also ported to updateRESTConfigs.
  4. Enabled PERSIST_BEST_POSSIBLE_ASSIGNMENT, as this is the setting we use for our controller clusters.

How was this PR tested?

Added unit and integration tests

Does this PR introduce any user-facing changes?

  • No. You can skip the rest of this section.
  • Yes. Make sure to explain your proposed changes and call out the behavior change.

@kvargha
[controller] Set rebalance preference to prioritize evenness when cre…
030564e 
…ating the controller cluster. Also set capacity keys to enable top-state even distribution
@nisargthakkar
Copy link
Contributor

Also, let's add some tests

@kvargha
Copy link
Contributor Author

kvargha commented Jan 29, 2025

Working on the tests rn

nisargthakkar
nisargthakkar reviewed Jan 31, 2025
View reviewed changes
Copy link
Contributor

@nisargthakkar nisargthakkar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of minor nits. LGTM

services/venice-controller/src/main/java/com/linkedin/venice/controller/ZkHelixAdminClient.java Outdated Show resolved Hide resolved
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java Outdated Show resolved Hide resolved
...e-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerClusterConfig.java Outdated Show resolved Hide resolved
@kvargha kvargha changed the title [controller] Set rebalance preference to prioritize evenness when creating the controller cluster. Also set capacity keys to enable top-state even distribution [controller] Make Helix rebalance preferences and capacity keys configurable for the controller Feb 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nisargthakkar nisargthakkar nisargthakkar requested changes

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kvargha @nisargthakkar