Skip to content

Latest commit

 

History

History
223 lines (181 loc) · 6.3 KB

rebalance.adoc

File metadata and controls

223 lines (181 loc) · 6.3 KB

AMQ Streams Rebalancing

For an introduction to this topic, we recommend reading the following blog and/or watching this video:

hq720

Enable Cruise Control

Add Cruise Control to the cluster configuration:

oc apply -f k8s/rebalance/01-cruise-control-metrics.yaml
oc patch kafka my-cluster --patch '{"spec":{"cruiseControl": {"metricsConfig":{"type":"jmxPrometheusExporter","valueFrom":{"configMapKeyRef":{"key":"metrics-config.yml","name":"cruise-control-metrics"}}}}}}' --type=merge

Simulate an unbalanced workload

To simulate an unbalanced workload, the following steps will configure the producer to write messages on the partitions which leader is hosted in the same Kafka server:

  1. Locate the partition leaders

    oc exec -it my-cluster-kafka-0 -- bin/kafka-topics.sh \
            --bootstrap-server my-cluster-kafka-bootstrap:9092 \
            --describe --topic event

  2. Update the producer configmap to select all the partitions which have the leaders located in the same server:

    oc edit configmap kafka-producer-config

  3. Update the following variables matching the desired partitions, e.g.:

      PRODUCER_PARTED: "true"
  PRODUCER_PARTITIONS: 0,3,6,9

  4. Bump the Kafka producer application:

    oc scale deployment/kafka-producer --replicas=0
oc scale deployment/kafka-producer --replicas=1

Monitor the cluster and engage the rebalancing

  1. Check that partitions grows at different paces, running the following command:

    oc exec -it my-cluster-kafka-0 -- bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic event

  2. Deploy the rebalance configuration which will trigger the optimization analysis:

    oc apply -f k8s/rebalance/02-kafka-rebalance-full.yaml

  3. Review the optimization proposal:

    oc describe kafkarebalance full-rebalance

  4. Approve the proposal

    oc annotate kafkarebalances.kafka.strimzi.io full-rebalance strimzi.io/rebalance=approve
    Tip

    It’s possible to trigger a new analysis on the existing rebalancing configuration:

    oc annotate kafkarebalances.kafka.strimzi.io full-rebalance strimzi.io/rebalance=refresh

  5. Rebalancing takes some time, you can check the progress through the following command:

    oc get kafkarebalance full-rebalance
    NAME             CLUSTER      PENDINGPROPOSAL   PROPOSALREADY   REBALANCING   READY   NOTREADY
full-rebalance   my-cluster                                     True

    Rebalancing is complete when the READY column displays True.

Analyze the optimization results

Run again the describe topic command, you should spot the overloaded partitions moved on different leaders:

oc exec -it my-cluster-kafka-0 -- bin/kafka-topics.sh \
            --bootstrap-server my-cluster-kafka-bootstrap:9092 \
            --describe --topic event

Scaling up and distribute the workload

When the Kafka cluster scales, existing topics do not leverage the newly available brokers, so they remain idle until new topics are created. The Cruise Control can be used to evenly distribute existing topics on the new available resources, as the following steps will show:

  1. Increase the Kafka replicas:

    oc patch kafka my-cluster --patch '{"spec":{"kafka": {"replicas": 4}}}' --type=merge

  2. Change the producer configurations to create an evenly distributed workload:

    oc edit configmap/kafka-producer-config

    Modify the environment variables:

      PRODUCER_PARTED: "false"
  PRODUCER_TICK_FREQUENCY: "10"

  3. Restart the producer and consumer application

  4. Open the Grafana dashboard, after a few minutes, the CPU graph should look like the following:

    Kafka CPU

    The new broker uses less resources.

  5. Watching at the topic information confirms that all partitions are on the first 3 brokers (0,1,2)

    oc exec -it my-cluster-kafka-0 -- bin/kafka-topics.sh \
            --bootstrap-server my-cluster-kafka-bootstrap:9092 \
            --describe --topic event

  6. Deploy the rebalance configuration mode: add-brokers, which is tailored to leverage the new available brokers:

    oc apply -f k8s/rebalance/03-kafka-rebalance-add-brokers.yaml

  7. Preparing an optimization proposal could take a couple of minutes, check the progress through the following command:

    oc get kafkarebalance add-brokers-rebalance
    NAME                    CLUSTER      PENDINGPROPOSAL   PROPOSALREADY   REBALANCING   READY   NOTREADY
add-brokers-rebalance   my-cluster                     True

    A proposal is available when the PENDINGPROPOSAL column displays True.

  8. Review and approve the optimization proposal:

    oc describe kafkarebalance add-brokers-rebalance
oc annotate kafkarebalances.kafka.strimzi.io add-brokers-rebalance strimzi.io/rebalance=approve

  9. Rebalancing takes a couple of minutes, you can monitor the Grafana dashboard to see the changes and the topic information to understand how the partitions and their replicas are reorganized across the brokers.

  10. Finally, you can ask the Cruise Control to shrink your partitions on less brokers and then scale the cluster down.

Clean up

In order to start the demo from scratch, with minimal effort: delete only the kafka broker and the topics:

oc delete kafkatopics --selector="strimzi.io/cluster=my-cluster"
oc delete kafka my-cluster

Drop the PVC:

oc delete pvc --selector="strimzi.io/cluster=my-cluster"

Delete kafka rebalance:

oc delete kafkarebalance full-rebalance
oc delete kafkarebalance add-brokers-rebalance
oc delete kafkarebalance remove-brokers-rebalance

In order to repeat the rebalancing demo, you have to create again the cluster and the topic.