Example: Partition Reassignment in Seconds

This guide aims to walk you through verifying AutoMQ's second-level partition reassignment functionality. First, create a new Kafka topic with one partition and start a producer to write a specified amount of data into it. Then, reassign the partition between different AutoMQ brokers using the reassign command, and observe the time it takes to complete the partition reassignment.

After successfully installing AutoMQ via Cluster Deployment on Linux▸, you will obtain a list of Bootstrap Server addresses similar to the following format:

192.168.0.1:9092,192.168.0.2:9092

If you installed it through Deploy Locally▸, the Bootstrap Server addresses will be:

broker1:9092,broker2:9092

Throughout all steps, ensure you replace the Bootstrap Server addresses with the actual addresses you obtained.

Prerequisites

• Linux/Mac/Windows Subsystem for Linux

• Docker

If downloading the container image is slow, please refer to Docker Hub Mirror Configuration▸

Create Topic

CMD='docker run --network automq_net automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --create --topic reassign-topic --bootstrap-server broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

Send Message

Execute the following commands to send a certain amount of data:

CMD='docker run --network automq_net  automqinc/automq:latest /bin/bash -c  "/opt/kafka/kafka/bin/kafka-producer-perf-test.sh --topic reassign-topic --num-records=1024000 --throughput 5120 --record-size 1024 --producer-props bootstrap.servers=broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

View Partition Distribution

CMD='docker run --network automq_net  automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --topic reassign-topic --describe --bootstrap-server broker1:9092,broker2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

Reassign Partitions Among Brokers

echo '{
    "partitions": [
        {
            "topic": "reassign-topic",
            "partition": 0,
            "replicas": [
                2
            ]
        }
    ],
    "version": 1
}' > move.json && (CMD='docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server  broker1:9092,broker2:9092 --reassignment-json-file /move.json --execute"' ; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD) && rm move.json

Check if Reassignment Is Complete

echo '{
    "partitions": [
        {
            "topic": "reassign-topic",
            "partition": 0,
            "replicas": [
                2
            ]
        }
    ],
    "version": 1
}' > move.json &&(CMD=' docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server broker1:9092,broker2:9092 --reassignment-json-file /move.json --verify"' ; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD) && rm move.json

Create a Topic

CMD='docker run automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --create --topic reassign-topic --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

Send Messages

CMD='docker run automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-producer-perf-test.sh --topic reassign-topic --num-records=1024000 --throughput 5120 --record-size 1024 --producer-props bootstrap.servers=192.168.0.1:9092,192.168.0.2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

View Partition Distribution

CMD='docker run automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --topic reassign-topic --describe --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092"'; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD

Reassign Partitions Among Brokers

echo '{
    "partitions": [
        {
            "topic": "reassign-topic",
            "partition": 0,
            "replicas": [
                2
            ]
        }
    ],
    "version": 1
}' > move.json && (CMD='docker run -v $(pwd)/move.json:/move.json automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092 --reassignment-json-file /move.json --execute"' ; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD) && rm move.json

Sure, Please Provide the English Content You Want Me to Polish.

echo '{
    "partitions": [
        {
            "topic": "reassign-topic",
            "partition": 0,
            "replicas": [
                2
            ]
        }
    ],
    "version": 1
}' > move.json &&(CMD=' docker run -v $(pwd)/move.json:/move.json automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092 --reassignment-json-file /move.json --verify"' ; [ "$(uname)" = "Linux" ] && eval "sudo $CMD" || eval $CMD) && rm move.json