Skip to content

Support for distributed migrations #130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

rcampos87
Copy link

Main motivation for this PR is to fix the handling of migrations performed by django through a load balancer, which can lead to inconsistent results if a clickhouse cluster with multiple nodes is behind a load balancer and round-robin is in effect. By making migrations distributed, all nodes are aware of the migration data and we can have much more consistent results when running manage.py migrate. It also makes the process of distributing migrations data automatic. (See discussion #114)

When having distributed_migrations and migration_cluster set, new distributed and local tables will be created for migrations, and all migration querysets will be routed to the distributed table.

In order to test the load balacing use case, a new docker compose service was added for HAProxy. For simplicity, already existent clickhouse nodes were used behind the HAProxy.

Example configuration would be

{
        "ENGINE": "clickhouse_backend.backend",
        "HOST": "load-balancer.dns",
        "PORT": 9004,
        ....
        "OPTIONS": {
            "distributed_migrations": True,
            "migration_cluster": "cluster",
         }
}

In my case, a clickhouse cluster with 3 nodes is behind an AWS ELB and everytime when running makemigrations or migrate, a different result could be achieved, and by using distributed migrations, all my issues were gone.

@jayvynl
Copy link
Owner

jayvynl commented Aug 22, 2025

Hi could you add some test for Undo a Migration That’s Already Been Applied

when distributed_migrations is on

@rcampos87
Copy link
Author

@jayvynl ok, tests added. When running the whole test suite, i ran into failures on the tests_datetime.py, perhaps due to my location.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants