Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
---
title: Achieving atomic inserts and multi-table consistency in ClickHouse Cloud
description: How to load data atomically and keep multiple tables consistent in ClickHouse Cloud without multi-statement transactions, using staging tables and partition-level operations.
date: 2026-06-01
tags: ['Data Ingestion', 'Best Practices']
keywords: ['atomic insert', 'staging table', 'MOVE PARTITION', 'ATTACH PARTITION', 'REPLACE PARTITION', 'multi-table consistency']
---

{frontMatter.description}
{/* truncate */}

## Problem {#problem}

ClickHouse Cloud does not support multi-statement transactions in the traditional RDBMS sense.
This creates two common challenges:

1. Single-table atomicity for bulk loads: A common approach is to insert into temporary partial keys, then copy records to the actual key and delete the temporary records. This approach performs poorly — particularly the delete step, which can consume over 90% of the total operation time.
2. Multi-table consistency: When a pipeline loads Table A successfully but fails on Table B, Table A is already committed and cannot be rolled back. Analysts querying across both tables see data that is out of sync.

## Background {#background}

ClickHouse guarantees atomicity at the single-insert, single-partition level: if an `INSERT` succeeds, all rows in that block are visible; if it fails, none are. However, there is no built-in mechanism to atomically commit data across multiple inserts or multiple tables.

The partition manipulation commands ([`MOVE PARTITION TO TABLE`](/core/reference/statements/alter/partition#move-partition-to-table), [`REPLACE PARTITION`](/core/reference/statements/alter/partition#replace-partition), [`ATTACH PARTITION FROM`](/core/reference/statements/alter/partition#attach-partition-from)) operate at the metadata level when the source and destination tables share the same storage policy.

This means they execute almost instantaneously regardless of data size, making them ideal building blocks for atomic-swap patterns.

## Recommended solution {#recommended-solution}

Instead of inserting directly into production tables and attempting to clean up on failure, use dedicated staging tables as a landing zone. After validating the data, use partition-level operations to atomically promote the data into production.

## Step-by-step for single-table atomicity {#step-by-step}

<Steps>
<Step>
### Create a staging table {#create-staging-table}

Use the same schema, partition key, `ORDER BY`, and storage policy as the production table.

```sql
CREATE TABLE my_table_staging AS my_table_prod;
```
</Step>
<Step>
### Insert data into the staging table {#insert-into-staging}

Run your insert against the staging table instead of the production table.

```sql
INSERT INTO my_table_staging SELECT ... FROM source;
```
</Step>
<Step>
### If the insert fails, truncate and retry {#on-failure}

Truncate the staging table and run the load again. No production data has been affected.

```sql
TRUNCATE TABLE my_table_staging;
```
</Step>
<Step>
### If the insert succeeds, move the partitions to production {#on-success}

To move the data into production and remove it from the staging table, use `MOVE PARTITION`.

```sql
ALTER TABLE my_table_staging MOVE PARTITION <partition_expr> TO TABLE my_table_prod;
```

Alternatively, copy the data into an existing partition in production with `ATTACH PARTITION`.

```sql
ALTER TABLE my_table_prod ATTACH PARTITION tuple() FROM my_table_staging;
```

Both operations are metadata-level changes on the same storage policy and complete almost instantaneously.
</Step>
<Step>
### Clean up the staging table {#clean-up}

Once all partitions have been moved, truncate the staging table to leave it empty for the next load.

```sql
TRUNCATE TABLE my_table_staging;
```
</Step>
</Steps>

## Multi-table consistency {#multi-table-consistency}

The same pattern solves pipelines where two or more tables must all be loaded before any of them becomes visible to analysts. Load each table's data into its own staging table and validate all of them, then run the partition moves together. Because each move is a near-instant metadata operation, the window in which the tables disagree shrinks from the duration of the full load to the time it takes to swap partitions.

## Requirements and constraints {#requirements-and-constraints}

For `MOVE PARTITION TO TABLE`, `REPLACE PARTITION`, and `ATTACH PARTITION FROM`, the source and destination tables must have:

- The same column structure
- The same partition key, `ORDER BY` key, and primary key
- The same storage policy
- The destination table must include all indices and projections from the source table

## Example {#example}

You can run the example below interactively using [fiddle](https://fiddle.clickhouse.com/7ef9ed84-ac14-4f2c-9ca5-d5913089769a):

```sql
CREATE TABLE prod
(
uid Int16,
name String,
age Int16
)
ENGINE=MergeTree
ORDER BY ();

CREATE TABLE staging
(
uid Int16,
name String,
age Int16
)
ENGINE=MergeTree
ORDER BY ();

-- Initial data
INSERT INTO prod VALUES (123, 'John', 33);
INSERT INTO prod VALUES (456, 'Ksenia', 48);
-- Load data
INSERT INTO staging VALUES (8811, 'Alice', 50);
INSERT INTO staging VALUES (8812, 'Bob', 23);

-- Validate import
SELECT 'Staging count:', COUNT() FROM staging;
-- Move partition
ALTER TABLE staging MOVE PARTITION tuple() TO TABLE prod; -- atomic op

-- Check data
SELECT 'Prod count:', COUNT() FROM prod;
SELECT * FROM prod;
```

## References {#references}

- [**Manipulating Partitions and Parts**](/core/reference/statements/alter/partition)
- To read more about this strategy, see the blog post [**Supercharging your large ClickHouse data loads - Part 3: Making a large data load resilient**](https://clickhouse.com/blog/supercharge-your-clickhouse-data-loads-part3).
1 change: 1 addition & 0 deletions resources/support-center/navigation.json
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@
{
"group": "Data Import & Export",
"pages": [
"resources/support-center/knowledge-base/data-import-export/achieving-atomic-inserts",
"resources/support-center/knowledge-base/data-import-export/cannot-append-data-to-parquet-format",
"resources/support-center/knowledge-base/data-import-export/file-export",
"resources/support-center/knowledge-base/data-import-export/importing-geojason-with-nested-object-array",
Expand Down