`DataFrame.to_gbq` with `if_exists="replace"` and `clustering_columns` can swap data between columns if order is different

[jbandoro]

Environment details

Python: 3.12.10 (main, Apr  9 2025, 03:49:38) [Clang 20.1.0 ]
bigframes==2.37.0
google-cloud-bigquery==3.40.1
pandas==3.0.1
pyarrow==23.0.1

Steps to reproduce

1. Create a table with Dataframe.to_gbq with clustering_columns
2. Reorder the clustering column and another column that has the same type
3. Write the updated dataframe to the same table with if_exists="replace"
4. Inspect the table to see columns with different data than in dataframe

Code example

import bigframes.pandas as bpd
import pandas as pd

TABLE_ID = "my_project.my_dataset.test_table"

df = pd.DataFrame(
    {"a": [10, 1, 2], "b": ["Justin", "Alex", "Jim"], "c": ["cat", "dog", "cat"]}
)
bdf = bpd.DataFrame(df)

# First time writing, all good
bdf.to_gbq(TABLE_ID, clustering_columns=["b"])

# Reorder the clustered column b and column c
bdf = bdf[["a", "c", "b"]]

# When replacing and writing, the b and c columns are swapped
bdf.to_gbq(TABLE_ID, if_exists="replace", clustering_columns=["b"])

While the dataframe looks good, the updated table now has columns b/c data swapped:

This doesn't happen if clustering_columns is not used in my testing.

[chelsea-lin]

Hi @jbandoro, thank you for reporting this. We expect a fix to be included in our next release, likely coming out next week. Please let us know if you have any further questions. Thanks!