Support partitions when tables are created and be able to replace a partition #1865
jherrmannNetfonds
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi everybody,
When ingesting data into a cloud data warehouse (BigQuery, Snowflake etc.) I want to partition my data (for example by execution_date), so that when I clear/rerun a DAG, only the data of the affected partition is replaced.
Currently Tables cannot automatically created with a partition. Maybe the
metadata
parameter of theTable
Class would be a option to add something like that. When I use the append operator I should be able to replace a partition.Are there any ideas into this direction yet?
I am not sure, what would be a good place to implement a "replacement of a partition", since the sdk has several operators which can store data into a cloud DWH, but with different option on how to store the data in the target table. For example has the load_file operator has more option to change the load behavior then a transform operator. At least as far as I understand.
Beta Was this translation helpful? Give feedback.
All reactions