[Bug] Adding external Parquet data files to an existing Paimon table

[TejMakode1523]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

I’m evaluating Apache Paimon for migrating an existing data lake and would like to clarify the intended design and supported patterns around adding external Parquet data files into a Paimon table.

Any solution on it?

Compute Engine

NA

Minimal reproduce step

NA

What doesn't meet your expectations?

NA

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[steFaiz]

Hello, here's the docs about migration. You can also find how to migrate from hive there. Could you explain why you consider this a bug?

[TejMakode1523]

@steFaiz , Thanks for the response.

To clarify, my question is not about migration documentation, but specifically about loading data that already exists as Parquet files into a Paimon table.

I want to understand whether Apache Paimon provides any supported mechanism to add or register external Parquet files into an existing Paimon table (for example, similar to Hive external tables or Hudi bootstrap), without rewriting the data through a full Flink INSERT job.

If this is intentionally unsupported, it would be helpful to confirm explicitly that:

External Parquet files cannot be added or referenced directly by a Paimon table

The only supported way to load data is by reading the Parquet files and rewriting them through Paimon writers (Flink SQL / Table API / DataStream)

[steFaiz]

@TejMakode1523 Thanks for your explaination!

whether Apache Paimon provides any supported mechanism to add or register external Parquet files into an existing Paimon table (for example, similar to Hive external tables or Hudi bootstrap), without rewriting the data through a full Flink INSERT job.

The answer is: yes! Hive Migration and Iceberge Migration will only involves metadata copy and data file rename, which is much more lighter than rewriting full-table. You can refer to implementations of org.apache.paimon.migrate.Migrator if you are interested. But there are some limitations:

1. only support migrating to an APPEND table in paimon
2. the original hive/iceberg table will disappear
3. more

You can refer to the docs for details.

[TejMakode1523]

@steFaiz , Thanks for the clarification.
To confirm my understanding: for generic external Parquet files (not managed by Hive or Iceberg metadata), Apache Paimon does not provide any built-in configuration or functionality to attach or register those files into an existing table.
The only supported way to load such data is to read the Parquet files using an engine (e.g. Flink SQL / Table API / DataStream) and rewrite the data through Paimon writers, so that snapshots and manifests are generated.

[steFaiz]

@TejMakode1523 As far as I know, yes

[TejMakode1523]

@steFaiz I didn't see any proper documentation for reference.