Skip to content

Commit cb615c9

Browse files
SteNicholasSteNicholas
committed
feat(format-table): support reading and writing format tables
A format table is a directory of same-format data files with no snapshots and no manifests, laid out like a standard Hive table. It is loaded through `Catalog::GetFormatTable()` and read and written through `FormatTableScan`, `FormatTableRead`, `FormatTableWrite` and `FormatTableCommit` rather than through the managed table path. It arrives as public API under `include/paimon/`: - `FormatTable` and its scan, read, write and commit, with partition discovery from the directory layout, split packing, projection, predicate pushdown, limit, and two-phase writes that stage their output under hidden names until the commit renames it into place. `parquet` and `orc` are the file formats it can read and write; a table naming any other is refused when it is created and when it is opened. - `FormatTableCatalog`, a capability interface a catalog implements when it can load a format table itself, so that `Catalog` gains no vtable slot; the rest catalog takes the location and the schema from one response rather than from two that could disagree. - Schema validation at create time through one entry point every catalog uses, so a table one catalog accepts is one every catalog can open, and the same rules again when the table is opened, since a schema can reach a catalog without having passed through creation here. - Path containment, visibility and partition-binding checks on every public split and commit message, since both are documented as decodable from untrusted bytes. A split holds whole files only: parquet and orc each record where their own row groups and stripes begin, so a reader handed a byte range of one would have to find that out for itself. `docs/source/user_guide/format_table.rst` describes the layout, the options and the limits that remain.
1 parent 223e566 commit cb615c9

63 files changed

Lines changed: 8464 additions & 52 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/source/api.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ API Reference
2727
api/commit
2828
api/scan
2929
api/read
30+
api/format_table
3031
api/predicate
3132
api/file_format
3233
api/file_system

‎docs/source/api/catalog.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,10 @@ Interface
2727
.. doxygenclass:: paimon::Catalog
2828
:members:
2929

30+
.. doxygenclass:: paimon::FormatTableCatalog
31+
:members:
32+
:undoc-members:
33+
3034
.. doxygenclass:: paimon::Identifier
3135
:members:
3236
:undoc-members:

‎docs/source/api/format_table.rst‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
.. Licensed to the Apache Software Foundation (ASF) under one
2+
.. or more contributor license agreements. See the NOTICE file
3+
.. distributed with this work for additional information
4+
.. regarding copyright ownership. The ASF licenses this file
5+
.. to you under the Apache License, Version 2.0 (the
6+
.. "License"); you may not use this file except in compliance
7+
.. with the License. You may obtain a copy of the License at
8+
9+
.. http://www.apache.org/licenses/LICENSE-2.0
10+
11+
.. Unless required by applicable law or agreed to in writing,
12+
.. software distributed under the License is distributed on an
13+
.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
.. KIND, either express or implied. See the License for the
15+
.. specific language governing permissions and limitations
16+
.. under the License.
17+
18+
============
19+
Format Table
20+
============
21+
22+
.. _cpp-api-format-table:
23+
24+
Table
25+
=====
26+
27+
.. doxygenclass:: paimon::FormatTable
28+
:members:
29+
:undoc-members:
30+
31+
Read
32+
====
33+
34+
.. doxygenclass:: paimon::FormatTableScan
35+
:members:
36+
:undoc-members:
37+
38+
.. doxygenclass:: paimon::FormatDataSplit
39+
:members:
40+
:undoc-members:
41+
42+
.. doxygenclass:: paimon::FormatTableRead
43+
:members:
44+
:undoc-members:
45+
46+
Write
47+
=====
48+
49+
.. doxygenclass:: paimon::FormatTableWrite
50+
:members:
51+
:undoc-members:
52+
53+
.. doxygenclass:: paimon::FormatCommitMessage
54+
:members:
55+
:undoc-members:
56+
57+
.. doxygenclass:: paimon::FormatTableCommit
58+
:members:
59+
:undoc-members:

‎docs/source/user_guide.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ User Guide
3333
user_guide/data_types
3434
user_guide/primary_key_table
3535
user_guide/append_only_table
36+
user_guide/format_table
3637
user_guide/system_tables
3738
user_guide/write
3839
user_guide/commit

‎docs/source/user_guide/format_table.rst‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
.. Licensed to the Apache Software Foundation (ASF) under one
2+
.. or more contributor license agreements. See the NOTICE file
3+
.. distributed with this work for additional information
4+
.. regarding copyright ownership. The ASF licenses this file
5+
.. to you under the Apache License, Version 2.0 (the
6+
.. "License"); you may not use this file except in compliance
7+
.. with the License. You may obtain a copy of the License at
8+
9+
.. http://www.apache.org/licenses/LICENSE-2.0
10+
11+
.. Unless required by applicable law or agreed to in writing,
12+
.. software distributed under the License is distributed on an
13+
.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
.. KIND, either express or implied. See the License for the
15+
.. specific language governing permissions and limitations
16+
.. under the License.
17+
18+
.. Ported from the Paimon documentation:
19+
.. https://github.com/apache/paimon/blob/master/docs/docs/concepts/rest/tables.mdx
20+
21+
.. default-domain:: cpp
22+
.. highlight:: cpp
23+
24+
Format Table
25+
============
26+
A format table is a directory that holds multiple files of the same format. It carries no
27+
snapshots and no manifests: the files in the directory are the table, so reading it lists
28+
directories and writing to it adds files. A table is a format table when its ``type`` option is
29+
``format-table``; ``file.format`` then names the format of every file in it, which here is
30+
``parquet`` or ``orc``.
31+
32+
A partitioned format table uses the standard Hive directory layout, and its partitions are
33+
discovered from that layout rather than from metadata. By default a partition directory is named
34+
``key=value``; setting ``format-table.partition-path-only-value`` names it by the value alone.
35+
36+
Because a directory of plain files records no row identity, a format table only accepts inserts,
37+
and reads return the table's own columns with no ``_VALUE_KIND`` field.
38+
39+
Reading and writing
40+
-------------------
41+
A format table is not served through :cpp:func:`Catalog::GetTable`, which describes a managed
42+
table; use :cpp:func:`Catalog::GetFormatTable` instead. It is then read and written through
43+
``FormatTableScan``, ``FormatTableRead``, ``FormatTableWrite`` and ``FormatTableCommit`` rather
44+
than through the managed table path.
45+
46+
A write is two-phase, since a directory has no metadata to switch atomically: written files are
47+
staged under hidden names that a scan skips, and only the commit renames them into place.
48+
49+
A writer starts a new file once the one it is filling reaches ``target-file-row-num`` rows or
50+
``target-file-size`` bytes. Both are checked between batches rather than between rows, because a
51+
batch is the unit this API writes in, so a file may pass either target by up to one batch. Java
52+
checks the row count on every row and the size every thousand rows, and its files therefore sit
53+
closer to the target.
54+
55+
Aborting a write
56+
----------------
57+
``FormatTableWrite::Abort()`` removes the files the write staged. It is the one call still allowed
58+
after ``PrepareCommit()``, so a commit that is prepared and then abandoned can still be cleaned up.
59+
60+
Path containment is checked on the path text, which stops a ``..`` from leaving the table but not
61+
a symbolic link pointing out of it - the same as Java's own local file system behaviour.
62+
63+
``FormatTableCommit::Abort()`` does the same for the messages a commit was given. **Neither undoes
64+
a commit that succeeded**: once a file has been renamed into place it is no longer staged, and
65+
nothing here will take it back. Both are best effort and never fail, so a warning in the log is the
66+
only signal that a file could not be removed.
67+
68+
Give ``FormatTableCommit`` only the messages this job's own writers produced. A message names a
69+
staged file by path, and a commit can tell that the path belongs to this table, sits in the
70+
partition the message declares, and is staged rather than already published - not whose staged file
71+
it is. A well-formed message from somewhere else is published, or discarded by ``Abort()``, like
72+
any other.
73+
74+
Relationship to Java Paimon
75+
---------------------------
76+
Java serves format tables from a Hive or REST catalog, which holds the schema. This implementation
77+
also serves them from a file system catalog, which keeps the schema under the table directory - an
78+
extension Java does not have. Only for such a table are the ``schema`` and ``branch`` directories
79+
below the location treated as metadata rather than as data.
80+
81+
A file system catalog keeps a table's schema in ``schema`` and its branches in ``branch`` below
82+
the table location, so under ``format-table.partition-path-only-value`` the first partition value
83+
may not be ``schema`` or ``branch``: the directory a write would use is the one holding the
84+
table's own metadata. Such a write is refused, as is an overwrite naming that partition - which
85+
would otherwise delete the schema. A table served from a REST or Hive catalog keeps its schema
86+
elsewhere, so there these are ordinary partition values and are read and written like any other.
87+
88+
Two smaller differences come from this library's own conventions:
89+
90+
* the ``type`` option is matched without regard to case, so ``FORMAT-TABLE`` names a format table
91+
here and not in Java, which compares it exactly;
92+
* a projection that names the same column twice is rejected when the read is built. Java reads
93+
such a column once per entry.
94+
95+
Current limits
96+
--------------
97+
Compared with Java Paimon, this implementation does not yet support:
98+
99+
* the ``csv``, ``json``, ``text`` and ``mosaic`` file formats, leaving ``parquet`` and ``orc``.
100+
All four are line-delimited text in Java, which shares one line-reading layer between them;
101+
this library has no text file format at all, so the first of them to be added has to bring that
102+
layer with it;
103+
* cutting one large data file into byte ranges so that several readers share it. Java does this
104+
only for its line-delimited text formats, which are the ones missing here; ``parquet`` and
105+
``orc`` each record where their own row groups and stripes begin, and a reader handed a byte
106+
range of one would have to find that out for itself;
107+
* ``metastore.partitioned-table``, which moves partition visibility into the catalog, and the
108+
Hive partition sync that goes with it;
109+
* partition filters beyond equality on partition values, where Java accepts a full predicate;
110+
* ``scan.ignore-corrupt-file`` and ``scan.ignore-lost-file``, which are not implemented: a
111+
corrupt or missing data file fails the read rather than being skipped;
112+
* ``partition.legacy-name``, which changes how a partition value is rendered into its directory
113+
name;
114+
* ``format-table.commit-hive-sync-url``, which registers committed partitions with a Hive
115+
metastore;
116+
* a table every one of whose columns is a partition column. Java projects the partition columns
117+
out of what it writes, leaving files that carry nothing but a row count; here such a schema is
118+
refused when the table is created and when it is opened;
119+
* ``TIMESTAMP``, ``DECIMAL``, ``FLOAT`` and ``DOUBLE`` partition columns, which Java allows. This
120+
is a restriction of the whole library rather than of format tables, and a table Java created
121+
with such a partition column fails to open here rather than at the first read.
122+
123+
``data-file.path-directory`` has no effect here, and none in Java either: Java's format table
124+
writer builds its paths from the table root rather than from that directory.
125+
``format-table.implementation`` is honoured by the engines rather than by the table - in Java
126+
Spark it selects between Paimon's own implementation and the engine's ``FileTable`` - so it has
127+
no meaning inside this library.
128+
129+
Validation
130+
----------
131+
A table Java can serve and this library cannot is refused at creation rather than accepted and
132+
then found unopenable, whichever catalog it is created through. It can still reach a catalog
133+
another way - written by Java, or by an older client - so the same checks run again when the
134+
table is opened.
135+
136+
Whitespace in a partition value is judged by ASCII rules here, while Java uses
137+
``Character.isWhitespace``; a value made only of non-ASCII whitespace therefore lands in a
138+
partition of its own rather than in the default one.

‎include/paimon/api.h‎

Lines changed: 24 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -20,23 +20,30 @@
2020

2121
#pragma once
2222

23-
#include "paimon/commit_context.h" // IWYU pragma: export
24-
#include "paimon/defs.h" // IWYU pragma: export
25-
#include "paimon/factories/factory.h" // IWYU pragma: export
26-
#include "paimon/file_store_commit.h" // IWYU pragma: export
27-
#include "paimon/file_store_write.h" // IWYU pragma: export
28-
#include "paimon/fs/file_system_factory.h" // IWYU pragma: export
29-
#include "paimon/memory/memory_pool.h" // IWYU pragma: export
30-
#include "paimon/predicate/predicate.h" // IWYU pragma: export
31-
#include "paimon/read_context.h" // IWYU pragma: export
32-
#include "paimon/reader/batch_reader.h" // IWYU pragma: export
33-
#include "paimon/record_batch.h" // IWYU pragma: export
34-
#include "paimon/result.h" // IWYU pragma: export
35-
#include "paimon/scan_context.h" // IWYU pragma: export
36-
#include "paimon/status.h" // IWYU pragma: export
37-
#include "paimon/table/source/table_read.h" // IWYU pragma: export
38-
#include "paimon/table/source/table_scan.h" // IWYU pragma: export
39-
#include "paimon/write_context.h" // IWYU pragma: export
23+
#include "paimon/commit_context.h" // IWYU pragma: export
24+
#include "paimon/defs.h" // IWYU pragma: export
25+
#include "paimon/factories/factory.h" // IWYU pragma: export
26+
#include "paimon/file_store_commit.h" // IWYU pragma: export
27+
#include "paimon/file_store_write.h" // IWYU pragma: export
28+
#include "paimon/fs/file_system_factory.h" // IWYU pragma: export
29+
#include "paimon/memory/memory_pool.h" // IWYU pragma: export
30+
#include "paimon/predicate/predicate.h" // IWYU pragma: export
31+
#include "paimon/read_context.h" // IWYU pragma: export
32+
#include "paimon/reader/batch_reader.h" // IWYU pragma: export
33+
#include "paimon/record_batch.h" // IWYU pragma: export
34+
#include "paimon/result.h" // IWYU pragma: export
35+
#include "paimon/scan_context.h" // IWYU pragma: export
36+
#include "paimon/status.h" // IWYU pragma: export
37+
#include "paimon/table/format/format_commit_message.h" // IWYU pragma: export
38+
#include "paimon/table/format/format_data_split.h" // IWYU pragma: export
39+
#include "paimon/table/format/format_table.h" // IWYU pragma: export
40+
#include "paimon/table/format/format_table_commit.h" // IWYU pragma: export
41+
#include "paimon/table/format/format_table_read.h" // IWYU pragma: export
42+
#include "paimon/table/format/format_table_scan.h" // IWYU pragma: export
43+
#include "paimon/table/format/format_table_write.h" // IWYU pragma: export
44+
#include "paimon/table/source/table_read.h" // IWYU pragma: export
45+
#include "paimon/table/source/table_scan.h" // IWYU pragma: export
46+
#include "paimon/write_context.h" // IWYU pragma: export
4047

4148
// IWYU pragma: begin_exports
4249
#include "paimon/realtime/realtime_context.h"

‎include/paimon/catalog/catalog.h‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ namespace paimon {
4040
using Instant = std::variant<std::string, int64_t>;
4141

4242
class Database;
43+
class FormatTable;
4344
class Table;
4445
class View;
4546
class Schema;
@@ -208,6 +209,24 @@ class PAIMON_EXPORT Catalog {
208209
/// snapshot id ascending, or an error status.
209210
virtual Result<std::vector<SnapshotInfo>> ListSnapshots(
210211
const Identifier& identifier, const std::string& branch = "") const = 0;
212+
213+
/// Gets a format table: a directory of data files laid out like a standard Hive table.
214+
///
215+
/// A format table is read and written through `FormatTableScan`, `FormatTableRead`,
216+
/// `FormatTableWrite` and `FormatTableCommit` rather than through the managed table path, so
217+
/// it is loaded through its own method instead of `GetTable()`.
218+
///
219+
/// Not virtual: `Catalog` is exported and derived from outside this library, so a new virtual
220+
/// here would have no slot in an already-compiled subclass. A catalog that can load a format
221+
/// table itself derives from `FormatTableCatalog` as well and this finds it with a
222+
/// `dynamic_cast`. Otherwise it reads the location and the schema through the methods above,
223+
/// which costs two requests that can disagree, and treats everything below the location as
224+
/// data.
225+
///
226+
/// @param identifier Identifier of the table to get.
227+
/// @return A result containing the format table, or an error status if the table does not
228+
/// exist or its `type` option is not `format-table`.
229+
Result<std::shared_ptr<FormatTable>> GetFormatTable(const Identifier& identifier) const;
211230
};
212231

213232
} // namespace paimon

‎include/paimon/catalog/format_table_catalog.h‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing, software
13+
* distributed under the License is distributed on an "AS IS" BASIS,
14+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
* See the License for the specific language governing permissions and
16+
* limitations under the License.
17+
*/
18+
19+
#pragma once
20+
21+
#include <memory>
22+
23+
#include "paimon/result.h"
24+
#include "paimon/visibility.h"
25+
26+
namespace paimon {
27+
28+
class FormatTable;
29+
class Identifier;
30+
31+
/// What a catalog implements when it can load a format table itself.
32+
///
33+
/// It is a separate base rather than a method on `Catalog` because `Catalog` is exported and
34+
/// derived from outside this library: a virtual added there would have no slot in a subclass that
35+
/// was already compiled, and calling it on such an object would run off the end of its vtable.
36+
/// A catalog that implements this inherits it alongside `Catalog`, and `Catalog::GetFormatTable()`
37+
/// finds it with a `dynamic_cast`, so nothing about `Catalog` itself has to change.
38+
///
39+
/// A catalog that does not implement it still serves format tables: `Catalog::GetFormatTable()`
40+
/// falls back to reading the location and the schema through the methods every catalog has.
41+
class PAIMON_EXPORT FormatTableCatalog {
42+
public:
43+
virtual ~FormatTableCatalog() = default;
44+
45+
/// Loads `identifier` as a format table.
46+
///
47+
/// The catalog decides both halves the fallback has to guess at: whether the location and the
48+
/// schema can be read in one round trip, and whether it put this table's metadata under the
49+
/// table path - which is what tells a `schema` or `branch` directory below the location from
50+
/// a partition directory of the same name.
51+
///
52+
/// @param identifier Identifier of the table to load.
53+
/// @return A result containing the format table, or an error status if the table does not
54+
/// exist or its `type` option is not `format-table`.
55+
virtual Result<std::shared_ptr<FormatTable>> LoadFormatTable(
56+
const Identifier& identifier) const = 0;
57+
};
58+
59+
} // namespace paimon

0 commit comments

Comments
 (0)