Portable tuple logical type - #40081
Conversation
6824aba to
8b014a5
Compare
|
Assigning reviewers: R: @shunping for label python. This pull request likely touches a core component ("core" label). Please review with scrutiny. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
8b014a5 to
9a10bec
Compare
|
Newly added test failing on Python3.10-3.12 but passing on Python3.13,3.14. Root cause: a CPython behavior change between 3.12 and 3.13 in
|
* Fixed-length tuple logical type (`beam:logical_type:fixed_tuple:v1`): Backed by a Row with anonymous fields (`f0`, `f1`, ...) and converts to/from Python `tuple`, properly supporting heterogeneous elements (e.g. `Tuple[str, int]`) while keep serialization exactly the same as non-portable TupleCoder. * Variable-length tuple logical type (`beam:logical_type:var_tuple:v1`): Backed by an Array and converts to/from Python `tuple` for `Tuple[T, ...]`. This also keeps serialization the same as IterableCoder backing ArrayType. * Static row encoding in `RowCoderImpl`: Added support for schema option `beam:option:row:static_encoding`, omitting the field count and null-mask bitset when fields are non-nullable to match `TupleCoder` wire efficiency. * Symmetrical `_from_runner_api` on `LogicalType`: Enabled logical types to reconstruct parameterized typing from their proto representations, keeping `typing_from_runner_api` generic and preserving exact type annotations across serialization boundaries. * Add a test case for Beam SQL using python tuples
9a10bec to
d0c241d
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #40081 +/- ##
============================================
+ Coverage 57.24% 58.53% +1.29%
- Complexity 3696 5980 +2284
============================================
Files 1200 1459 +259
Lines 194031 208567 +14536
Branches 3889 5364 +1475
============================================
+ Hits 111070 122093 +11023
- Misses 79366 82177 +2811
- Partials 3595 4297 +702
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Fix #40079
Fixed-length tuple logical type (
beam:logical_type:fixed_tuple:v1): Backed by a Row with anonymous fields (f0,f1, ...) and converts to/from Pythontuple, properly supporting heterogeneous elements (e.g.Tuple[str, int]) while keep serialization exactly the same as non-portable TupleCoder.Variable-length tuple logical type (
beam:logical_type:var_tuple:v1): Backed by an Array and converts to/from PythontupleforTuple[T, ...]. This also keeps serialization the same as IterableCoder backing ArrayType.Static row encoding in
RowCoderImpl: Added support for schema optionbeam:option:row:static_encoding, omitting the field count and null-mask bitset when fields are non-nullable to matchTupleCoderwire efficiency.Symmetrical
_from_runner_apionLogicalType: Enabled logical types to reconstruct parameterized typing from their proto representations, keepingtyping_from_runner_apigeneric and preserving exact type annotations across serialization boundaries.Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.