Commit 7e66ccb
authored
feat: Add tell() to OutputStream writers (#2998)
# Rationale for this change
Currently, PyIceberg writes one manifest per snapshot operation
regardless of manifest size. In order to eventually support this we need
to be able to track written bytes without closing the file, so that we
can roll to a new file once we hit target size.
We had some of this work done in #650, but we can keep this simple and
add writers as a follow up. The nice thing is that the underlying
streams we support already have a tell() method and we just need to
expose it.
With this change in the follow up we can do:
```
with write_manifest(...) as writer:
writer.add_entry(entry)
if writer.tell() >= target_file_size:
# roll to new file
```
## Are these changes tested?
Yes, added a test :)
## Are there any user-facing changes?
No1 parent c0e7c6d commit 7e66ccb
File tree
4 files changed
+44
-0
lines changed- pyiceberg
- avro
- io
- tests/utils
4 files changed
+44
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
317 | 317 | | |
318 | 318 | | |
319 | 319 | | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
140 | 140 | | |
141 | 141 | | |
142 | 142 | | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
143 | 146 | | |
144 | 147 | | |
145 | 148 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1059 | 1059 | | |
1060 | 1060 | | |
1061 | 1061 | | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
1062 | 1065 | | |
1063 | 1066 | | |
1064 | 1067 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
897 | 897 | | |
898 | 898 | | |
899 | 899 | | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
0 commit comments