-
Notifications
You must be signed in to change notification settings - Fork 927
Add ObjectStore::append #3791
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ObjectStore::append #3791
Conversation
If we're happy with this interface, I can look into implementing it for LocalFileSystem |
I agree with the API design, I made a POC on synnada-ai#4. Thanks for the effort 😀 |
LGTM, thanks Raphael 👍 |
/// * Concurrent calls to [`ObjectStore::put`] may result in data loss / corruption | ||
/// * Concurrent calls to [`ObjectStore::append`] may result in data loss / corruption | ||
/// | ||
/// Additionally some stores, such as Azure, may only support appending to objects created |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 the docstrings make a lot of sense -- thank you for spelling it out so clearly
Ok - I'll get this in
Looks good from a quick look, if you would like to get it polished up and raise a PR here I'll give a more in-depth review |
Benchmark runs are scheduled for baseline = 4f67296 and contender = 661bbad. 661bbad is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Part of apache/arrow-rs-object-store#181
Rationale for this change
Adds an append API to allow for appending to files where supported
What changes are included in this PR?
Are there any user-facing changes?
FYI @metesynnada @ozankabak @alamb