Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support ingesting small parquet files #11

Open
cmenguy opened this issue Feb 1, 2023 · 1 comment
Open

Support ingesting small parquet files #11

cmenguy opened this issue Feb 1, 2023 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@cmenguy
Copy link
Collaborator

cmenguy commented Feb 1, 2023

Currently the small file API support in ingestion seems incomplete, as in the method uploadSmallFile it expects data of type Union[list, dict] which works fine for JSON, but for Parquet format you would be passing in bytes, I couldn't figure out how to get this working with some Parquet data so ended up using JSON. But I think either the prototype of the function needs to change since the notion of multiline isn't really applicable to Parquet, or we need to actually change the code to fully support Parquet binary payload.

@pitchmuc pitchmuc self-assigned this Feb 2, 2023
@pitchmuc pitchmuc added the enhancement New feature or request label Feb 2, 2023
@skanjila
Copy link
Collaborator

@pitchmuc please assign this to me as well

@pitchmuc pitchmuc assigned skanjila and unassigned pitchmuc Feb 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants