Skip to content

[wip] feat: Add StreamingRawDataset for cloud storage streaming (early stage) #652

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 21 commits into
base: main
Choose a base branch
from
Draft
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
c27fa65
feat add support for initial raw dataset
bhimrazy Jul 6, 2025
2f1836e
update
bhimrazy Jul 6, 2025
37937db
refactor: remove unused get_classes method from StreamingRawDataset
bhimrazy Jul 6, 2025
741bbbb
update docs
bhimrazy Jul 6, 2025
f9767d9
feat: enhance StreamingRawDataset with caching and preloading capabil…
bhimrazy Jul 6, 2025
229385f
feat: add adaptive preloading and cache statistics to StreamingRawDat…
bhimrazy Jul 6, 2025
8132010
update
bhimrazy Jul 6, 2025
f803019
Merge branch 'main' into feat/add-support-for-raw-dataset
bhimrazy Jul 13, 2025
6237e0e
feat: enhance StreamingRawDataset with file indexing and caching impr…
bhimrazy Jul 15, 2025
d32ad5e
now the indexing part is abit better and modular
bhimrazy Jul 15, 2025
32432ad
update
bhimrazy Jul 16, 2025
5368559
feat: refactor StreamingRawDataset and CacheManager for improved cach…
bhimrazy Jul 16, 2025
b8a2530
implement ZSTD compression for file index caching in StreamingRawDataset
bhimrazy Jul 16, 2025
f4c40c0
fix: update index file path in BaseIndexer to use index_path for caching
bhimrazy Jul 16, 2025
4468e2c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 16, 2025
7b733d9
fix types
bhimrazy Jul 16, 2025
a667bfd
fix types
bhimrazy Jul 16, 2025
09e76de
Merge branch 'main' into feat/add-support-for-raw-dataset
bhimrazy Jul 16, 2025
e4355ee
add cache for len
bhimrazy Jul 16, 2025
8b84764
finally added the sync way of downloading the files
bhimrazy Jul 16, 2025
d2fcf50
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 16, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading