What would you like to happen?
Motivation
Beam's Python ml/rag ingestion module currently supports Milvus and Qdrant (sdks/python/apache_beam/ml/rag/ingestion/). Chroma is one of the most widely-adopted open-source vector databases, especially for local and embedded RAG workloads, and is the missing piece in Beam's OSS vector-DB coverage.
Proposal
Add sdks/python/apache_beam/ml/rag/ingestion/chroma.py implementing a WriteToChroma PTransform matching the shape of the existing Milvus and Qdrant connectors:
- Accepts
Chunk records from the existing RAG chunking transforms
- Supports both
chromadb.HttpClient (remote) and chromadb.PersistentClient
(embedded / local) client modes
- Configurable collection name, embedding + metadata mapping, batch size
- Standard retry/backoff
Benefits
- Completes Beam's coverage of the OSS local-first vector-DB tier
- Chroma is Apache-2.0 licensed — no dependency-license concerns
PersistentClient runs in-process, so integration tests do not need
Docker or a remote service — a meaningful advantage over Milvus/Qdrant
which require running servers in CI
Testing plan
- Unit tests mirroring
qdrant_test.py
- Integration test using
chromadb.PersistentClient against a tmpdir
(no external service required)
- Docs + example consistent with existing Milvus / Qdrant docs
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components
What would you like to happen?
Motivation
Beam's Python ml/rag ingestion module currently supports Milvus and Qdrant (
sdks/python/apache_beam/ml/rag/ingestion/). Chroma is one of the most widely-adopted open-source vector databases, especially for local and embedded RAG workloads, and is the missing piece in Beam's OSS vector-DB coverage.Proposal
Add
sdks/python/apache_beam/ml/rag/ingestion/chroma.pyimplementing aWriteToChromaPTransform matching the shape of the existing Milvus and Qdrant connectors:Chunkrecords from the existing RAG chunking transformschromadb.HttpClient(remote) andchromadb.PersistentClient(embedded / local) client modes
Benefits
PersistentClientruns in-process, so integration tests do not needDocker or a remote service — a meaningful advantage over Milvus/Qdrant
which require running servers in CI
Testing plan
qdrant_test.pychromadb.PersistentClientagainst a tmpdir(no external service required)
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components