Skip to content

[Feature Request]: Add Chroma vector database ingestion connector for RAG pipelines #40047

Description

@nitinware

What would you like to happen?

Motivation

Beam's Python ml/rag ingestion module currently supports Milvus and Qdrant (sdks/python/apache_beam/ml/rag/ingestion/). Chroma is one of the most widely-adopted open-source vector databases, especially for local and embedded RAG workloads, and is the missing piece in Beam's OSS vector-DB coverage.

Proposal

Add sdks/python/apache_beam/ml/rag/ingestion/chroma.py implementing a WriteToChroma PTransform matching the shape of the existing Milvus and Qdrant connectors:

  • Accepts Chunk records from the existing RAG chunking transforms
  • Supports both chromadb.HttpClient (remote) and chromadb.PersistentClient
    (embedded / local) client modes
  • Configurable collection name, embedding + metadata mapping, batch size
  • Standard retry/backoff

Benefits

  • Completes Beam's coverage of the OSS local-first vector-DB tier
  • Chroma is Apache-2.0 licensed — no dependency-license concerns
  • PersistentClient runs in-process, so integration tests do not need
    Docker or a remote service — a meaningful advantage over Milvus/Qdrant
    which require running servers in CI

Testing plan

  • Unit tests mirroring qdrant_test.py
  • Integration test using chromadb.PersistentClient against a tmpdir
    (no external service required)
  • Docs + example consistent with existing Milvus / Qdrant docs

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Prism Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions