Skip to content

perf(index): parallelize FMIndex partition builds#7422

Open
jackye1995 wants to merge 1 commit into
lance-format:mainfrom
jackye1995:jack/fix-fmindex-build-regression
Open

perf(index): parallelize FMIndex partition builds#7422
jackye1995 wants to merge 1 commit into
lance-format:mainfrom
jackye1995:jack/fix-fmindex-build-regression

Conversation

@jackye1995

@jackye1995 jackye1995 commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Summary:

  • Build streamed FMIndex partitions with bounded parallel CPU workers.
  • Add row and byte partition controls with deterministic file ordering.
  • Cover byte-based partition splits and worker write failures.

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer performance labels Jun 23, 2026
@codecov

codecov Bot commented Jun 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.45161% with 42 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance-index/src/scalar/fmindex.rs 86.45% 30 Missing and 12 partials ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

A-index Vector index, linalg, tokenizer performance

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jackye1995