Skip to content

feat: turboquant encoding for vectors #7167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
lwwmanning wants to merge 21 commits into
base: develop
Choose a base branch
from
+3,364 −1
Draft

feat: turboquant encoding for vectors #7167

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
6cdf3aa
feat[turboquant]: add TurboQuant vector quantization encoding
lwwmanning Mar 25, 2026
327c430
feat[turboquant]: add TurboQuantCompressor and WriteStrategyBuilder i…
lwwmanning Mar 25, 2026
dc1e27b
refactor[turboquant]: integrate into BtrBlocks compressor directly
lwwmanning Mar 25, 2026
5b6eb15
bench[turboquant]: add compression/decompression throughput benchmarks
lwwmanning Mar 25, 2026
9043d5a
perf[turboquant]: replace dense rotation with randomized Hadamard tra…
lwwmanning Mar 25, 2026
903ec7a
test[turboquant]: add theoretical error bound and inner product bias …
lwwmanning Mar 25, 2026
84bb0ba
chore[turboquant]: fix review issues and generate public-api.lock
lwwmanning Mar 25, 2026
943a0c8
chore[turboquant]: review cleanup — tighter tests, naming, validation
lwwmanning Mar 25, 2026
a9dd626
docs[turboquant]: add crate-level docs with compression ratios and er…
lwwmanning Mar 25, 2026
c128529
feat[turboquant]: support 1-8 bit quantization
lwwmanning Mar 25, 2026
4e01049
feat[turboquant]: support 9-bit Prod for tensor core int8 GEMM
lwwmanning Mar 25, 2026
0238e29
bench[turboquant]: add dim 1024 and 1536 benchmarks
lwwmanning Mar 26, 2026
8d62ce2
feat[turboquant]: add rotation sign export/import and hot-path inverse
lwwmanning Mar 26, 2026
5f3e96f
feat[turboquant]: define TurboQuantMSEArray and TurboQuantQJLArray
lwwmanning Mar 26, 2026
10a67a7
feat[turboquant]: add new compression functions for cascaded arrays
lwwmanning Mar 26, 2026
3eed3de
refactor[btrblocks]: simplify TurboQuant compressor for cascaded arrays
lwwmanning Mar 26, 2026
4e85cb5
chore[turboquant]: regenerate public-api.lock for new array types
lwwmanning Mar 26, 2026
601068a
refactor[turboquant]: restructure into subdirectory modules, delete d…
lwwmanning Mar 26, 2026
83ff8d1
test[turboquant]: improve test coverage and add explanatory comments
lwwmanning Mar 26, 2026
f1c0716
perf[turboquant]: restore fast SIMD-friendly decode by expanding stor…
lwwmanning Mar 26, 2026
de598ad
Merge remote-tracking branch 'origin/develop' into claude/admiring-li…
lwwmanning Mar 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ members = [
"encodings/zigzag",
"encodings/zstd",
"encodings/bytebool",
"encodings/turboquant",
# Benchmarks
"benchmarks/lance-bench",
"benchmarks/compress-bench",
Expand Down Expand Up @@ -282,6 +283,7 @@ vortex-sequence = { version = "0.1.0", path = "encodings/sequence", default-feat
vortex-session = { version = "0.1.0", path = "./vortex-session", default-features = false }
vortex-sparse = { version = "0.1.0", path = "./encodings/sparse", default-features = false }
vortex-tensor = { version = "0.1.0", path = "./vortex-tensor", default-features = false }
vortex-turboquant = { version = "0.1.0", path = "./encodings/turboquant", default-features = false }
vortex-utils = { version = "0.1.0", path = "./vortex-utils", default-features = false }
vortex-zigzag = { version = "0.1.0", path = "./encodings/zigzag", default-features = false }
vortex-zstd = { version = "0.1.0", path = "./encodings/zstd", default-features = false }
Expand Down
34 changes: 34 additions & 0 deletions encodings/turboquant/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
[package]
name = "vortex-turboquant"
authors = { workspace = true }
categories = { workspace = true }
description = "Vortex TurboQuant vector quantization encoding"
edition = { workspace = true }
homepage = { workspace = true }
include = { workspace = true }
keywords = { workspace = true }
license = { workspace = true }
readme = { workspace = true }
repository = { workspace = true }
rust-version = { workspace = true }
version = { workspace = true }

[lints]
workspace = true

[dependencies]
prost = { workspace = true }
rand = { workspace = true }
rand_distr = { workspace = true }
vortex-array = { workspace = true }
vortex-buffer = { workspace = true }
vortex-error = { workspace = true }
vortex-fastlanes = { workspace = true }
vortex-mask = { workspace = true }
vortex-session = { workspace = true }
vortex-utils = { workspace = true }
parking_lot = { workspace = true }

[dev-dependencies]
rstest = { workspace = true }
vortex-array = { workspace = true, features = ["_test-harness"] }
Loading
Loading