Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Vector quantization #5

Merged
merged 31 commits into from
Aug 25, 2024
Merged

Vector quantization #5

merged 31 commits into from
Aug 25, 2024

Conversation

mrjleo
Copy link
Owner

@mrjleo mrjleo commented Aug 25, 2024

This PR adds support for index compression via vector quantization:

  • Base class/interface for quantizers is added.
  • (Trained) quantizers are serialized and stored in the index itself.
  • A simple product quantizer based on nanopq is implemented.
  • API change: Index dimension is not required in Index constructors anymore but will instead be inferred from the first batch of vectors that's added to the index.

mrjleo added 30 commits August 20, 2024 13:00
@mrjleo
add abstract base class for quantizers
c6d48e1
@mrjleo
create module for quantizers and add kwargs to Quantizer.fit
83fcf10
@mrjleo
Quantizer: update dtype property, add _encode and _decode methods
5ef4ed9
@mrjleo
implement Quantizer.set_attached
206baf0
@mrjleo
Quantizer: fix error messages
9eddd3a
@mrjleo
remove logger, add missing cls
cd8750e
@mrjleo
update type of quantizer state
c8cd540
@mrjleo
initial implementation of NanoPQQuantizer
cf9035f
@mrjleo
remove Quantizer.get_state and Quantizer.load
439b931
@mrjleo
add Quantizer.dim property
ed00a36
@mrjleo
rename NanoPQQuantizer to NanoPQ
56d32bb
@mrjleo
implement (de)serialization for quantizers
f4faed2
@mrjleo
remove dim and dtype arguments from indexes and infer them automatica…
c7f4379 
…lly when _add is called for the first time
@mrjleo
use both types of indexes in test_early_stopping
3d40358
@mrjleo
make quantizers return dimensions of vectors before and after quantiz…
97fb805 
…ation
@mrjleo
implement basic quantizer support for the Index base class
05016ee
@mrjleo
call quantizer.set_attached in Index constructor
b7f3466
@mrjleo
implement storing a serialized quantizer in an index on disk
3ae2a96
@mrjleo
make Index.batch_iter use quantizers
a68a077
@mrjleo
NanoPQ._from_state: account for attributes and data that may not exist
6ab8952
@mrjleo
use normal constructor in NanoPQ._from_state
6df1edd
@mrjleo
create abstract Index._internal_dim method that is irrespective of qu…
45ca29b 
…antizers
@mrjleo
update OnDiskIndex.to_memory to include quantizer
fa3c6b8
@mrjleo
add quantizer test cases
c7bb8c4
@mrjleo
add test cases for quantized indexes
4875c94
@mrjleo
add nanopq dependency
8e65265
@mrjleo
add test cases for OnDiskIndex.load and OnDiskIndex.to_memory with qu…
ae3a677 
…antizers
@mrjleo
Index: rename _internal_dim to _get_internal_dim
74a2560
@mrjleo
clarify docstrings
e82f26e
@mrjleo
update index documentation to reflect API changes
d5ef4f0
@mrjleo mrjleo merged commit 5ea0708 into main Aug 25, 2024
4 checks passed
@mrjleo mrjleo deleted the quantization branch August 25, 2024 11:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@mrjleo