Repository navigation
Conversation
Lets a PostingsList/PostingsIterator implementation opt into segment-merge support and pooled reuse without changing the base PostingsList/PostingsIterator contract, so existing implementations (older zapx major versions) keep compiling unchanged.
An optional capability of a PostingsList, for consumers that want the scalars a scorer needs -- doc number, term frequency, norm -- and nothing else, and want to pay for an interface call once per batch of postings rather than once per posting. * BlockCursorProvider (on PostingsList) hands out a BlockCursor. * BlockCursor fills a PostingsBlock of up to 128 docs, freqs and norms with NextBlock or SeekBlock; deleted docs never appear. Count and LiveCount give the number of postings, the latter exactly and without moving the cursor (free when nothing is deleted). * BlockMaxCursor (optional, on a BlockCursor) bounds the scores of a list: the bounds of the block holding a target without decoding or moving anything (BoundsAt), of the block last decoded, and of the whole list. This is what block-max pruning (WAND) is made of. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 4, 2026
The postings list hands out the variant of PostingsIterator that yields postings a block at a time, and its method is named for that. The types (BlockCursor, BlockCursorProvider, BlockMaxCursor) keep their names. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
PostingsList.Iterator returns a PostingsIterator, so the postings list that can hand out postings a block at a time has BlockIterator, which returns a BlockPostingsIterator. Following the names of the other optional interfaces of the package (MergeablePostingsList, MergeablePostingsIterator): BlockCursorProvider -> BlockPostingsList BlockCursor -> BlockPostingsIterator BlockMaxCursor -> BlockMaxPostingsIterator method BlockCursor -> BlockIterator PostingsBlock, BlockBounds, NextBlock and SeekBlock keep their names. The file is block_iterator.go. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The length of a block of postings is that of the bit-packed format's codec, so it comes from there instead of being written down a second time. This makes freeway a dependency of the module (it has none of its own), and the module's go directive becomes 1.25.0, which is freeway's. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two additions to
scorch_segment_api, both optional capabilities so existing implementations keep compiling:block_iterator.go): a variant ofPostingsIteratorthat yields postings a block at a time, for consumers that want only the scalars a scorer needs (doc number, term frequency, norm) and want to pay for an interface call once per batch instead of once per posting.BlockPostingsList(optional, on aPostingsList) hasBlockIterator, the counterpart ofIterator, which returns aBlockPostingsIterator.BlockPostingsIteratorfills aPostingsBlockof up to 128 docs, freqs and norms withNextBlock/SeekBlock; deleted docs never appear.CountandLiveCountgive the number of postings, the latter exactly and without moving the iterator.BlockMaxPostingsIterator(optional, on aBlockPostingsIterator) bounds the scores of a list: the bounds of the block holding a target without decoding or moving anything (BoundsAt), of the block last decoded, and of the whole list. This is what block-max pruning (WAND / MAXSCORE) is made of.segment.go): lets an implementation opt into segment-merge support and pooled reuse without changing the basePostingsList/PostingsIteratorcontract.New dependency (read this before merging)
PostingsBlockLenisbitpack.BlockLenfrom blevesearch/freeway instead of a second128: the block length is a property of the codec, and it is now defined once and handed on from here (zapx and bleve usesegment.PostingsBlockLen).github.com/blevesearch/freeway, which has no dependencies of its own.godirective is 1.25.0, so this module'sgoline moves from 1.24.0 to 1.25.0 (bleve and zapx v17 / v18 are already on 1.25.0).Why
The bleve per segment search path (blevesearch/bleve,
perseg/term-path) reads postings through these interfaces instead of oneNextper posting.Testing
go build ./...andgo vet ./.... The package has no tests of its own; the interfaces are exercised by the zapx implementation and bleve's tests in the PRs above and below this one.Stack (review order)
bitpack.BlockLenis the block length everything else uses)BlockPostingsIteratorinterfaces;PostingsBlockLenfrom freewayBlockPostingsIteratorover bit-packed postings (basebitpack-simd)Only the last one is the design under review. freeway, scorch_segment_api and zapx are the pieces it reads postings and scores through (the block length is defined once, in freeway, and handed on by scorch_segment_api), and vellum is independent FST work that bleve builds against.
🤖 Generated with Claude Code