Repository navigation
Conversation
An exported adapter over the unexported blockCursor, handing out a term's postings 128 at a time as bare doc numbers, frequencies and norms. * 1-hit lists, the deleted-docs bitmap and the norms column (flattened so the per-block loop doesn't re-decide its shape) are handled the way PostingsIterator handles them. Norms are the same float32 factors the iterator reads. * SeekBlock decodes the block that holds a target and drops what's below it. * LiveCount is the number of postings minus deletions: the footer's count when nothing is deleted, otherwise a walk over the doc numbers only. * BlockMaxCursor: the skip entries already record each block's highest frequency (saturated at 255, which decodes as unbounded) and shortest field length, so BoundsAt is a binary search over the in-memory skip entries; it decodes nothing and moves nothing. TermBounds folds all of them. Tests compare the cursor with PostingsIterator over lists of every shape (many blocks, a tail only, 1-hit, absent), with and without deletions; check that the bounds bound the postings of their block, and that a saturated frequency is reported as unbounded. go.mod: replace scorch_segment_api with the local checkout that has the interfaces, for development on this branch. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
fillNorms resolves the norm factor of each doc of a block: a gather, since each doc number picks a byte of the norms column, which picks a factor of a 256 entry table, and neither NEON nor SSE2 has a gather. It is a good part of what a block costs next to the SIMD scoring of it, so the loop does only that: * the column is checked once for the block (docs ascend, so the last tells whether any is past it), not for each doc; * the other bounds checks are hoisted by slicing docs and norms to n. About 20% faster on a block (49 -> 39ns for 128 docs, 1 cycle per doc, which is the scalar load throughput; the loop was already at the limit of what a scalar gather does). The old per-doc version is kept in the tests as the reference, and TestFillNormsMatchesReference checks the two on columns that are dense, short and constant, for blocks of every length; BenchmarkFillNorms compares them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A pruner asks for the bounds of the blocks in order, so the answer is the block the cursor is on or one of the few after it. BoundsAt looks at those first (unless the target is behind the cursor) and only then falls back to the binary search over the skip entries. A test asks for bounds at random targets while the cursor moves with NextBlock and SeekBlock, against the blocks walked once beforehand. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 4, 2026
…ives in posting.go The exported BlockCursor differed from the unexported blockCursor (the raw format cursor it is built on) by the case of one letter. Name it for what it is, the counterpart of PostingsIterator for segment.BlockCursor consumers, and say in its doc how it relates to blockCursor. PostingsList.BlockCursor, which implements segment.BlockCursorProvider, moves next to the other PostingsList methods in posting.go. No behavior change; segment.BlockCursor and the PostingsList.BlockCursor method keep their names. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
It is the variant of PostingsIterator that hands out postings a block at a time, so name it as one. segment.BlockCursor and the PostingsList.BlockCursor method keep their names, as they are the segment API. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
It implements segment.BlockCursorProvider, whose method is renamed the same way, and returns the BlockPostingsIterator. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The method that implements segment.BlockPostingsList is BlockIterator, the counterpart of Iterator, and what it returns is a BlockPostingsIterator, the counterpart of PostingsIterator. The segment API's interfaces are renamed the same way (BlockPostingsList, BlockPostingsIterator, BlockMaxPostingsIterator). The files are block_postings_iterator.go and its test. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The block length is defined once, by freeway's codec, and handed on by the segment API as PostingsBlockLen. zapx no longer keeps a constant of its own for it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
loadInto decoded each block into the raw cursor's own buffer and then copied it out to the PostingsBlock the caller handed in: a memmove of up to two 512 byte arrays per block, which showed up in profiles under loadInto. blockCursor gets loadBlockInto(docs, freqs), which unpacks the block into buffers of the caller's. It is not a blind copy of loadBlock's result: the decoding is one function (decodeBlock) that both go through, and it records what it has seen of the block (numDocs, lastDoc, bytesRead) as loadBlock does. What it leaves alone is the cursor's buf, and with it the loaded flag, which says whether buf holds the block (a loadBlock after a loadBlockInto decodes it into buf as usual). The block is padded with the terminator after its postings whichever buffer it is in, as the in-block search needs. freqs may be nil. BlockPostingsIterator.loadInto uses it, and no longer allocates the cursor's buf at all. The entries of the block below a seek target are moved to the front after decoding: once per seek, not per block. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
BlockPostingsIterator.TermBounds, which a pruner asks for before it reads anything, walked every skip entry of the term: 1.65 ns an entry, so 13 us for a list of a million postings, per term and segment and query, growing with the length of the list while the pruning it serves doesn't. A term with at least termBoundsMinEntries (4) skip entries now has, in its footer, the lowest minNormID and the highest maxTF of its entries: two bytes, the same lossy bytes a skip entry has, folded together by the serializer as it writes the term (fresh builds and merges both go through it, so the merged terms have their own). TermBounds is then two loads and the last skip entry (for LastDoc), about 2 ns whatever the length. Terms of fewer entries, which is nearly every term of a dictionary, don't pay the bytes: walking three entries costs nothing. termHasBounds in the footer's flags says which have it, so a footer without it, as segments written before were, is read by the walk, which is kept (termBoundsByScan). The bytes read a footer is accounted for stay as they were (termFooterReadCharge), so what is reported doesn't move with the format. Tests: the serializer writes exactly the term's pair (including frequencies that saturate the byte, and at the 254/255 edge), fresh and merged segments (with deletions) have it where they should and TermBounds agrees with the walk and with the true values of the postings, segments without the pair read as before, and the footer round trips. Eight mutations of the writer and reader each fail them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Implements
segment.BlockPostingsIterator(andBlockMaxPostingsIterator) over the bit-packed postings blocks of thebitpack-simdformat.PostingsList.BlockIterator(inposting.go, thesegment.BlockPostingsListmethod; the counterpart ofIterator) returns aBlockPostingsIterator(block_postings_iterator.go), the variant ofPostingsIteratorthat yields postings a block at a time. LikePostingsIteratorit sits on the existing rawblockCursor, and it adds deleted-doc filtering, 1-hit lists, norm resolution and block-max bounds.NextBlock/SeekBlockfill a block of up to 128 docs, freqs and norms directly from the packed blocks; deleted docs are filtered out.PostingsBlock: the rawblockCursorgetsloadBlockInto(docs, freqs), which shares one decoder (decodeBlock) withloadBlock, so there is no copy out of the cursor's own buffer (that copy showed up asruntime.memmoveunderloadIntoin profiles). It records the block (numDocs,lastDoc,bytesRead) asloadBlockdoes and leavesbuf, and theloadedflag that describes it, alone. Measured end to end in bleve (200k docs): about 3-7% on term and AND searches, none on OR.BoundsAtfor the block holding a target, for the block last decoded, and for the whole list), so a caller can prune with block-max algorithms.BoundsAtprobes from the iterator's block before searching.9 commits on top of
bitpack-simd; 7 files, +1417 / -44. New:block_postings_iterator.go,term_bounds_test.goandblock_postings_iterator_test.go. Edited:posting.go(addsBlockIterator),postings_format.go(the term footer, below),postings_block.go,go.mod, and the existing tests, where the localpostingsBlockLenconstant is replaced bysegment.PostingsBlockLen(a mechanical rename; the value is the same freewaybitpack.BlockLen).Format change: the term footer carries the block-max pair of the term
TermBounds(the bounds of a whole list, which block-max pruning asks for before it reads anything) used to walk every skip entry of the term: about 1.65 ns an entry, so 13 us for a list of a million postings, per term, segment and query, growing with the list while the pruning it serves doesn't. A term with at least 4 skip entries now has, in its footer, the lowestminNormIDand highestmaxTFof its entries (two bytes, the same lossy bytes a skip entry has), folded together by the serializer as it writes.TermBoundsis then two loads and the last skip entry, about 2 ns whatever the length (13 us -> 2 ns at a million postings, 2.6 us -> 2 ns at 200k).termHasBoundsin the footer flags says which do. A footer without it, as segments written before this were, is read by the walk, which is kept.termFooterReadCharge), so the reportedbytes readdoesn't move with the format.End to end in bleve at 200k docs it is within noise (the walk was about 3% of collection time in the worst shape there); the point is the scaling at millions of postings a list.
Testing
go test ./...passes.Depends on
perseg/block-cursor: theBlockPostingsIteratorinterfaces andPostingsBlockLen. That module now requires freeway (private, for now; see its description). Until that lands,go.modhere hasreplace github.com/blevesearch/scorch_segment_api/v2 => ../scorch_segment_api-bitpack-simd, which is why this PR can't build in CI.Consumed by blevesearch/bleve, branch
perseg/term-path.Stack (review order)
bitpack.BlockLenis the block length everything else uses)BlockPostingsIteratorinterfaces;PostingsBlockLenfrom freewayBlockPostingsIteratorover bit-packed postings (basebitpack-simd)Only the last one is the design under review. freeway, scorch_segment_api and zapx are the pieces it reads postings and scores through (the block length is defined once, in freeway, and handed on by scorch_segment_api), and vellum is independent FST work that bleve builds against.
🤖 Generated with Claude Code