Skip to content

Fix HNSW entry-point races and construction failure handling - #131

Merged
gouyt13 merged 1 commit into
mainfrom
fix/hnsw-construction-review
Oct 6, 2026
Merged

gouyt13 merged 1 commit into
mainfrom
fix/hnsw-construction-review

Conversation

@gouyt13

@gouyt13 gouyt13 commented Oct 6, 2026

Copy link
Copy Markdown
Member

Summary

Fix four correctness failures in HNSW construction and insertion: a parallel entry-point race, invalid cleanup after an initial build failure, corruption after rejecting a second build, and undefined behavior for M = 0 or M = 1.

  • Read the entry point and maximum level together under the existing mutex in both raw and quantized insertion. This adds no lock acquisitions.
  • Initialize each reserved link pointer before an allocation can fail, own checked-out construction visited sets with RAII, and discard partial construction state after workers join. A failed initial build leaves an empty index that can be built again.
  • Reject construction on a built or loaded index before modifying it. Publish Python cluster metadata only after successful construction, preserving the original index after a rejected rebuild.
  • Reject M < 2 during construction and loading. Add C++ allocation-failure/retry tests and C++/Python regressions for invalid degrees and rejected rebuilds.

Public signatures, serialized layouts, and dependencies are unchanged. Previously accepted index files with M = 1 are now rejected. Scheduling and numerical formulas are unchanged; Python cluster-ID conversion is outside this change.

Testing

  • Full C++ CTest suite: 295 passed, with ASan/UBSan and RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF.
  • Python HNSW, quantization factors, portable indexes, thread counts, and persistence paths: 241 passed against a rebuilt and installed wheel; imported extension path and wheel contents verified.
  • Both targeted ThreadSanitizer reproducers passed: parallel construction and parallel add(), each with 800 points, eight workers, and forced delays after unlocking the entry-point mutex. HNSW header paths were instrumented; the linked compiled core was not TSan-instrumented. Deadlock detection was disabled for these race-focused checks.
  • Original allocation-failure/destruction and rejected-rebuild reproducers passed under ASan/UBSan.
  • Full clang-tidy over all 37 configured production translation units, C++ formatting, Python formatting/lint, and git diff --check passed.

Validation ran on Linux x86-64 with GCC 14.3.0. LeakSanitizer was disabled because of the tracing sandbox. ARM64 was not executed. This is a correctness fix; no benchmark or performance improvement is claimed.

@gouyt13
gouyt13 merged commit 6d34e4d into main Oct 6, 2026
35 checks passed
@gouyt13
gouyt13 deleted the fix/hnsw-construction-review branch October 6, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant