chore(db): drop unused ANN indexes on embedding

Approximate retrieval is served by the compact embedding_search projection.
The only vector ordering left on embedding is an exact rerank wrapped as
(distance) + 0, which the planner cannot match to an index expression, and
the binary_quantize expression indexes were never referenced by any query.
The last app version that ordered by a bare distance has drained, so these
ten indexes were maintained on every chunk write while never being scanned.

Author: waleedlatif1

apps/sim/lib/knowledge/vector-columns.ts

@@ -58,16 +58,17 @@ export function embeddingVectorValues(
 }
 
 /**
- * Cosine distance between a chunk's vector and the query vector, in the exact
- * form the width's HNSW index was built on.
+ * Cosine distance between a chunk's vector and the query vector, used for the
+ * search layer's exact rerank over an already-bounded candidate set.
  *
- * The 3,072 column is compared through a `halfvec` cast because pgvector
- * indexes `vector` only up to 2,000 dimensions, so its index is on that cast
- * expression. Postgres matches an expression index by the expression, so a
- * plain `<=>` against the column here would silently drop to a sequential scan
- * — and the cast belongs here rather than at each call site precisely because
- * getting it wrong is invisible in the results and only shows up as latency.
- * `packages/db/schema.ts` records what the half-precision comparison costs.
+ * `embedding` carries no ANN index — approximate retrieval runs against the
+ * compact `embedding_search` projection — so this expression is never expected
+ * to match one, and its caller wraps it to keep the planner from trying.
+ *
+ * The 3,072 column keeps its `halfvec` cast: the width that made the cast
+ * necessary is unchanged, and `packages/db/schema.ts` records that the
+ * half-precision comparison moved no distance measurably for the one model
+ * that emits this width.
  */
 export function embeddingDistance(
   dimensions: KbEmbeddingDimensions,

packages/db/migrations/0360_drop_unused_embedding_hnsw_indexes.sql

@@ -0,0 +1,27 @@
+-- Drops the ten unused ANN indexes on "embedding". Approximate retrieval moved to the
+-- compact "embedding_search" projection; the only vector ordering left on this table is an
+-- exact rerank wrapped as (distance) + 0, which the planner cannot match to an index.
+-- The last app version that ordered by a bare distance has drained, so nothing reads these.
+COMMIT;--> statement-breakpoint
+SET lock_timeout = 0;--> statement-breakpoint
+-- migration-safe: DROP INDEX CONCURRENTLY IF EXISTS is idempotent on replay and takes no blocking lock.
+DROP INDEX CONCURRENTLY IF EXISTS "embedding_vector_hnsw_idx";--> statement-breakpoint
+-- migration-safe: DROP INDEX CONCURRENTLY IF EXISTS is idempotent on replay and takes no blocking lock.
+DROP INDEX CONCURRENTLY IF EXISTS "embedding_384_vector_hnsw_idx";--> statement-breakpoint
+-- migration-safe: DROP INDEX CONCURRENTLY IF EXISTS is idempotent on replay and takes no blocking lock.
+DROP INDEX CONCURRENTLY IF EXISTS "embedding_768_vector_hnsw_idx";--> statement-breakpoint
+-- migration-safe: DROP INDEX CONCURRENTLY IF EXISTS is idempotent on replay and takes no blocking lock.
+DROP INDEX CONCURRENTLY IF EXISTS "embedding_1024_vector_hnsw_idx";--> statement-breakpoint
+-- migration-safe: DROP INDEX CONCURRENTLY IF EXISTS is idempotent on replay and takes no blocking lock.
+DROP INDEX CONCURRENTLY IF EXISTS "embedding_3072_vector_hnsw_idx";--> statement-breakpoint
+-- migration-safe: DROP INDEX CONCURRENTLY IF EXISTS is idempotent on replay and takes no blocking lock.
+DROP INDEX CONCURRENTLY IF EXISTS "embedding_binary_hnsw_idx";--> statement-breakpoint
+-- migration-safe: DROP INDEX CONCURRENTLY IF EXISTS is idempotent on replay and takes no blocking lock.
+DROP INDEX CONCURRENTLY IF EXISTS "embedding_384_binary_hnsw_idx";--> statement-breakpoint
+-- migration-safe: DROP INDEX CONCURRENTLY IF EXISTS is idempotent on replay and takes no blocking lock.
+DROP INDEX CONCURRENTLY IF EXISTS "embedding_768_binary_hnsw_idx";--> statement-breakpoint
+-- migration-safe: DROP INDEX CONCURRENTLY IF EXISTS is idempotent on replay and takes no blocking lock.
+DROP INDEX CONCURRENTLY IF EXISTS "embedding_1024_binary_hnsw_idx";--> statement-breakpoint
+-- migration-safe: DROP INDEX CONCURRENTLY IF EXISTS is idempotent on replay and takes no blocking lock.
+DROP INDEX CONCURRENTLY IF EXISTS "embedding_3072_binary_hnsw_idx";--> statement-breakpoint
+SET lock_timeout = '5s';