Serf long shot plan system#16

Draft

rjurney wants to merge 41 commits intorjurney/v0.1from

cursor/serf-long-shot-plan-system-b0d4

Commits on Mar 8, 2026

Add build plan for SERF implementation

cursoragent
and
rjurney
committed
Convert from Poetry to uv, replace black/isort/flake8 with Ruff, update config and module structure

cursoragent
and
rjurney
committed
Add pipeline types, type generator, and DSPy signatures with tests

cursoragent
and
rjurney
committed
Add blocking module: embeddings, FAISS blocker, name normalization, pipeline with tests

cursoragent
and
rjurney
committed
Add matching and merging modules: UUID mapper, matcher, few-shot, merger with tests

cursoragent
and
rjurney
committed
Add evaluation metrics, benchmarks, dataset analysis, and edge resolution with tests

cursoragent
and
rjurney
committed
Add Spark integration (schemas, utils, iceberg, graph) and DSPy agents with tests

cursoragent
and
rjurney
committed
Add complete CLI with analyze, block, match, eval, resolve, benchmark, download commands

cursoragent
and
rjurney
committed
Update benchmarks to use Leipzig dataset source, add benchmark scripts, fix text column detection

cursoragent
and
rjurney
committed
Add README, LICENSE, PyPI packaging setup

cursoragent
and
rjurney
committed
Add benchmark results: DBLP-ACM F1=0.83, Abt-Buy F1=0.46, DBLP-Scholar F1=0.90 (embedding baseline)

cursoragent
and
rjurney
committed
Port scripts into CLI: add benchmark-all command, --use-llm/--no-llm flag, remove scripts/ directory

cursoragent
and
rjurney
committed
Add serf run command for end-to-end ER on any CSV/Parquet/Iceberg with optional YAML config

cursoragent
and
rjurney
committed
Remove embedding-based matching: use embeddings only for blocking, LLM for all matching

cursoragent
and
rjurney
committed
Enhance serf analyze to generate LLM-powered ER config YAML with --output flag

cursoragent
and
rjurney
committed
Add integration tests: LLM-generated ER config from benchmark data with validation

cursoragent
and
rjurney
committed
Add Publication/Product types, fix block sizes (target=30, max=100), fix analyze LLM guidance, add auto-convergence

cursoragent
and
rjurney
committed
Fix DSPy threading with dspy.context, add head-to-head benchmark comparison test, fix analyze config generation

cursoragent
and
rjurney
committed
Default to name-only embedding for blocking, add blocking_fields config for agentic override

cursoragent
and
rjurney
committed
Add critical rule: embeddings for blocking only, never for matching. All matching via LLM.

cursoragent
and
rjurney
committed
Set max_tokens=8192 for LLM matcher to prevent output truncation

cursoragent
and
rjurney
committed
Add --limit and --concurrency options, tqdm progress bar for LLM matching

cursoragent
and
rjurney
committed
Benchmark results: DBLP-ACM P=0.895 R=0.625 F1=0.736 with LLM matching (Gemini Flash, 30 concurrent)

cursoragent
and
rjurney
committed
Address Gemini code review: fix string ID crash, remove incorrect Phase 1 recovery, use optional-dependencies, rename _resolve_blocks_with_llm, document FAISS type ignores

cursoragent
and
rjurney
committed
Auto-scale block size for --limit test runs: target=5 when limit<=20

cursoragent
and
rjurney
committed
Extract predicted pairs from source_ids as well as matches, fix FAISS block splitting for small targets

cursoragent
and
rjurney
committed
Convert async tests to use pytest-asyncio with @pytest.mark.asyncio

cursoragent
and
rjurney
committed
Add SCALABILITY.md: vector engine recommendations for beyond-RAM blocking

cursoragent
and
rjurney
committed
Add rigorous source ID and UUID tracking across pipeline

cursoragent
and
rjurney
committed
Port Abzu er_eval.py rigor: comprehensive evaluator with dedup, skip analysis, UUID validation, PASS/FAIL checks

cursoragent
and
rjurney
committed
Improve install instructions: add pip and conda paths, note about faiss-cpu

cursoragent
and
rjurney
committed
Add pyspark-mcp dependency

cursoragent
and
rjurney
committed

Commits on Mar 9, 2026

Commits on Mar 10, 2026

Add QUICKSTART.md: end-to-end guide for using SERF

cursoragent
and
rjurney
committed