Build each file's generated_by at reconcile from the imported lineage (#577)

[NoopDog]

Closes #577

What changed

• Reconcile builds each file's generated_by from the imported lineage (src/meta_disco/reconcile_lineage.py, new):
  • It reads the lineage files of the run's catalog (Import lineage from the source tables as evidence, word for word #583) and translates each line through the activity map (Add the activity translation table for lineage raw words #584).
  • It resolves the child and the file parent within the child's dataset, by file_id or drs_uri. An IGVF accession was already written as a DRS URI by the importer.
  • The lookup shares the slot-evidence join's single pass over the run's records.
• edges.merge_steps merges those steps with inference's own (the name edges) into one generated_by:
  • Sources naming the same activity and parent are listed together in named_by.
  • A lineage source is cited as {source_type, rule_id: <activity-map row id>, activity_id?, source: ClaimSource}.
  • Conflicts give no generated_by: two activities, two parents in a role that takes one, or, beside a specific activity, a generic Activity step whose parent no specific source names. The report lists them.
• edges.misfits flags steps that don't fit their activity's declaration. The step is still written.
• reconcile_report.json gains a lineage block, rendered as a new section of docs/reconcile-report.md and the dashboard:
  • per source type and dataset: offered, resolved, not_in_dataset, several_match, untranslated, sample_parent, child_not_in_run, child_several_match, parent_is_child;
  • who named each file's step;
  • conflicts, with who said what;
  • misfits.
• The rules report lists the authored activity-map rows among the edge rules, counted by the reconciled inputs that cite them.
• Supporting changes:
  • activities.agreed holds the "generic Activity agrees with any" rule.
  • reconcile_lineage.applies is the one catalog rule for slot evidence and lineage.
  • output_utils.relative_to replaces two copies of the same helper.
  • The activity map records its sha256 in the reconciled envelope.
  • tests/lineage_fixtures.py holds the shared lineage test helpers.

Why

Before this change, a file got a parent only from its own name: 224K index and checksum steps. For example, T2T_CHRY's 14K .tbis whose name matches two VCFs got none (#438), and neither did any T2T QC file or IGVF derived file. The source tables say exactly which file each was made from. #577 joins that to the run, which is what #571 (inheritance) needs.

Assumptions I made

These follow Dave's calls in the #577 walk-through, recorded in the issue:

• An unresolved parent gives no input. It is counted, and nothing is written on the record. parent_file stays required and comes from the parent record's own name.
• A conflict gives no generated_by, including inference's own step. The report lists who said what.
• A generic Activity step (no authored row names it today), beside a specific activity, counts only where its parent matches one the specific source names; any other parent is an edge conflict. A file whose only steps are generic gets no generated_by and no conflict. It is counted as generic_only and held back until an activity-map row naming Activity is reviewed.
• Lineage is read for the input's catalog, the same rule as slot evidence. An input naming no catalog reads only the latest catalog's lineage.
• Lineage is scoped by the envelope's target dataset, as slot evidence is.
• The lineage root is lineage_evidence beside --evidence-root by default, so a test's temporary evidence root never reads the real lineage. --lineage-root overrides it, and --no-evidence excludes lineage too.
• Sample parents (used_biosample_id) are counted and wait for Record sample and donor lineage, SampleCollectionActivity and isBiologicalChildOf, in a run's edges.jsonl #582.
• parent_kind is read off the parent's name, as inference does. The misfit check judges the parent's inferred data_type instead.
• I did not re-reconcile the shared run or regenerate the Pages reports. The shared output/ run is read by the meta-disco browser, so that is a separate operational step after merge.

How to verify

Reconcile a copy of the latest run, so the shared output/ is untouched:

R=output/anvil/20260930_071213; C=$TMPDIR/run/20260930_071213; mkdir -p $C
for f in $R/*.json; do ln -sf "$PWD/$f" $C/; done
uv run python scripts/reconcile.py --run $C

1. A T2T_CHRY .tbi that Index files must not opine when two candidate parents share a filename #438 declines. In $C/reconciled/index_classifications.ndjson, simons_data_sample_207.chr17.hc.vcf.gz.tbi (two of them, one per assembly) has generated_by.activity == "IndexActivity". Its one indexed input is its own VCF: parent_key b8caea57… for one and ad95dfaf… for the other. Its named_by lists repository_activity (activity.indexing) and repository_metadata (activity.t2t_index).
2. HG01536.samtools.stats.txt (in remaining_classifications.ndjson) has QualityControlActivity with reported_on = HG01536.cram, a different CRAM key in ANVIL_T2T (3b029f77…) and in ANVIL_T2T_CHRY (2a4f10bf…).
3. IGVFFI3781KRJF.bai has IndexActivity with indexed = IGVFFI2580STXM.bam. IGVFFI2580STXM.bam has AlignmentActivity with three FASTQs as reads.
4. Name and lineage agree: HG01466.chr15.hc.vcf.gz.tbi in ANVIL_T2T_CHRY has one input with named_by = filename_rule, repository_metadata, repository_activity.
5. The report: uv run python scripts/generate_reconcile_report.py --run-dir $C --markdown $TMPDIR/rr.md --html $TMPDIR/rr.html. Section "Lineage: how each file was made" should show:
   • repository_activity 239,166 resolved, 84,029 sample parents and 8,246 untranslated; repository_metadata 241,821 resolved and 15,017 untranslated;
   • Conflicts: "None";
   • Misfits: AlignmentActivity with required role reference missing, 4,396 files in all.
6. Inference output is unchanged: the inference files are only symlinked and are never written. Hash them before and after with shasum $R/*.json. Every slot result in reconcile_report.json (slots, values, conflicts, inputs) equals the shared run's existing report.
7. make test.

Definition of done, from #577:

• Given a T2T_CHRY .tbi that Index files must not opine when two candidate parents share a filename #438 declines, then its reconciled generated_by is an IndexActivity named by anvil_activity, with its VCF's parent_key. → step 1 (and test_ac1_*)
• Given HG01536.samtools.stats.txt on a T2T participant row, then it has a QualityControlActivity with its own row's CRAM. Given a sample with a GRCh38 and a CHM13 CRAM, then each stats file gets the CRAM on its own row. → step 2 (and test_ac2_*)
• Given IGVFFI3781KRJF.bai, then it has IndexActivity → IGVFFI2580STXM.bam, and that BAM has AlignmentActivity with its FASTQs as reads. → step 3 (and test_ac3_*)
• Given a .tbi whose name and anvil_activity name one VCF, then it has one input with both sources in named_by. Given they name different VCFs, then the report lists an edge conflict. → step 4, and test_ac4_* for the conflict (0 occur in the real data)
• The inference output is unchanged (6.3), and the report shows each section above. → steps 5–6 (and test_ac5_*)

🤖 Generated with Claude Code

https://claude.ai/code/session_016p3omgkuHduf1UQjLH4K6t

[Copilot]

Copilot review overview

🟡 Changes recommended

Generic-only activity steps currently produce generated_by instead of the required edge conflict.

Review effort: Balanced
Findings: 1

Open (1)

• Reject generic-only lineage without a matching specific parent · New

What changed in this PR

Adds lineage-based generated_by reconciliation and reporting.

Changes:

• Resolves and merges imported lineage with inferred steps.
• Reports lineage outcomes, conflicts, misfits, and rule usage.
• Adds comprehensive fixtures and acceptance tests.

File	Description
src/​meta_disco/​reconcile.py	Integrates lineage reconciliation.
src/​meta_disco/​reconcile_lineage.py	Translates and resolves lineage.
src/​meta_disco/​edges.py	Merges steps and detects conflicts.
src/​meta_disco/​activities.py	Adds activity agreement logic.
src/​meta_disco/​activity_map.py	Exposes lineage details and map digest.
src/​meta_disco/​output_utils.py	Adds shared relative-path helper.
scripts/​generate_reconcile_report.py	Renders lineage reporting.
scripts/​generate_rules_report.py	Includes activity-map edge rules.
docs/​reconcile-dashboard-template.html	Displays lineage metrics.
docs/​claims-contract.md	Documents lineage reconciliation.
CLAUDE.md	Updates architecture guidance.
tests/​test_reconcile_lineage.py	Tests acceptance criteria and edge cases.
tests/​test_edges.py	Tests step merging and misfits.
tests/​test_reconcile_report.py	Tests lineage report rendering.
tests/​test_rules_report.py	Tests activity-map rule inclusion.
tests/​test_activity_map.py	Updates activity-map integration coverage.
tests/​lineage_fixtures.py	Provides shared lineage fixtures.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

It changes corpus-scale lineage resolution and provenance semantics across reconciliation, reporting, and downstream inheritance.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Reject generic-only lineage without a matching specific parent

[Copilot]

Copilot review overview

🟢 Approval recommended

The implementation matches the stated lineage contract and is supported by focused coverage of resolution, conflicts, provenance, reporting, and artifact preservation.

Review effort: Balanced
Findings: None