Add the activity translation table for lineage raw words (#584)

[NoopDog]

Closes #584

What changed

• src/meta_disco/rules/activity_map.yaml (new) maps what a lineage line says about its step to an activity of activities.yaml and the role its parent takes. It holds nine authored rows and 31 seeded ones.
  • Authored: AnVIL's Indexing, Checksum and Alignment: STAR.
  • T2T sample tables: QC, alignment, variant calling and indexing.
  • IGVF: index-of-alignments/fragments and alignments-from-reads.
• src/meta_disco/activity_map.py (new) holds the loader, row selection, seed() and review().
  • A row's match names source_type and any of the other parts: table, raw_activity, child_column, parent_column, child_data_type, parent_data_type.
  • Each part is exact, null (the line has none), or a list.
  • No two rows may match one line.
  • The loader refuses an activity activities.yaml does not declare, and a role that activity does not declare.
• make seed-activity-map appends a row for each key no row matches yet. It never rewrites a row.
• make review-queue gains a lineage section in docs/review-queue-report.md and docs/review-queue.html: the unauthored keys by kind of source, with line counts, then the authored rows.
• Shared, not copied: src/meta_disco/yaml_rows.py now holds the YAML walk and the write-then-rename seeding step, used by both value_map and activity_map. ReportColumn and md_rows move to summaries.py. value_map's behaviour is unchanged, and its tests pass as they were.

Why

The lineage importer (#583) writes each lineage link in the source's own words: Indexing, a column pair like chr7_hcvcf_gz / cram, or IGVF's bare derived_from. Reconcile (#577) needs to know which activity each one is, and which role the parent plays. That role decides what the parent passes on to the file made from it.

Assumptions I made

• A step is keyed by up to seven parts, not one string. Submitter tables and IGVF lineage have the same source_type (repository_metadata), so nothing per source says which key applies. Every row names the parts it cares about instead, and nothing is joined into one string.
• IGVF's step is keyed by the child's and the parent's content_type. It is read as the raw data_type value that the same catalog, dataset, table and kind of source gives each file in the slot evidence (Give the 10 IGVF SHARE-seq files a data_modality from their own evidence #570). The parts are named child_data_type / parent_data_type, not after IGVF's column. A file its table gives two data_type values is keyed with none, and lands in the queue.
• Only the latest catalog is read. The latest is the catalog holding the newest import stamp, not the one whose name sorts last (anvil9 would sort after anvil16). An older catalog left on disk is history and is never read, and only slot evidence of the same catalog is joined. Archiving obsolete catalogs is a separate issue.
• Rows are scoped by table, not dataset. A future dataset whose participant or file table has the same columns would match those rows without review. An optional dataset scope, where the narrowest row wins, is Let an activity-map row name its dataset, the narrowest row winning #587.
• A seeded row names all seven parts, with null where the line has none. That keeps two seeded rows from ever matching one line. Authoring a row that covers seeded keys means deleting those seeded rows, or the table will not load.
• The authored anvil_activity rows also name parent_column: used_file_id. Alignment: STAR has sample parents too (634 lines). All 1,268 of its file parents in anvil15 are FASTQs.
• The T2T index row matches every combination of its two column lists, not only the named pairs, and its reason says so. No other pair occurs in the data.
• Configuration parents (seqspec, barcode files) stay seeded, and no configuration role is added yet. Their children's own activity (kallisto output, count matrices) is still undecided.
• Matching is exact, with no casefolding, so Quantificatioin: salmon stays as AnVIL spells it.

How to verify

Run everything from the repository root.

1. make seed-activity-map. Expect Scanned 16 lineage files, 588,279 lines; added 0 seeded rows: the committed table already holds every key.
2. Delete the 31 seeded rows (every row without a reason) from a scratch copy, then run uv run python scripts/activity_map.py --table <copy> seed twice. Expect 31 added on the first run and 0 on the second.
3. In a scratch copy, change activity.checksum's role to reads, then run uv run python -c "from pathlib import Path; from meta_disco.activity_map import load_activity_map; load_activity_map(Path('<copy>'))". Expect a ValueError naming activity.checksum, saying that ChecksumActivity does not declare reads.
4. make review-queue. Open docs/review-queue.html and read the section "Lineage: what each step is". Expect the Repository activity group (Unknown, Sequencing, Track: bedGraphToBigWig, …), then the Submitter group (the IGVF kallisto and count-matrix pairs), then the authored mappings with their line counts (activity.indexing at 223,724).
5. make test. It includes tests/test_activity_map.py.

Definition of done, from #584:

• Given the anvil15 lineage evidence, then seeding lists every raw activity and column pair it holds, and a second seed adds nothing. → steps 1 and 2
• Given a row naming an undeclared activity, or a role its activity does not declare, then loading fails naming the row. → step 3 (and test_ac2_*)
• Given the first authored rows, then they load, and the review queue lists the rest. → step 4
• Inference output is unchanged. → test_ac4_nothing_but_the_review_queue_imports_the_table: nothing but the review-queue script imports the module, and nothing in a run or in reconcile reads the table.

🤖 Generated with Claude Code

https://claude.ai/code/session_016p3omgkuHduf1UQjLH4K6t

[Copilot]

Copilot review overview

🔵 Needs a closer look

The domain-specific mappings and corpus-derived lineage joins warrant final human validation.

Review effort: Balanced
Findings: None

What changed in this PR

Adds the activity translation layer required to convert imported lineage evidence into declared activities and parent roles for future reconciliation.

Changes:

• Adds activity-map loading, matching, validation, seeding, and tests.
• Shares YAML-row and report-table utilities with the existing value map.
• Extends review-queue reports and Make targets with lineage activity mappings.

File	Description
Makefile	Adds activity-map seeding commands.
CLAUDE.md	Documents the activity-map architecture.
src/​meta_disco/​activity_map.py	Implements activity-map processing.
src/​meta_disco/​rules/​activity_map.yaml	Defines authored and seeded mappings.
src/​meta_disco/​yaml_rows.py	Centralizes shared YAML utilities.
src/​meta_disco/​value_map.py	Adopts shared YAML/report utilities.
src/​meta_disco/​summaries.py	Hosts shared report-table helpers.
scripts/​activity_map.py	Adds the activity-map CLI.
scripts/​generate_review_queue.py	Adds lineage sections to reports.
tests/​test_activity_map.py	Tests validation, matching, seeding, and reporting.
docs/​review-queue-report.md	Publishes the lineage review queue.
docs/​review-queue.html	Publishes the HTML lineage report.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

Dataset filtering can incorrectly select lineage from an obsolete catalog.

Review effort: Balanced
Findings: 1

Open (1)

• Filter dataset only after selecting the latest catalog · New

[Copilot]

Copilot review overview

🟢 Approval recommended

The implementation matches the stated acceptance criteria and includes focused coverage for its validation, matching, seeding, and reporting behavior.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Filter dataset only after selecting the latest catalog