Skip to content

implementation.md

Latest commit

 

History

History
694 lines (547 loc) · 54.2 KB

implementation.md

File metadata and controls

694 lines (547 loc) · 54.2 KB
Raw

BioFSharp.INSDC roadmap: persistent, curatable ArcIR

Status: ACTIVE (2026-08-27). This is the single authoritative implementation plan for the repository. Historical plans are retained in place and classified in the plan registry. The state-artifact and ARC-native provenance model was clarified on 2026-08-27 before persistence work began.

Purpose

The repository already reads, writes, crawls, stores, and maps INSDC metadata into a useful proof-of-concept property graph. The next objective is to turn that proof of concept into a target-neutral intermediate representation that can be persisted and curated inside an ARC:

  • F1 ingests source metadata into one initial immutable ArcIR state artifact.
  • Curation processes consume a selected state artifact and generate a new immutable state artifact while preserving fragment-level provenance.
  • F2 compiles a selected ArcIR state into target artifacts without changing the IR.
  • The curation ARC's native process graph records every process and its designated input and output entities. In an ISA-formatted ARC these are ordinary process rows, not a parallel BioFSharp provenance format.

The design of this system is itself an ARC. Source XML, SSSOM mapping sets, diagnostic reports, immutable ArcIR state files, and generated target files are ordinary artifacts connected through the ARC process graph. Transformation semantics are identified by terms from a small versioned ontology and represented in the ARC's native process model; there is no standalone transformation-record text format.

source artifacts + SSSOM --F1 process--> ArcIR state 0 --curation process*--> selected ArcIR state --F2 process*--> target artifacts

Locked architectural decisions

  1. Immutable complete IR states. F1 and every completed curation transaction generate a new, complete *.arcir.json artifact at a new immutable path. Earlier states remain in the ARC. Filenames use stable process IDs, digests, or another opaque scheme rather than requiring sequential numbers; the native ARC process graph establishes succession and the selected F2 input.
  2. No ArcDocument domain envelope. ArcIR is the in-memory graph. Its JSON root is only the persistence boundary { "formatVersion": "1.0", "graph": ... }; process history and reports do not live in it.
  3. No embedded diagnostics. F1, curation, and F2 use one Diagnostic/DiagnosticReport model. Reports are sibling artifacts generated by processes, so later validation cannot mutate an older IR state.
  4. No patch chain as authority. Every state is independently loadable. Git diffs remain useful for review and patches may be exported as derived conveniences, but replaying patches is never required to reconstruct a state.
  5. Fragments designate occurrences; processes establish provenance. A selector is meaningful only with the immutable artifact path and digest, plus an optional Git commit for resolution. That pair designates an existing entity or value occurrence. A native ARC process relates designated inputs to generated outputs; the process topology establishes an output entity's identity by provenance, while a selector designates its persisted representation.
  6. Stable IRI identity for curatable units. Objects, type assertions, properties, relations, annotations, diagnostics, and reports that can be curated or referenced independently use the same validated absolute Iri/URN identity type and are serialized as keyed maps, not positional arrays. Their containing map or typed navigation case supplies the role; parallel nominal wrappers such as AssertionId and RelationId are not introduced unless a role later acquires genuinely different validation semantics.
  7. Scalar occurrences are designated, not intrinsically identified. ArcValue and AnnotationValue literals do not gain IDs. Their occurrence is addressed through the containing immutable ArcIR artifact and a typed JSON Pointer. A repeated value that must be curated independently is represented as its own identified assertion; ArcValue.List remains atomic.
  8. SSSOM contains reusable semantic mappings only. Extraction paths, parsing, splitting, graph construction, identifier minting, record-specific execution, and provenance do not belong in mapping rows. Fragment selectors designate native process inputs and outputs, transformation ontology terms identify process semantics, protocols and parameters carry execution details, and native process topology expresses provenance. Mapping sets may be partial; an unmapped local term remains valid ArcIR state without a placeholder row.
  9. F1 runs once per curation lineage. Later corrections are explicit curation processes. A materially new source import starts an explicit new lineage; it does not silently replace curated work.
  10. F2 is repeatable and non-mutating. It consumes one exact ArcIR state artifact and generates target artifacts, native ARC process entries, and a diagnostic report. This roadmap defines the interface but no concrete F2 target.
  11. Breaking changes are accepted. There is no compatibility facade for the proof-of-concept ArcIR API or its current HTML export.
  12. Local terms remain legitimate. An unmapped INSDC or curation-created domain term is preserved as a stable local term and diagnosed; absence of an external ontology match does not make the IR invalid.
  13. The first workbench is read-only. Editing and committing curation actions through the UI belong to a later phase after the curation library and native ARC process integration are stable.
  14. Terms are shared graph-level definitions. ArcIR owns a Terms: Map<Iri, OntologyTerm> registry. Types, predicates, term-valued assertions, annotation properties, and units reference a term by IRI rather than embedding repeated term records. The registry describes terms actually used by the graph; it is not a bundled ontology, and conflicting definitions are merge conflicts rather than last-writer-wins updates.
  15. No parallel provenance serialization. Any in-memory addressing or operation helpers must translate into the ARC's native process model. For an ISA-formatted ARC, transformations become ISA process rows with designated inputs and outputs and ontology-typed protocols or parameters; they are not persisted as BioFSharp-specific transformation records.
  16. ArcIR mapping is format-neutral; SSSOM loading is an adapter. BioFSharp.ArcIR defines only neutral mapping claims and an additive graph-enrichment process. BioFSharp.INSDC.ArcIR may reference PolyglotSSSOM to parse and validate versioned SSSOM artifacts and project their rows into those neutral claims, but no PolyglotSSSOM type enters the ArcIR graph model or the neutral mapper API. The top-level application still owns claim selection, conflict policy, and recording used mapping artifacts and records as native ARC process inputs.

Phase 1 — Stabilize the existing repository

Status: COMPLETE (2026-08-24). The pre-change inventory is recorded in phase-1-baseline.md. The FAKE RunTests acceptance path now includes dependency auditing and generated-artifact drift checks.

Complete this phase before changing the ArcIR model so failures introduced by the redesign are distinguishable from existing debt.

Build, dependencies, and generated artifacts

  • Establish a green FAKE build/test baseline and record the SDK, tool, package, and test counts.
  • Resolve current package vulnerability warnings, beginning with the vulnerable SQLitePCLRaw.lib.e_sqlite3 dependency, while retaining netstandard2.0 for shipped libraries and net8.0 for the crawler.
  • Add a CI/FAKE dependency-audit gate with an explicit, reviewed suppression mechanism rather than accepting warning-only restores.
  • Make regenerateInsdcTypes, generateFragmentSelectors, and generateStructuralOntology reproducible and add a drift-check target that fails when committed generated output differs.
  • Keep generated C# under BioFSharp.FileFormats.INSDC/Generated/ generator-owned; all naming changes continue through typename-substitutions.txt.

Tests and documentation

  • Split the monolithic test file into focused modules without changing coverage or fixture ownership. Keep normal tests offline and live endpoint checks opt-in.
  • Add focused SQLite schema, foreign-key, identifier, and per-entity round-trip tests that close the unfinished acceptance work in the historical SQLite plan.
  • Document the current package boundaries, crawler exceptions, generator workflow, and IR terminology in package READMEs and XML documentation.
  • Remove stale references that still describe the structural decompilation overlay or embedded HTML as the intended final architecture.

SQLite and crawler hardening

  • Introduce an explicit SQLite schema version and ordered forward migrations; test a new database and upgrades from every committed schema version.
  • Make database writes transactional at the public operation boundary and preserve the crawler's intentional soft-reference behavior explicitly rather than through undocumented foreign-key assumptions.
  • Audit crawler retries, cancellation, partial failures, atomic file writes, resume behavior, and malformed upstream responses. Preserve injectable fetch seams and the offline fixture suite.
  • Keep R1 and R2 as crawler-materialized raw-artifact formats. They are not F2 implementations and do not require ArcIR.

Phase acceptance

  • The full FAKE build and offline suite pass with generated-artifact drift and dependency-audit gates enabled.
  • SQLite has focused migration and round-trip coverage independent of crawler integration tests.
  • Public projects have current package documentation and no undocumented warning suppressions.

Phase 2 — Extract the target-neutral ArcIR core

Status: COMPLETE (2026-08-26). BioFSharp.ArcIR now owns the validated, normalized graph model, lossless operations, validation, and persistence contracts. BioFSharp.INSDC.ArcIR is the INSDC-specific F1 adapter, and all eight entity converters plus supplementary ingestion build against the core.

Create a new packable BioFSharp.ArcIR project containing only the graph model, identity/value types, graph operations, validation, and persistence contracts. BioFSharp.INSDC.ArcIR becomes the INSDC-specific F1 adapter and depends on the core. The core must not reference generated INSDC types, INSDC vocabulary, crawler code, SQLite, or any target format.

Public model changes

Use one validated absolute Iri type for every URI-shaped model identity. Do not introduce nominal ArcId, AssertionId, RelationId, AnnotationId, or DiagnosticId wrappers merely to restate the role already expressed by a record field, map, or discriminated-union case.

  • ArcIR.Terms is Map<Iri, OntologyTerm>, where the map key is the authoritative term identifier and the value carries reusable label/source metadata without duplicating the ID.
  • ArcIR.Objects is Map<Iri, ArcObject>.
  • ArcObject.Types becomes an IRI-keyed map of type assertions rather than an unaddressable set. Each assertion references its ontology term through ArcIR.Terms.
  • ArcObject.Properties becomes Map<Iri, ArcProperty>. ArcProperty carries its IRI identity, predicate term IRI, value, and IRI-keyed annotations.
  • Object annotations become Map<Iri, ArcAnnotation>; relation annotations follow the same rule. Annotation properties, term values, and units reference ArcIR.Terms by IRI.
  • ArcIR.Relations becomes Map<Iri, ArcRelation>, and ArcRelation.Id is required rather than optional. Subjects, objects, evidence, source objects, and ArcValue.Ref also use Iri.
  • Repeated independently curatable values are separate property assertions. ArcValue.List is retained only for an ordered value that is deliberately curated atomically.

F1 mints deterministic absolute IRIs/URNs from the source artifact identity, source selector, mapping/rule ID, and assertion role. Identities are never derived from the current assertion value, so a value edit cannot rename its slot. Curation-created assertions use opaque URI-shaped identities. Existing identities survive unchanged across later states unless an explicit identity-changing process records their replacement.

Replace silent overwrite behavior with explicit operations:

  • add fails on an incompatible duplicate ID.
  • upsert is available only where replacement is intentional and returns the replaced value.
  • merge reports conflicts rather than selecting a winner silently.
  • Validation detects missing endpoints, references to missing term definitions, duplicate semantic slots where forbidden, and invalid IRIs. Artifact selector construction and resolution belong to Phase 3 rather than the state-only graph model.

Migration

  • Convert the current graph builders and all eight INSDC converters to the new maps and required IDs in one breaking change.
  • Do not retain aliases or obsolete overloads for the proof-of-concept shapes.
  • Add compile-time and golden-fixture tests before deleting the old core types from BioFSharp.INSDC.ArcIR.

Phase acceptance

  • The target-neutral package builds and tests without an INSDC dependency.
  • Every independently curatable graph element has a stable identity.
  • Every referenced ontology term has one reusable graph-level definition, and term-definition conflicts are reported explicitly.
  • Merging graphs cannot silently lose a property, annotation, or relation.
  • All eight current INSDC entity converters build against the extracted core.

Phase 3 — Deterministic state artifacts and fragment addressing

Status: COMPLETE (2026-08-27). BioFSharp.ArcIR now provides a strict, schema-versioned canonical JSON codec, immutable atomic state writes, SHA-256 artifact revisions, and typed RFC 6901 fragment designations for every selectable graph element and scalar value occurrence. The core suite contains 34 passing tests, including deterministic byte output, exhaustive location resolution, adversarial parsing, create-new concurrency, and a two-state fixture whose stable assertion IRI is unambiguous through artifact-qualified input and output selectors. Transformation vocabulary and native ARC process materialization remain deferred to Phase 6.

Canonical .arcir.json format

Add a reversible, schema-versioned serializer and parser for this normalized root:

{
  "formatVersion": "1.0",
  "graph": {
    "terms": {},
    "objects": {},
    "relations": {}
  }
}

"Canonical" describes the wire format, not one mutable file. F1 and each completed curation transaction write a new complete state artifact, for example arcir/states/<opaque-state-id>.arcir.json; an existing state path is never overwritten.

  • Commit a JSON Schema for each supported major format version.
  • Serialize all addressable collections as objects keyed by stable IDs.
  • Order keys deterministically, use invariant scalar formatting and UTF-8, and produce byte-identical output for the same graph.
  • Reject unknown major versions; migrate known older versions explicitly before model decoding.
  • Write a new state atomically with create-new semantics so interruption cannot leave a truncated artifact and an existing immutable state path can never be replaced.
  • Keep process history and transformation semantics outside the graph state.
  • Treat GraphML, JSON-LD, and visualization data as derived views, never as authoritative editable state.

Example physical locations are:

#/graph/terms/<escaped-term-iri>
#/graph/objects/<escaped-object-id>
#/graph/objects/<escaped-object-id>/properties/<escaped-assertion-id>
#/graph/objects/<escaped-object-id>/properties/<escaped-assertion-id>/value
#/graph/relations/<escaped-relation-id>

Typed locations and entity designations

Add neutral addressing types to the core library:

type ArtifactRevision =
    { Path: string
      Sha256: string
      Commit: string option }

type FragmentSelector =
    { ConformsTo: Iri
      Value: string }

type FragmentRef =
    { Artifact: ArtifactRevision
      Selector: FragmentSelector }

These support types do not add provenance fields, revision history, tombstones, or scalar IDs to the ArcIR state records completed in Phase 2.

Add a separate navigation type to the canonical JSON persistence component. It must cover every independently selectable graph element and every scalar value occurrence that can participate in a transformation, including at least:

type ArcJsonLocation =
    | Term of termId: Iri
    | Object of objectId: Iri
    | TypeAssertion of objectId: Iri * assertionId: Iri
    | Property of objectId: Iri * assertionId: Iri
    | PropertyValue of objectId: Iri * assertionId: Iri
    | ObjectAnnotation of objectId: Iri * annotationId: Iri
    | ObjectAnnotationValue of objectId: Iri * annotationId: Iri
    | PropertyAnnotation of objectId: Iri * assertionId: Iri * annotationId: Iri
    | PropertyAnnotationValue of objectId: Iri * assertionId: Iri * annotationId: Iri
    | Relation of relationId: Iri
    | RelationProperty of relationId: Iri * assertionId: Iri
    | RelationPropertyValue of relationId: Iri * assertionId: Iri
    | RelationAnnotation of relationId: Iri * annotationId: Iri
    | RelationAnnotationValue of relationId: Iri * annotationId: Iri

Include corresponding cases for annotations attached to relation properties. ArcValue.List is selected as one atomic value; there is no positional selector for a list member.

ArcJsonLocation is not part of the ArcIR graph model and is never stored in ArcIR. It is an ephemeral public navigation type used only to construct or validate a designation against one serialized state artifact. This lets a literal with no intrinsic ID be designated as a native process input or output, for example <state artifact>#/.../properties/<assertion>/value.

ArcIRJson.selector is the only component that converts an ArcJsonLocation to an RFC 6901 URI-fragment JSON Pointer. F1, curation, F2, and UI code must not hard-code the wire layout or escaping rules.

Use FragmentRef as an in-memory, format-neutral designation helper for source XML XPointer, RFC 7111 table fragments, ArcIR JSON Pointer, SSSOM mapping-record identity, and diagnostic targets. It does not define provenance or a new on-disk process format. Phase 6 translates designations into native ARC process inputs and outputs.

For persisted boundary entities, a native process records distinct input and output fragment references even when their embedded semantic ID is preserved across state artifacts. Intermediate process entities need not acquire an ArcIR scalar ID merely to participate in the process graph. ArtifactRevision.Commit is resolver metadata and may be absent from an artifact generated in that same commit; immutable path plus SHA-256 digest is sufficient to designate the bytes.

Phase acceptance

  • Semantic and byte-for-byte round trips pass for representative and adversarial graphs.
  • Golden files demonstrate deterministic ordering and JSON Pointer escaping for /, ~, Unicode, and URI-shaped IDs.
  • Every generated ArcJsonLocation, including literal and term-valued property or annotation values, resolves to exactly one fragment in its declared state artifact.
  • Attempting to write a state to an existing path fails without changing the existing bytes.
  • A two-state fixture demonstrates an updated assertion retaining its IRI while its input and output occurrences remain unambiguous through different artifact-qualified selectors.

Phase 4 — Canonical INSDC structural ontology modules and base mappings

Status: COMPLETE (2026-08-27). ER_ontologies now owns the five canonical INSDCER record modules and the five ARC structural ontologies (INVMSO, STDMSO, ASSMSO, AFSO, and APGSO). The reviewed base SSSOM 1.1 set contains seven profile-scoped exact administrative mappings with stable record IDs. All 11 OBO files in that repository parse as 534 unique terms, every mapping endpoint resolves, and the SSSOM artifact round-trips byte-for-byte through PolyglotSSSOM. F1 consumption of the base set remains Phase 5 work.

Create two kinds of versioned configuration assets for the integrating application, in this order:

  1. one canonical OBO-format ontology module per concrete supported INSDC record type; and
  2. after those source terms are stable, one base SSSOM mapping set containing only reviewed mappings for which the INSDC structural term and endpoint ontology term are known to be semantically equivalent.

The initial ontology scope is deliberately limited to these record types, authored in dependency order:

  1. Project/BioProject;
  2. Study;
  3. Sample/BioSample;
  4. Experiment; and
  5. Run.

Each concrete source record type gets its own .obo file. If inspection of the pulled schemas and ontology repository shows that Project and BioProject, or Sample and BioSample, are distinct source record types rather than aliases, keep them as separate modules instead of collapsing their semantics. Analysis, Submission, Receipt, and other record types remain outside this first ontology authoring pass.

The initial mapping scope is deliberately conservative. Administrative metadata such as titles, descriptions, people, affiliations, and other clearly equivalent Investigation metadata are good candidates. Similar labels, convenient target shapes, or mappings that require normalization, parsing, splitting, contextual interpretation, or value-level curation are not sufficient for the base set.

Ontology and mapping boundaries

  • Each structural ontology module describes one INSDC record entity, its fields, and their source semantics. It does not describe transformation processes, XPath execution, graph construction, or provenance.
  • Refactor the existing structural vocabulary into canonical .obo sources with stable ontology IRIs, version metadata, stable term IRIs, labels, definitions, and source annotations. Generated views may be derived from them, but are never the editing authority.
  • Keep record-context-specific concepts distinct even when their source labels are identical. For example, a project title, study title, sample title, experiment title, and run title do not become one structural term merely because the XML uses the same lexical field name.
  • Express cross-record relationships with stable references between modules. Do not copy a term into several modules or introduce a shared umbrella ontology in the first pass unless inspection demonstrates a genuine shared source concept.
  • Treat the pulled endpoint ontologies as the target vocabulary. Do not copy or silently redefine their terms in the INSDC ontology.
  • Keep the ARC structural ontologies in the endpoint repository as their editing authority. Copies embedded or packaged by ARCTokenization are synchronized compatibility snapshots, not independent ontology sources.
  • The base SSSOM file maps structural terms to endpoint terms. Its first version is primarily term-to-term structural mapping; literal and controlled-value mappings are added only when separately justified.
  • Give every mapping row a stable URI-shaped record ID and retain mapping-set metadata, mapping predicate, justification, creators, and version information.
  • Keep the set intentionally partial. An absent row means "not mapped yet" and creates no placeholder. sssom:NoTermFound is reserved for an explicit curated conclusion that no matching term exists.
  • Extraction selectors, normalization instructions, record-specific execution, and ARC provenance never appear in the ontology or SSSOM rows.

All reusable structural mappings still hard-coded in converters for the in-scope record types, plus reusable claims formerly expressed by the removed generated structural ontology, become candidates for the reviewed base mapping. Extraction itself remains explicit typed converter logic.

Runtime ownership

These assets do not introduce a PolyglotSSSOM dependency in the ArcIR core. The INSDC integration layer may depend on one exact PolyglotSSSOM revision to load, validate, and resolve mapping rows into format-neutral claims. Generic SSSOM format fixes remain upstream; no fallback TSV parser or generic SSSOM validator is added here. Pin the published PolyglotSSSOM package to the exact reviewed version rather than depending on a mutable sibling checkout.

Work sequence

  1. Inspect the endpoint ontology repository, source schemas, existing structural vocabulary, namespaces, and release conventions.
  2. Inventory the terms currently minted or hard-coded by the Project/BioProject, Study, Sample/BioSample, Experiment, and Run converters.
  3. Define the module/file naming scheme, ontology IRIs, term namespace, identifier policy, versioning policy, and cross-module reference rules.
  4. Author and validate the Project/BioProject module, then Study, Sample/BioSample, Experiment, and Run, without changing source meaning merely to resemble a target ontology.
  5. Review the completed source definitions against the endpoint ontology terms.
  6. Only then draft and review the conservative base SSSOM mapping set for obvious exact administrative mappings.
  7. Canonicalize and validate the ontology and SSSOM artifacts and record the inclusion/deferment review. Add the focused fixture showing how the base mapping configures initial F1 output with the format-neutral seam in Phase 5.

Phase acceptance

  • Every in-scope concrete record type has its own canonical, versioned OBO module and each module passes the chosen ontology validation path.
  • Project/BioProject, Study, Sample/BioSample, Experiment, and Run field coverage is explicitly inventoried; out-of-scope record types are not represented by speculative placeholder modules.
  • Every term used as the subject of a base mapping has a stable definition whose meaning is supported by the source metadata model.
  • Every base row is independently reviewable as an exact semantic mapping to a term from the pulled endpoint ontologies and carries stable record identity and mapping metadata.
  • The SSSOM artifact round-trips canonically through PolyglotSSSOM; no local parser, writer, or conformance implementation is introduced.
  • Uncertain, contextual, value-level, and not-yet-mapped terms remain absent from the base mapping without making the ontology or a resulting ArcIR graph invalid.
  • BioFSharp.ArcIR references neither PolyglotSSSOM nor the endpoint ontologies. Phase 5 may add the parser dependency to the INSDC integration layer, but ontology and mapping artifacts remain versioned external inputs rather than embedded runtime vocabularies.

Phase 5 — Additive semantic enrichment, complete F1 accounting, and unified diagnostics

Status: COMPLETE (2026-08-27). The neutral additive mapper, PolyglotSSSOM claim adapter, and occurrence-level accounting now cover every supported F1 input without adding a parallel provenance artifact.

Implemented slices: the shared diagnostic and field-accounting core, neutral additive mapper, PolyglotSSSOM adapter, and artifact-qualified accounting for all eight supported INSDC XML entities are implemented. Their tests resolve source XML XPointers and emitted ArcIR JSON fragments, including the source-assertion handoff to an additive base mapping, the connected BioSample-to-Experiment-to-Run chain, analysis files and references, submission agents, and receipt acknowledgements. Supplementary JATS front matter uses XPointer designations; count-header cells use W3C Data Position Selectors over the exact immutable bytes, so CSV, TSV, and materialized archive entries share one resolvable path. Every emitted output is qualified against and resolved from a persisted ArcIR state, and identical inputs produce deterministic reports and graph fragments.

One result and diagnostic model

Replace stage-specific F1Report/F2Diagnostic concepts with one diagnostic model:

type DiagnosticSeverity =
    | Info
    | Warning
    | Error

type Diagnostic =
    { Id: Iri
      Code: Iri
      Severity: DiagnosticSeverity
      Message: string
      Targets: FragmentRef list
      Related: FragmentRef list }

type DiagnosticReport =
    { Id: Iri
      Diagnostics: Map<Iri, Diagnostic> }

The producing ARC process establishes whether a report came from F1, curation, validation, or F2; Diagnostic therefore has no stage field. Historical diagnostics are immutable. A later curation process may designate a diagnostic occurrence in its immutable report artifact as addressed, but it never changes that old report or relies on a bare diagnostic ID without artifact context.

Do not introduce TransformationOperation, SourceOutputBinding, TransformationRecord, or an intermediate ProcessIntent persistence contract. Effectful operations become ontology-typed native ARC processes when Phase 6 instruments the F1 execution path directly.

Source-preserving additive mapping

F1 first emits source-faithful assertions using stable INSDC structural terms. A separate format-neutral enrichment pass applies caller-selected mapping claims by adding companion assertions to the same owning object or relation. It never replaces or removes the source assertion and never rewrites its term in place. The graph term registry retains both the source and mapped definitions.

Source term occurrence Additive result
Object type A second independently identified type assertion
Property or annotation predicate A companion assertion carrying the same value
Relation predicate A parallel relation with the same endpoints
Term-valued assertion A companion assertion carrying the mapped term value

The first implementation materializes only equivalence-safe claims such as the reviewed skos:exactMatch base rows. Broader, narrower, related, literal, and ambiguous claims remain explicit candidates until application policy authorizes a specific operation. A mapped assertion ID is deterministic from the source occurrence and target term, so applying the same claim repeatedly is idempotent. Generated companion IDs occupy a reserved ArcIR namespace and are not treated as fresh source occurrences by later mapper passes; replay therefore cannot derive companion-of-companion assertions.

The neutral application result retains the mapping claim ID and the input/output locations of every added or already-present companion assertion. This is an in-memory execution result, not a persisted transformation log. Phase 6 combines those locations with immutable artifact revisions and the SSSOM record identity to emit native ARC process inputs and outputs.

PolyglotSSSOM claim adapter

BioFSharp.INSDC.ArcIR uses PolyglotSSSOM for SSSOM decoding, validation, and declared CURIE expansion, then projects complete applicable rows into the neutral claim type. It does not implement another SSSOM parser and does not expose PolyglotSSSOM model types through the neutral mapper.

  • Preserve the stable mapping record_id, subject, predicate, object, justification, labels, and source metadata needed by the neutral claim.
  • Return all valid candidate claims; do not silently select among conflicting records or collapse them to a last-writer-wins Iri -> Iri map.
  • The top-level application selects the claims it authorizes. The initial profile selects only the seven reviewed exact rows from the Phase 4 base artifact.
  • Missing mappings leave the source assertions untouched. Invalid, unsupported, or ambiguous records produce diagnostics rather than invented expansions.
  • Canonical writing and mapping curation continue to use PolyglotSSSOM directly; the graph mapper consumes claims and never writes SSSOM.

F1 field accounting and resolvable designations

  • Complete explicit F1 mapping for all eight supported INSDC entities and supplementary paper/count metadata against the new core.
  • Give every extraction/emission rule a stable ID.
  • For every present source leaf from xpathEntries, record exactly one field-accounting outcome: emitted, intentionally ignored with a rule and reason, unsupported with a diagnostic, or failed with a diagnostic.
  • Construct resolvable designations for each source fragment and emitted ArcIR fragment, including literal value occurrences. Keep them separate from diagnostics and do not turn selectors into graph nodes.
  • Identify meaning- or representation-changing steps such as trimming changed whitespace, parsing a date, splitting a value, or adding a mapped companion assertion so Phase 6 can type the corresponding native processes. Mere selection and copying must not be padded into artificial operation records.

F1 generates the initial immutable ArcIR state artifact and a diagnostic report. Phase 6 extends the execution path to collect the stable extraction rule, mapping-record designations, transformation terms, and parameters while emitting them directly into the native ARC process model; Phase 5 does not persist or expose an intermediate provenance document.

Phase acceptance

  • Every leaf in each committed eight-entity fixture is accounted for.
  • Tests resolve every candidate source and output designation against its declared artifact.
  • Repeated F1 execution with identical artifacts, recipe, and resolved mapping context produces semantically identical IR and deterministic IDs.
  • The reviewed base mapping fixture loads through PolyglotSSSOM and produces neutral claims without introducing a PolyglotSSSOM dependency in BioFSharp.ArcIR.
  • Applying those claims preserves each original INSDC assertion, adds the expected ARC companion assertion on the same owner, and is deterministic and idempotent.
  • Conflicting target definitions or assertion identities are reported without returning a partially enriched graph; multiple applicable mapping claims are never resolved by silent precedence.
  • Missing mappings, lossy conversions, conflicts, and unsupported fields produce stable diagnostics while preserving usable source information.
  • No BioFSharp-specific transformation log or binding document is serialized.

Phase 6 — Top-level ARC-native F1 and curation integration

Status: READY FOR OVERARC (2026-08-27). All prerequisites owned outside the integrating application are complete. ER_ontologies owns the initial versioned Curation Transformation Ontology (CTRO), and playground/crawl_arcir.fsx demonstrates live project crawl, additive base mapping, validation, and canonical immutable-state publication. The remaining work in this phase—mapping policy, curation transactions, native ARC/ISA process emission, validation artifacts, and Git coordination—is owned by OverARC. This repository must not introduce a parallel provenance format or an ARC representation dependency to complete that application work.

Add the top-level integrating application layer that applies typed graph operations, configures F1 from mapping artifacts, and contributes to the ARC's native process model. The process graph, not an ArcIR file or BioFSharp-specific log, is the authority for lineage, agents, software versions, used/generated entities, and process relationships. Use the established representation library for each ARC realization; for an ISA-formatted ARC, emit ordinary ISA assay process rows rather than serializing a parallel provenance document.

Selected-literal core primitive

Status: COMPLETE (2026-08-28). ArcIRJson.parseLocation now round-trips supported typed selectors, and LiteralMapping.apply performs the immutable, source-preserving selected-literal transformation required by OverARC.

The core accepts an ArcJsonLocation, exact source literal, and already registered target term. Object and relation string properties, direct literal annotations, and nested property annotations receive deterministic term-valued companions. Lists and non-string scalars are deliberately rejected. Compatible replay is idempotent; missing or changed inputs, unregistered targets, invalid graphs, and deterministic companion collisions return typed failures without a partial graph. The primitive has no SSSOM, CTRO, ProcessCore, HTTP, or OverARC dependency; the application remains responsible for parsing mapping artifacts, choosing claims, and recording native provenance.

SSSOM ownership and initial F1 configuration

Use the exact PolyglotSSSOM revision introduced by the Phase 5 INSDC adapter and run its upstream acceptance path at that revision. The top-level application owns selection and provenance even though the adapter supplies parsing and claim projection:

  • Load and validate the Phase 4 base mapping plus later ARC-owned curated mapping sets through the Phase 5 PolyglotSSSOM adapter.
  • Preserve complete mapping claims, including predicates, justifications, creators, provenance, modifiers, extension metadata, and original documents required for round trips; never collapse the stored claim to Iri -> Iri.
  • Resolve CURIEs only through each mapping set's declared prefix map. Unresolved prefixes and conflicting applicable records produce diagnostics rather than invented expansions or silent precedence.
  • Require stable URI-shaped mapping record IDs. Designate a mapping by immutable mapping-set artifact revision plus record ID, never by filename alone, row number, or subject/predicate/object tuple.
  • Apply explicit application policy when combining the reviewed base mapping with project-specific curated mappings, then supply the resulting format-neutral context to F1.
  • Record each mapping record actually used by F1 or curation as an input to the corresponding native ARC process. Merely loading a mapping set does not imply that every row was applied.
  • For literal mappings, keep the literal ID-less. Its ArcIR FragmentRef designates the exact occurrence, while the mapping-set revision and record ID designate the reusable semantic claim.

The ArcIR core gains no PolyglotSSSOM dependency, registry client, remote prefix resolver, or duplicate SSSOM parser. The INSDC adapter uses only the pinned PolyglotSSSOM dependency and declared document prefix maps; it adds no remote resolver or fallback codec.

Versioned Curation Transformation Ontology

The initial Curation Transformation Ontology is maintained as ontologies/CTRO/CTRO.obo in nfdi4plants/ER_ontologies, with its competency examples beside it. It is the small ontology used by downstream integrations to identify the semantics of curation and conversion processes. It belongs to the provenance integration layer, not to the ArcIR graph model, canonical JSON schema, or fragment-addressing API. CTRO covers lexical normalization, splitting, parsing, additive semantic mapping application, creation, replacement, and deletion.

  • Give every transformation concept a stable absolute IRI, label, definition, and versioned ontology artifact.
  • Distinguish creating or curating a mapping claim from applying an existing mapping claim to an entity or literal occurrence. SEMAPV remains authoritative for mapping-claim creation and review; CTRO supplies mapping-application terms.
  • Treat a SSSOM mapping record as an input to a semantic-mapping process. The transformation term says that a mapping was applied; it does not duplicate the SSSOM subject, predicate, object, justification, or provenance.
  • Keep delimiters, parse formats, and similar execution details as process parameters rather than minting ontology terms for every parameter value.
  • Use the terms in the ARC's native process representation, such as ontology-typed ISA protocols or process parameters. Do not define a transformation-log serialization or make BioFSharp.ArcIR depend on this vocabulary.

State transitions

  • F1 native process topology consumes designated source artifacts/fragments and mapping artifacts/records and generates both the initial immutable ArcIR state artifact and its selected output fragments.
  • A completed curation transaction consumes one selected immutable ArcIR state artifact and generates a new complete state artifact at a different path. The native process topology records both whole-artifact lineage and the selected input/output fragments that explain fine-grained transformations. Earlier states are never overwritten or removed.
  • Each represented transformation uses a CTRO term together with native process inputs, outputs, protocols, and parameters. FragmentRef is only the library-side designation used to populate those native entities.
  • An updated assertion may retain its IRI while its input and output are distinguished by different state artifacts. Split, merge, replacement, creation, and deletion use the appropriate input/output cardinality and new IDs where semantic identity changes.
  • A scalar literal requires no ID: its artifact-qualified value selector designates the process input. A generated intermediate entity may receive identity from its position in the native process graph even when it is not persisted as an independently identified ArcValue; if it is persisted in ArcIR, its output fragment is also designated.
  • Validation runs before commit and generates its own report artifact when findings exist.
  • Git groups the new state, SSSOM edits, diagnostics, and native ARC process metadata into reviewable commits. It adds collaboration, integrity, and rollback but does not replace the semantic process graph. No JSON Patch chain is required.

Term correction and mapping promotion

A curation transaction may both correct an instance and add a reusable mapping for future ingests. Keep two transformation semantics distinct:

  1. Mapping curation consumes the relevant evidence and a prior SSSOM artifact when one exists, then generates a new SSSOM artifact containing a complete mapping record.
  2. Mapping application consumes the selected ArcIR fragment and the designated SSSOM mapping record and generates the curated ArcIR fragment.

Do not prematurely require these to be one or two rows in every ARC representation. Emit the native process topology that corresponds to the actual curation actions, and allow all resulting process entries and artifacts to be committed together. A pending mapping creates no incomplete SSSOM row; sssom:NoTermFound is used only for an explicit no-match conclusion. Promoting a mapping affects future F1 lineages and does not silently rerun F1 over the currently curated IR.

Phase acceptance

  • CTRO competency examples cover one-to-one, one-to-many, literal-to-term, creation, replacement, and deletion without defining a custom provenance document.
  • The Phase 4 base mapping is loaded through the Phase 5 INSDC adapter at its pinned PolyglotSSSOM revision and configures exact administrative mappings during initial F1 without entering the ArcIR core dependency graph.
  • An ISA-formatted fixture ARC demonstrates F1, an assertion update, a one-to-many split, a literal-to-term mapping, mapping curation, and validation using native process rows.
  • Every input and output designation resolves in its declared immutable artifact; deleted or replaced IDs remain resolvable in the process input state.
  • Reverting a commit restores the exact set of ArcIR/SSSOM/diagnostic artifacts and native process metadata that preceded it.
  • The native process graph can answer which source fragment, transformation term, mapping record, curator, and process produced any selected assertion.
  • No BioFSharp-specific transformation-record artifact or parallel provenance graph is written.

Phase 7 — Define the F2 compiler boundary

Status: TRANSFERRED TO OVERARC (2026-08-27). F2 orchestration selects an immutable state, a target profile, native ARC process metadata, and generated artifacts, so the concrete boundary belongs to the integrating application. BioFSharp.ArcIR remains the target-neutral state and fragment API and gains no ARC- or target-specific dependency for this phase.

Define a target-neutral compiler contract, a target-profile contract, and test doubles; do not implement an ARC, RO-Crate, ISA, readiness-level, or other concrete F2 output profile in this roadmap. The compiler reuses the Phase 6 ARC integration only to record F2 provenance in the curation ARC's native process graph.

An F2 compiler:

  • consumes one exact immutable ArcIR state artifact plus an immutable target profile or recipe;
  • validates profile preconditions without mutating the IR;
  • generates one or more target artifacts and the common diagnostic report;
  • contributes native ARC process entries whose designated inputs are the selected ArcIR fragments and whose designated outputs are the generated target fragments;
  • identifies transformation semantics with CTRO rather than a custom F2 record format;
  • is deterministic for identical inputs unless the profile explicitly declares a nondeterministic dependency.

F2 can run repeatedly for different profiles or after later curation commits. It never generates a new IR state merely because target validation found a problem; any resulting correction is a separate curation process.

Phase acceptance

  • A test compiler proves the interface with multi-file output, warnings/errors, native ARC input/output process entries, and deterministic reruns.
  • The Phase 6 fixture can select any immutable ArcIR state as F2 input and resolve every generated target binding through the native process graph.
  • F1, curation, validation, and F2 reports deserialize through the same diagnostic schema.
  • No target-specific types or dependencies enter BioFSharp.ArcIR.

Phase 8 — Replace the visualization layer

Status: TRANSFERRED TO OVERARC (2026-08-27). The maintained frontend now belongs to OverARC. The BioFSharp playground HTML writer remains only a low-effort inspection aid and is not a second workbench implementation.

Create the read-only React/TypeScript/Vite workbench in OverARC, using pinned stable Sigma v3, Graphology, and React Sigma packages. It reads one or more immutable .arcir.json state artifacts plus optional diagnostic and SSSOM artifacts and the native ARC process graph through shared, schema-validating loaders.

Workbench behavior

  • Search and filter by object kind, type, predicate, source artifact, mapping record, diagnostic severity/code, and process/state.
  • Use stable object/relation/assertion IDs for selection and deep links.
  • Show a detail inspector for properties, annotations, provenance, source/output fragment references, mappings, diagnostics, and process history.
  • Provide state and process views that explain what changed and why; use immutable ArcIR artifacts, native ARC process data, and Git rather than inventing history from the selected graph.
  • Distinguish missing endpoints and invalid fragments visually instead of silently dropping them.
  • Keep layouts deterministic where practical, move expensive layout work off the UI thread, and degrade gracefully for large graphs through filtering and progressive detail.
  • Support keyboard navigation, accessible color/shape encodings, copyable selectors, and export of the current filtered view.

Remove the embedded Html.fs viewer after the workbench covers its smoke-test use cases. Retain and improve GraphML as a derived interoperability export: preserve stable node/relation IDs, full IRIs, types, annotations, provenance references, and missing-node placeholders without claiming reversible persistence.

Phase acceptance

  • Unit tests cover schema loading, graph projection, filtering, fragment resolution, and deterministic style assignment.
  • Browser tests cover loading the representative fixture, searching, selecting assertions, viewing diagnostics/provenance, switching immutable states, and exporting a filtered view.
  • A large synthetic graph remains interactive within a documented performance budget.
  • No workbench action mutates or commits the curation ARC in this phase.

End-to-end evaluation planning gate

End-to-end evaluation is a human-in-the-loop activity and is not authorized by implementation of the earlier phases. When the libraries, test compiler, and workbench are ready for this review, stop and plan the methodology with the project owner before creating or publishing an evaluation run. Do not select a record, claim semantic/scientific validity, create an external evaluation repository, or push to a DataHUB instance autonomously.

The review must not begin from an empty script. Before the planning session, prepare a proposed versioned protocol and workspace scaffold for review, but do not execute it. The planning session must explicitly lock:

  • a representative INSDC record and the reasons it exercises the intended mapping and curation risks;
  • an evaluation workspace outside this source repository, structured as an ARC and initialized as its own Git repository;
  • the exact source, mapping-set, immutable ArcIR state, diagnostic, native ARC process, target, and review entities/artifacts to retain, including expected commits and digest bindings;
  • the automated structural/accounting checks, the manual semantic review checklist, expected curator interventions, pass/fail criteria, and who records approval;
  • the target DataHUB development instance/project, access and data-handling constraints, and a separately authorized push/publish procedure;
  • invalidation rules requiring review again when relevant code, mappings, protocol, source record, or generated artifact digests change.

Automated tests may establish structural validity, deterministic persistence, complete field accounting, and traceability. Only the planned human review may establish that the selected record's semantic mapping and curated result are acceptable.

End-to-end completion criteria

These criteria are the starting checklist for that future planning session, not permission for an autonomous end-to-end run:

  • A real committed INSDC fixture set can be ingested once into an initial .arcir.json state, inspected, curated through multiple immutable state artifacts and Git commits, and consumed by the test F2 compiler.
  • Any independently curatable assertion or designated scalar occurrence can be selected in its exact artifact and traced through the native ARC process graph to source fragments, transformation terms, mappings, diagnostics, processes, and generated target fragments.
  • IR persistence is deterministic, reversible, schema-versioned, and independent of INSDC or any F2 target.
  • Diagnostics are external immutable process outputs, while every ArcIR artifact contains only one graph state.
  • The read-only workbench replaces the barebones HTML visualization and accurately exposes the same provenance and addressing model used by the libraries.

Explicit non-goals for this roadmap

  • A compatibility layer for the proof-of-concept ArcIR API or serializer-default shapes.
  • A second automatic F1 pass over an already curated lineage.
  • Overwriting earlier ArcIR states, patch-chain persistence, requiring sequentially numbered state filenames, or embedding diagnostics/history in the IR.
  • A BioFSharp-specific transformation-record file or parallel provenance graph outside the ARC's native process model.
  • Assigning intrinsic IDs to scalar ArcValue or AnnotationValue occurrences that are already designated by artifact and selector.
  • A concrete production F2 target.
  • A browser-based curation editor or Git writer.
  • Requiring every local INSDC term to have an external ontology mapping.
  • Treating GraphML, JSON-LD, the workbench projection, or SSSOM as the ArcIR persistence format.

Historical-plan registry

DONE means the original acceptance criteria were implemented and the document remains historical design context. SUPERSEDED means the plan is incomplete or its architecture was replaced; implemented portions remain historical, while outstanding work is consolidated here.

Status Historical plan Reason
DONE claude/implementation.md Generated INSDC model, eight IO modules, fixtures, build, and original verification were completed.
DONE claude/crawler.md Discovery, fetching, SQLite persistence, retries, logging, resume, and offline/live-gated tests were implemented.
DONE claude/r1-crawlers.md The R1A/R1B raw-artifact crawlers and their shared-fetch behavior are implemented.
DONE claude/r2-crawlers.md XML, paper, DEE2, and composed R2 crawling are implemented.
DONE claude/arcir-conversion-restructure.md Explicit, decompilation-decoupled converters were rolled out to all eight entities.
DONE claude/arcir-graphml.md GraphML and the embedded HTML proof-of-concept viewer were implemented; their future differs as described above.
DONE claude/arcir-ingest.md Supplementary paper/count ingestion and RFC 7111 fragments are implemented.
DONE claude/xml-fragment-selectors.md Generated structural selectors, runtime XPath/XPointer lookup, and serializable XPath entries are implemented.
SUPERSEDED claude/arcir-mapping.md Its graph foundation landed, but its settled structural-decompilation overlay was removed by the explicit conversion restructure.
SUPERSEDED claude/arcir-export-readiness.md R1/R2 moved to the crawler; the proposed R0/R3/R4 F2 architecture is incomplete and replaced by the compiler boundary here.
SUPERSEDED claude/sqlite-store.md The store landed, but its focused acceptance suite and migration work are consolidated into Phase 1.
SUPERSEDED glm-5.2/arcir-export-readiness.md This earlier export draft was already partially superseded and is now fully historical.
SUPERSEDED glm-5.2/r2-crawlers.md Replaced directly by the as-built claude/r2-crawlers.md.

Execution order and release discipline

  • Deliver phases in order. A later phase may prototype against an earlier branch, but it does not merge until the preceding phase's acceptance criteria pass.
  • Keep model/wire-format changes and generated migrations in explicit release notes. Increment the ArcIR format major version for breaking wire changes and package major versions for breaking public API changes.
  • Use FAKE targets as the solution-wide source of truth; CI remains a thin invoker.
  • Each phase ends with updated schemas, fixtures, XML documentation, package READMEs, release notes, and a phase-scoped offline example where appropriate. Such examples are not the end-to-end evaluation, which remains behind the explicit human planning gate above.