Warn when relationship to_columns does not cover a declared key - #330
Conversation
The spec defines to_columns as "Primary/unique key columns in the 'to' dataset", but the validator only checked that relationship datasets exist. A relationship joining to non-key columns breaks many-to-one semantics downstream (see apache#301 for a converter emitting exactly this and consumers rejecting it). validate_references now checks that to_columns covers the to dataset's primary_key or one of its unique_keys. Coverage rather than exact equality: a superset of a key still guarantees the join cardinality, and the databricks converter's _covers_unique_key already applies the same semantics. Reported as a warning rather than an error because declared keys may be an incomplete recovery of the dataset's real keys, and datasets that declare no keys are skipped entirely. Shape guards keep the semantic check from crashing or misreporting on documents that already fail schema validation (null unique_keys, non-list to_columns, flat unique_keys). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: km <kayemkim@gmail.com>
|
Two updates since opening this. #337 touches the same Also a correction to the summary above: "all still pass" overstates it. A few repo files fail validation on main as well, for reasons unrelated to this change: |
Extends the validator test suite added in apache#330 with cases for the metric scoping and metric name checks. Each test under "reported in review" corresponds to a defect found in review of apache#343 and fails against the validator as it stood before that review: raw-cased qualifier comparison, three-part STRUCT paths read as dataset references, local aliases and subquery sources reported as cross-dataset references, a traceback on an explicitly null expression, non-deterministic collision output, and the missing field/metric name collision check. Also covers the deliberately permitted cases, so a later change does not constrain them by accident: two datasets may reuse a metric name, and a model-scoped metric may take the name of a field or of a dataset. Follows the module-loading and importorskip pattern established by the existing tests. sqlglot is skipped rather than asserted, since the scoping checks no-op without it and would otherwise pass without asserting anything. Assisted-by: Cortex Code <noreply@snowflake.com>
Rule 1 said a dataset-scoped metric's expression references fields of its dataset. That forces an author to declare a field for a column they only want to aggregate, and then hide it. It also narrowed something the spec already states: the Fields section describes fields as "row-level attributes that can be used for grouping, filtering, and in metric expressions". The expression now reaches both namespaces, with a distinct spelling for each. A declared field is written dataset_name.field_name, which is how a metric reuses a field's expression instead of repeating it. A column of the source is written unqualified. Two spellings rather than one shared namespace avoids a shadowing rule. A field and a source column may share a name without ambiguity, so declaring a field named after an existing column does not change the meaning of an expression already using the bare name. This replaces the hard error on self-qualification, since orders.amount now means the declared field amount, with a check that a qualified reference names a declared field of the declaring dataset. That is verifiable from the model, so SUM(orders.tax) is reported and points the author at SUM(tax). Whether a bare name is a real column stays unchecked, because it needs catalog metadata the model does not carry, and nothing else in validate.py checks a field's expression against real columns either. validate.py: _leading_qualifiers becomes _qualified_references, returning (qualifier, name) pairs so the name a qualifier introduces can be resolved against the dataset's field list. spec.md: the worked example now demonstrates both spellings on a derived field rather than two identity fields, and the second INVALID case is a qualified reference to an undeclared name instead of a self-qualifier. Tests: 21 functions added overall, collecting 32 cases with the 10 from apache#330. Reverting only validate.py to 592db69 fails 11 of the added functions and none of the 10 pre-existing ones. Assisted-by: Cortex Code <noreply@snowflake.com>
Summary
The spec defines
to_columnsas "Primary/unique key columns in the 'to' dataset" (core-spec/spec.md, osi-schema.json), but the validator only checks that the relationship's datasets exist. A relationship that joins to non-key columns silently breaks many-to-one semantics downstream — joins fan out, and consumers that trust the declared cardinality produce wrong results. #301 shows this class of document being produced in practice and rejected by downstream consumers; a validator-side check catches it at the document level regardless of which tool produced it.validate_referencesnow checks thatto_columnscovers thetodataset'sprimary_keyor one of itsunique_keys. Two deliberate softenings, both drawn from how the repo already treats this rule:to_columnsthat is a superset of a key still guarantees the join cardinality (tenant-sharded joins are a common shape), and the Databricks converter's_covers_unique_keyalready applies exactly these semantics.unique_keysfrom a single join'srelyhint), so a non-coveringto_columnsis suspicious but not provably wrong. The message goes through the validator's existing warning channel and does not fail validation. Happy to tighten this to an error if you'd rather trust key declarations as exhaustive.Datasets that declare no keys are skipped — both
primary_keyandunique_keysare optional, so there is nothing to check against. The check also guards against shapes that already fail schema validation (nullunique_keys, non-listto_columns, flatunique_keys) so the semantic pass can't crash or mislead on an invalid document.Ran against every semantic-model YAML in the repo (examples plus converter fixtures): no regressions, all still pass. A relationship pointing at non-key columns now reports:
A couple of adjacent gaps I noticed but left out of scope, flagging in case they're worth issues: no CI workflow currently runs
validation/tests(this suite included), and the orionbelt converter's mirror validator (which documents itself as mirroringvalidate.py) doesn't have this rule — I can follow up on either. #307 (pending) adds from/to column count validation to the same function and introduces the same test file path; the checks are complementary and I'm happy to rebase if it lands first.Related Issues
Related to #301 (validator-side guard for the same class of invalid document; does not fix the converter itself).
Checklist
Specification
core-spec/and follow the existing structureOntology
ontology/are consistent with spec changesConverters
converters/is updated to reflect spec or ontology changesValidation
validation/are updated if the spec changedDocumentation
docs/is updated to reflect any user-facing changesCONTRIBUTING.mdis updated if the contribution process changedExamples
examples/are added or updated for any new spec constructs or converter supportTests
pytest/ CI green)Compliance