How to measure the test coverage of a schema?

[ralfhandl]

Given a schema and a bunch of instances that pass validation: how can I measure which parts of the schema have been "used" or "satisfied", and which (properties, oneOf/anyOf alternatives, ...) would need additional instances to be "hit"?

Pretty much what branch coverage means for source code.

Which tools are out there that I could use in a build pipeline? That is command-line interface, no GUI.

Could this tool be used for this?

Thanks in advance!

[jdesrosiers]

That's a good question. This library doesn't have that functionality out-of-the-box and either does any library I've ever heard of. But, I do expose a lot of low-level components that could be used to build a simple script to get that functionality. It would be trivial to get the list of keyword locations in a schema and the list of keywords that failed validation across a number of schemas. I'd have to think about if there's a way to get a list of locations that passed validation, but if there is, you can easily use those three sets to determine the test coverage for the schema. I'll think about that last part an let you know.

[jdesrosiers]

There's not a good way to determine keywords that passed validation, but here's a hack I came up with. You can run it as a script that takes two arguments: a path to a schema and a path to a directory of JSON instances. Example: node coverage.js subject.schema.json test.

import { readdir, readFile } from "node:fs/promises";
import { join } from "node:path";
import { argv } from "node:process";
import "@hyperjump/json-schema/draft-2020-12";
import { compile, getSchema, interpret, Validation } from "@hyperjump/json-schema/experimental";
import * as Instance from "@hyperjump/json-schema/instance/experimental";

/**
 * @import { AST } from "@hyperjump/json-schema/experimental"
 * @import { Json } from "@hyperjump/json-schema"
 */


/** @type (testDirectory: string) => AsyncGenerator<Json> */
const tests = async function* (testDirectory) {
  for (const file of await readdir(testDirectory, { recursive: true, withFileTypes: true })) {
    if (!file.isFile() || !file.name.endsWith(".json")) {
      continue;
    }

    const testPath = join(file.parentPath, file.name);
    const testJson = await readFile(testPath, "utf8");
    yield JSON.parse(testJson);
  }
};

/** @type (testDirectory: string) => Promise<void> */
const runTests = async (testDirectory) => {
  for await (const test of tests(testDirectory)) {
    const instance = Instance.fromJs(test);

    interpret(compiled, instance);
  }
};

/** @type (ast: AST) => string[] */
const keywordLocations = (ast) => {
  /** @type string[] */
  const locations = [];
  for (const schemaLocation in ast) {
    if (schemaLocation === "metaData") {
      continue;
    }

    if (Array.isArray(ast[schemaLocation])) {
      for (const keyword of ast[schemaLocation]) {
        if (Array.isArray(keyword)) {
          locations.push(keyword[1]);
        }
      }
    }
  }

  return locations;
};

///////////////////////////////////////////////////////////////////////////////

const schema = await getSchema(argv[2]);
const compiled = await compile(schema);

/** @type Set<string> */
const visitedLocations = new Set();
const baseInterpret = Validation.interpret;
Validation.interpret = (url, instance, ast, dynamicAnchors, quiet) => {
  if (Array.isArray(ast[url])) {
    for (const keywordNode of ast[url]) {
      if (Array.isArray(keywordNode)) {
        visitedLocations.add(keywordNode[1]);
      }
    }
  }
  return baseInterpret(url, instance, ast, dynamicAnchors, quiet);
};

await runTests(argv[3]);
Validation.interpret = baseInterpret;

console.log("Covered:", visitedLocations);

const allKeywords = keywordLocations(compiled.ast);
console.log("NOT Covered:", allKeywords.filter((location) => !visitedLocations.has(location)));

[ralfhandl]

Thanks a lot, this is just what I needed!

Shamelessly copied it to

• main: schema test coverage OAI/OpenAPI-Specification#4173

This uses a lot of experimental things: how stable is this going to be?

[ralfhandl]

Another thing: the interpret function returns an object with a boolean property valid and nothing else, whereas the validate function also returns a property errors with error details if valid is false.

Instead of sorting the test instances into folders pass and fail I was toying with the idea of adding a property x-should-fail or similar to test instances that should fail, ideally also describing why it should fail, with info that can be compared to (the first item of) errors.

Which leads to more questions:

• How stable is the presence and contents of errors in the return object of validate?
• How can I get the same return object structure in the coverage analysis, allowing me to "interpret" or "validate" each test instance only once?

[jdesrosiers]

This uses a lot of experimental things: how stable is this going to be?

I was going to mention that, but forgot, so thanks for asking. Technically, experimental features can have backwards incompatible changes with any minor version bump. Realistically, any changes (compatible or not) are unlikely within this major version (v1). I have some thoughts for v2 that will include some major API changes, but that's a long way off and will be a major version bump anyway.

the interpret function returns an object with a boolean property valid and nothing else, whereas the validate function also returns a property errors with error details if valid is false.

Actually, interpret and validate have the same default behavior. validate is just a wrapper around compile and interpret. They use the FLAG output format by default. You can get error information by passing BASIC as the third parameter to interpret (same as validate). [Reference]

JSON Schema defines several standard output formats. These formats are considered unstable and are still evolving. This implementation supports the FLAG and BASIC formats as defined in 2019-09. Although BASIC is probably a more sensible default, I didn't want to have it enabled by default because it's unstable. I decided to consider FLAG stable because it's so simple it's not likely to change. That's why FLAG is the default, but it confuses everyone.

How can I get the same return object structure in the coverage analysis, allowing me to "interpret" or "validate" each test instance only once?

I didn't understand this question.

[ralfhandl]

Thanks, passing BASIC was the answer to the last question, which I should have formulated more clearly as

• How can I get the errors property in the response if valid is false?

Now I can combine the coverage analysis with the vitest schema tests you wrote for the OpenAPI specification schemas.

[ralfhandl]

@jdesrosiers Hi Jason,

I noticed a problem with the above approach:

• nested sub-schemas work just fine
• referenced sub-schemas are treated differently and report keywords as uncovered that are reported as covered if the sub-schema is nested instead of referenced.

Given this minimal instance

primitive: yeah
structured-nested:
  req: sure
  opt: nope
structured-ref:
  req: nope
  opt: sure

and a corresponding schema that should be completely covered by the minimal instance

$id: https://foo.org/schema
$schema: https://json-schema.org/draft/2020-12/schema
description: schema for testing schema coverage
type: object
$comment: root object
properties:
  primitive:
    description: Root property foo
    type: string
  structured-nested:
    $comment: nested object
    description: nested object
    type: object
    properties:
      req:
        description: required property
        type: string
      opt:
        description: optional property
        type: string
    required:
      - req
  structured-ref:
    $ref: "#/$defs/sub-object"
required:
  - primitive
$defs:
  sub-object:
    $comment: reusable object
    description: reusable object
    type: object
    properties:
      req:
        description: required property
        type: string
      opt:
        description: optional property
        type: string
    required:
      - req

the coverage result is

Covered: Set(22) {
  'https://foo.org/schema#/description',
  'https://foo.org/schema#/type',
  'https://foo.org/schema#/$comment',
  'https://foo.org/schema#/properties',
  'https://foo.org/schema#/required',
  'https://foo.org/schema#/$defs',
  'https://foo.org/schema#/properties/primitive/description',
  'https://foo.org/schema#/properties/primitive/type',
  'https://foo.org/schema#/properties/structured-nested/$comment',
  'https://foo.org/schema#/properties/structured-nested/description',
  'https://foo.org/schema#/properties/structured-nested/type',
  'https://foo.org/schema#/properties/structured-nested/properties',
  'https://foo.org/schema#/properties/structured-nested/required',
  'https://foo.org/schema#/properties/structured-nested/properties/req/description',
  'https://foo.org/schema#/properties/structured-nested/properties/req/type',
  'https://foo.org/schema#/properties/structured-nested/properties/opt/description',
  'https://foo.org/schema#/properties/structured-nested/properties/opt/type',
  'https://foo.org/schema#/properties/structured-ref/$ref',
  'https://foo.org/schema#/$defs/sub-object/properties/req/description',
  'https://foo.org/schema#/$defs/sub-object/properties/req/type',
  'https://foo.org/schema#/$defs/sub-object/properties/opt/description',
  'https://foo.org/schema#/$defs/sub-object/properties/opt/type'
}
NOT Covered: 5 of 27
[
  'https://foo.org/schema#/$defs/sub-object/$comment',
  'https://foo.org/schema#/$defs/sub-object/description',
  'https://foo.org/schema#/$defs/sub-object/type',
  'https://foo.org/schema#/$defs/sub-object/properties',
  'https://foo.org/schema#/$defs/sub-object/required'
]
Covered: 22 of 27 (81%)

The keywords that are reported as "NOT covered" in the reuse sub-schema below $defs are reported as covered in the identical nested sub-schema.

Looking at your code I cannot figure out what causes this difference.

I've packaged your code, the schema, and the test instance in GitHub repo https://github.com/ralfhandl/json-schema-coverage.

To reproduce

• clone repo
• go into folder
• npm install
• npm test

[jdesrosiers]

Sorry for the slow response. I've been away most of last week. It turns out that that dirty hack of Validation.interpret doesn't work for $ref because of how I defined that keyword. We can fix it with another dirty hack of for $ref.

// Add getKeyword
import { compile, getKeyword, getSchema, interpret, Validation } from "@hyperjump/json-schema/experimental";

const schema = await getSchema(argv[2]);
const compiled = await compile(schema);

const visitedLocations = new Set();
const baseInterpret = Validation.interpret;
Validation.interpret = (url, instance, ast, dynamicAnchors, quiet) => {
  if (Array.isArray(ast[url])) {
    for (const keywordNode of ast[url]) {
      if (Array.isArray(keywordNode)) {
        visitedLocations.add(keywordNode[1]);
      }
    }
  }
  return baseInterpret(url, instance, ast, dynamicAnchors, quiet);
};
// Hack $ref
getKeyword("https://json-schema.org/keyword/ref").interpret = Validation.interpret;

await runTests(argv[3]);

Validation.interpret = baseInterpret;
// Restore $ref
getKeyword("https://json-schema.org/keyword/ref").interpret = baseInterpret;

console.log("Covered:", visitedLocations);

const allKeywords = keywordLocations(compiled.ast);
console.log("NOT Covered:", allKeywords.filter((location) => !visitedLocations.has(location)));

The comments indicate lines that changed. This forces the $ref keyword to use the instrumented version of Validation.interpret. $dynamicRef has the same problem, but there isn't a way to hack around it like there is with $ref. I'll do a patch release tomorrow so $ref and $dynamicRef work like the other keywords without this extra hack.

[ralfhandl]

Cool, thanks for the quick fix! My test project now reports 100% coverage with version 1.9.9 😄

[ralfhandl]

@jdesrosiers The schema coverage measurement worked fine until version 1.11.0 of @hyperjump/json-schema, then it broke when we switched to version 1.12.1, without me noticing 😢

It now reports zero coverage, so something seems to be broken with collecting visited schema locations in
https://github.com/OAI/OpenAPI-Specification/blob/v3.2-dev/scripts/schema-test-coverage.mjs#L96-L110

Could you please have a look into what needs to be changed in https://github.com/OAI/OpenAPI-Specification/blob/v3.2-dev/scripts/schema-test-coverage.mjs to make measuring schema coverage work again?

Thanks in advance!

[ralfhandl]

@jdesrosiers Found it: the signature of Validation.interpret changed from

(url, instance, ast, dynamicAnchors, quiet)

to

(url, instance, context)

and the former ast now is context.ast. After changing this the schema coverage is correctly measured again.

How can schema coverage measurement become a standard feature of this package?

Thanks in advance!

[jdesrosiers]

Sorry about the breaking changes. I didn't think I'd be making changes in this area until v2, but inspiration strikes unexpectedly sometimes. Ironically, the change that broke your script is the foundation for providing this functionality in a stable way instead of the dirty hack we came up with.

The change allows you to provide an Evaluation Plugin where you can hook into the validation process to do all kinds of things including test coverage. I need to do a little more still to make it easier to use, but ideally the plugin interface should be stable and any changes I make to the internals shouldn't affect your plugin.

I'm going to try to prioritize finishing that up and maybe contributing an update to your script. I think test coverage is the perfect example to use to introduce is new feature!

[jdesrosiers]

@ralfhandl, I got inspired by this since our last update and just released @hyperjump/json-schema-coverage! It improves on what we created here, updates to use istanbul to provide familiar coverage reports, and integrates with Vitest so it's easy to use. Have a look! I used the OpenAPI tests to scare out the bugs. I'll be sending a PR soon.