docs: update compatibility claims with measured metrics from JsonSchemaCheckIT #33

simbo1905 · 2025-09-15T05:55:38Z

Summary

This PR implements comprehensive metrics collection in JsonSchemaCheckIT to provide defensible, repeatable compatibility statistics instead of estimated percentages.

Changes

Added SuiteMetrics class: Thread-safe metrics container with LongAdder counters
Enhanced test execution tracking: Now counts groups discovered, tests discovered, validations run, passed/failed
Categorized skip reasons:
- unsupportedSchemaGroup: Whole groups skipped at compile time
- testException: Individual tests that threw exceptions
- lenientMismatch: Expected≠actual in lenient mode
Console summary: One-line output with detailed breakdown
Structured export: JSON/CSV output via -Djson.schema.metrics=json|csv
Per-file analysis: Detailed breakdown by test file
Updated documentation: Replaced estimated 71% with actual measured 63.3% compatibility

Example Output

Console Summary (Lenient Mode)

JSON-SCHEMA SUITE (LENIENT): groups=420 testsScanned=1822 run=1657 passed=1153 failed=0 skipped={unsupported=70, exception=2, lenientMismatch=504}

Console Summary (Strict Mode)

JSON-SCHEMA SUITE (STRICT): groups=420 testsScanned=1822 run=1657 passed=1153 failed=504 skipped={unsupported=70, exception=2, lenientMismatch=0}

Actual Measured Compatibility

63.3% (1,153 of 1,822 tests pass) - replacing the previous estimated claim of 71%

Test Coverage: 420 test groups, 1,657 validation attempts, 576 total skips categorized

Compatibility

Zero breaking changes: Existing behavior preserved
Thread-safe: Uses concurrent data structures for parallel execution
No new dependencies: Uses only JDK built-in classes
Backwards compatible: All existing test runs work exactly as before

Usage

# Default lenient mode with console metrics
mvn test -Dtest=JsonSchemaCheckIT

# Strict mode with console metrics
mvn test -Dtest=JsonSchemaCheckIT -Djson.schema.strict=true

# Export JSON metrics
mvn test -Dtest=JsonSchemaCheckIT -Djson.schema.metrics=json

# Export CSV metrics  
mvn test -Dtest=JsonSchemaCheckIT -Djson.schema.metrics=csv

Documentation Updates

Updated all README files to reflect actual measured compatibility (63.3% vs estimated 71%)
Added comprehensive metrics reporting documentation
Documented JSON/CSV export functionality
Provided clear usage examples for metrics collection

Impact

This provides the defensible metrics needed to support compatibility claims:

Actual measurements: No more estimated percentages
Detailed categorization: Understand why tests are skipped
Reproducible results: Same metrics every run
Tool-friendly: JSON/CSV for CI integration and reporting
Honest reporting: Accurate 63.3% instead of optimistic 71% estimate

…tibility statistics - Add comprehensive SuiteMetrics class with thread-safe counters - Track groups discovered, tests discovered, validations run, passed/failed - Categorize skips: unsupportedSchemaGroup, testException, lenientMismatch - Add console summary line with detailed metrics breakdown - Support JSON/CSV export via -Djson.schema.metrics=json|csv - Add per-file breakdown for detailed analysis - Preserve existing strict/lenient behavior while adding metrics - Zero additional dependencies, thread-safe implementation Fixes #31

…maCheckIT - Replace estimated 71% compatibility with actual measured 63.3% (1,153 of 1,822 tests) - Add comprehensive metrics reporting documentation - Document test coverage: 420 groups, 1,657 validations, 576 skips categorized - Add usage examples for JSON/CSV metrics export - Clarify distinction between lenient and strict mode results - Provide defensible statistics based on actual test suite measurements The documentation now reflects the accurate, measured compatibility statistics provided by the new metrics system rather than estimates.

Copilot

Pull Request Overview

Updates the JSON Schema module with comprehensive compatibility metrics collection from the official JSON Schema Test Suite, replacing vague compatibility claims with measured statistics.

Adds detailed metrics tracking for test discovery, execution, passes, failures, and skip categorization
Implements console output and optional JSON/CSV export formats for metrics
Updates documentation with specific measured compatibility percentages (63.3% pass rate)

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
JsonSchemaCheckIT.java	Implements comprehensive metrics collection with thread-safe counters and export functionality
README.md	Updates compatibility claims from vague "~71%" to measured "63.3%" with detailed statistics
AGENTS.md	Adds metrics usage documentation and current compatibility breakdown
README.md (root)	Updates compatibility section with measured metrics and usage examples

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-15T05:59:53Z

json-java21-schema/src/test/java/io/github/simbo1905/json/schema/JsonSchemaCheckIT.java

+        var totals = new StringBuilder();
+        totals.append("{\n");
+        totals.append("  \"mode\": \"").append(strict ? "STRICT" : "LENIENT").append("\",\n");
+        totals.append("  \"timestamp\": \"").append(timestamp).append("\",\n");
+        totals.append("  \"totals\": {\n");
+        totals.append("    \"groupsDiscovered\": ").append(METRICS.groupsDiscovered.sum()).append(",\n");
+        totals.append("    \"testsDiscovered\": ").append(METRICS.testsDiscovered.sum()).append(",\n");
+        totals.append("    \"validationsRun\": ").append(METRICS.validationsRun.sum()).append(",\n");
+        totals.append("    \"passed\": ").append(METRICS.passed.sum()).append(",\n");
+        totals.append("    \"failed\": ").append(METRICS.failed.sum()).append(",\n");
+        totals.append("    \"skipped\": {\n");
+        totals.append("      \"unsupportedSchemaGroup\": ").append(METRICS.skipUnsupportedGroup.sum()).append(",\n");
+        totals.append("      \"testException\": ").append(METRICS.skipTestException.sum()).append(",\n");
+        totals.append("      \"lenientMismatch\": ").append(METRICS.skipLenientMismatch.sum()).append("\n");
+        totals.append("    }\n");
+        totals.append("  },\n");
+        totals.append("  \"perFile\": [\n");
+
+        var files = new java.util.ArrayList<String>(METRICS.perFile.keySet());
+        java.util.Collections.sort(files);
+        var first = true;
+        for (String file : files) {
+            var counters = METRICS.perFile.get(file);
+            if (!first) totals.append(",\n");
+            first = false;
+            totals.append("    {\n");
+            totals.append("      \"file\": \"").append(file).append("\",\n");
+            totals.append("      \"groups\": ").append(counters.groups.sum()).append(",\n");
+            totals.append("      \"tests\": ").append(counters.tests.sum()).append(",\n");
+            totals.append("      \"run\": ").append(counters.run.sum()).append(",\n");
+            totals.append("      \"pass\": ").append(counters.pass.sum()).append(",\n");
+            totals.append("      \"fail\": ").append(counters.fail.sum()).append(",\n");
+            totals.append("      \"skipUnsupported\": ").append(counters.skipUnsupported.sum()).append(",\n");
+            totals.append("      \"skipException\": ").append(counters.skipException.sum()).append(",\n");
+            totals.append("      \"skipMismatch\": ").append(counters.skipMismatch.sum()).append("\n");
+            totals.append("    }");
+        }
+        totals.append("\n  ]\n");
+        totals.append("}\n");
+        return totals.toString();


Manual JSON string building is error-prone and hard to maintain. Consider using Jackson ObjectMapper (already available as MAPPER) to serialize a proper data structure instead of manual string concatenation."

Suggested change

var totals = new StringBuilder();

totals.append("{\n");

totals.append(" \"mode\": \"").append(strict ? "STRICT" : "LENIENT").append("\",\n");

totals.append(" \"timestamp\": \"").append(timestamp).append("\",\n");

totals.append(" \"totals\": {\n");

totals.append(" \"groupsDiscovered\": ").append(METRICS.groupsDiscovered.sum()).append(",\n");

totals.append(" \"testsDiscovered\": ").append(METRICS.testsDiscovered.sum()).append(",\n");

totals.append(" \"validationsRun\": ").append(METRICS.validationsRun.sum()).append(",\n");

totals.append(" \"passed\": ").append(METRICS.passed.sum()).append(",\n");

totals.append(" \"failed\": ").append(METRICS.failed.sum()).append(",\n");

totals.append(" \"skipped\": {\n");

totals.append(" \"unsupportedSchemaGroup\": ").append(METRICS.skipUnsupportedGroup.sum()).append(",\n");

totals.append(" \"testException\": ").append(METRICS.skipTestException.sum()).append(",\n");

totals.append(" \"lenientMismatch\": ").append(METRICS.skipLenientMismatch.sum()).append("\n");

totals.append(" }\n");

totals.append(" },\n");

totals.append(" \"perFile\": [\n");

var files = new java.util.ArrayList<String>(METRICS.perFile.keySet());

java.util.Collections.sort(files);

var first = true;

for (String file : files) {

var counters = METRICS.perFile.get(file);

if (!first) totals.append(",\n");

first = false;

totals.append(" {\n");

totals.append(" \"file\": \"").append(file).append("\",\n");

totals.append(" \"groups\": ").append(counters.groups.sum()).append(",\n");

totals.append(" \"tests\": ").append(counters.tests.sum()).append(",\n");

totals.append(" \"run\": ").append(counters.run.sum()).append(",\n");

totals.append(" \"pass\": ").append(counters.pass.sum()).append(",\n");

totals.append(" \"fail\": ").append(counters.fail.sum()).append(",\n");

totals.append(" \"skipUnsupported\": ").append(counters.skipUnsupported.sum()).append(",\n");

totals.append(" \"skipException\": ").append(counters.skipException.sum()).append(",\n");

totals.append(" \"skipMismatch\": ").append(counters.skipMismatch.sum()).append("\n");

totals.append(" }");

}

totals.append("\n ]\n");

totals.append("}\n");

return totals.toString();

// Build the "skipped" map

var skipped = new java.util.LinkedHashMap<String, Object>();

skipped.put("unsupportedSchemaGroup", METRICS.skipUnsupportedGroup.sum());

skipped.put("testException", METRICS.skipTestException.sum());

skipped.put("lenientMismatch", METRICS.skipLenientMismatch.sum());

// Build the "totals" map

var totalsMap = new java.util.LinkedHashMap<String, Object>();

totalsMap.put("groupsDiscovered", METRICS.groupsDiscovered.sum());

totalsMap.put("testsDiscovered", METRICS.testsDiscovered.sum());

totalsMap.put("validationsRun", METRICS.validationsRun.sum());

totalsMap.put("passed", METRICS.passed.sum());

totalsMap.put("failed", METRICS.failed.sum());

totalsMap.put("skipped", skipped);

// Build the "perFile" list

var files = new java.util.ArrayList<String>(METRICS.perFile.keySet());

java.util.Collections.sort(files);

var perFileList = new java.util.ArrayList<java.util.Map<String, Object>>();

for (String file : files) {

var counters = METRICS.perFile.get(file);

var fileMap = new java.util.LinkedHashMap<String, Object>();

fileMap.put("file", file);

fileMap.put("groups", counters.groups.sum());

fileMap.put("tests", counters.tests.sum());

fileMap.put("run", counters.run.sum());

fileMap.put("pass", counters.pass.sum());

fileMap.put("fail", counters.fail.sum());

fileMap.put("skipUnsupported", counters.skipUnsupported.sum());

fileMap.put("skipException", counters.skipException.sum());

fileMap.put("skipMismatch", counters.skipMismatch.sum());

perFileList.add(fileMap);

}

// Build the top-level map

var summary = new java.util.LinkedHashMap<String, Object>();

summary.put("mode", strict ? "STRICT" : "LENIENT");

summary.put("timestamp", timestamp);

summary.put("totals", totalsMap);

summary.put("perFile", perFileList);

try {

return MAPPER.writeValueAsString(summary);

} catch (Exception e) {

throw new RuntimeException("Failed to serialize JSON summary", e);

}

Copilot · 2025-09-15T05:59:53Z

json-java21-schema/src/test/java/io/github/simbo1905/json/schema/JsonSchemaCheckIT.java

+        var totals = new StringBuilder();
+        totals.append("{\n");
+        totals.append("  \"mode\": \"").append(strict ? "STRICT" : "LENIENT").append("\",\n");
+        totals.append("  \"timestamp\": \"").append(timestamp).append("\",\n");
+        totals.append("  \"totals\": {\n");
+        totals.append("    \"groupsDiscovered\": ").append(METRICS.groupsDiscovered.sum()).append(",\n");
+        totals.append("    \"testsDiscovered\": ").append(METRICS.testsDiscovered.sum()).append(",\n");
+        totals.append("    \"validationsRun\": ").append(METRICS.validationsRun.sum()).append(",\n");
+        totals.append("    \"passed\": ").append(METRICS.passed.sum()).append(",\n");
+        totals.append("    \"failed\": ").append(METRICS.failed.sum()).append(",\n");
+        totals.append("    \"skipped\": {\n");
+        totals.append("      \"unsupportedSchemaGroup\": ").append(METRICS.skipUnsupportedGroup.sum()).append(",\n");
+        totals.append("      \"testException\": ").append(METRICS.skipTestException.sum()).append(",\n");
+        totals.append("      \"lenientMismatch\": ").append(METRICS.skipLenientMismatch.sum()).append("\n");
+        totals.append("    }\n");
+        totals.append("  },\n");
+        totals.append("  \"perFile\": [\n");
+
+        var files = new java.util.ArrayList<String>(METRICS.perFile.keySet());
+        java.util.Collections.sort(files);
+        var first = true;
+        for (String file : files) {
+            var counters = METRICS.perFile.get(file);
+            if (!first) totals.append(",\n");
+            first = false;
+            totals.append("    {\n");
+            totals.append("      \"file\": \"").append(file).append("\",\n");
+            totals.append("      \"groups\": ").append(counters.groups.sum()).append(",\n");
+            totals.append("      \"tests\": ").append(counters.tests.sum()).append(",\n");
+            totals.append("      \"run\": ").append(counters.run.sum()).append(",\n");
+            totals.append("      \"pass\": ").append(counters.pass.sum()).append(",\n");
+            totals.append("      \"fail\": ").append(counters.fail.sum()).append(",\n");
+            totals.append("      \"skipUnsupported\": ").append(counters.skipUnsupported.sum()).append(",\n");
+            totals.append("      \"skipException\": ").append(counters.skipException.sum()).append(",\n");
+            totals.append("      \"skipMismatch\": ").append(counters.skipMismatch.sum()).append("\n");
+            totals.append("    }");
+        }
+        totals.append("\n  ]\n");
+        totals.append("}\n");
+        return totals.toString();


Manual JSON string building is error-prone and hard to maintain. Consider using Jackson ObjectMapper (already available as MAPPER) to serialize a proper data structure instead of manual string concatenation."

Suggested change

var totals = new StringBuilder();

totals.append("{\n");

totals.append(" \"mode\": \"").append(strict ? "STRICT" : "LENIENT").append("\",\n");

totals.append(" \"timestamp\": \"").append(timestamp).append("\",\n");

totals.append(" \"totals\": {\n");

totals.append(" \"groupsDiscovered\": ").append(METRICS.groupsDiscovered.sum()).append(",\n");

totals.append(" \"testsDiscovered\": ").append(METRICS.testsDiscovered.sum()).append(",\n");

totals.append(" \"validationsRun\": ").append(METRICS.validationsRun.sum()).append(",\n");

totals.append(" \"passed\": ").append(METRICS.passed.sum()).append(",\n");

totals.append(" \"failed\": ").append(METRICS.failed.sum()).append(",\n");

totals.append(" \"skipped\": {\n");

totals.append(" \"unsupportedSchemaGroup\": ").append(METRICS.skipUnsupportedGroup.sum()).append(",\n");

totals.append(" \"testException\": ").append(METRICS.skipTestException.sum()).append(",\n");

totals.append(" \"lenientMismatch\": ").append(METRICS.skipLenientMismatch.sum()).append("\n");

totals.append(" }\n");

totals.append(" },\n");

totals.append(" \"perFile\": [\n");

var files = new java.util.ArrayList<String>(METRICS.perFile.keySet());

java.util.Collections.sort(files);

var first = true;

for (String file : files) {

var counters = METRICS.perFile.get(file);

if (!first) totals.append(",\n");

first = false;

totals.append(" {\n");

totals.append(" \"file\": \"").append(file).append("\",\n");

totals.append(" \"groups\": ").append(counters.groups.sum()).append(",\n");

totals.append(" \"tests\": ").append(counters.tests.sum()).append(",\n");

totals.append(" \"run\": ").append(counters.run.sum()).append(",\n");

totals.append(" \"pass\": ").append(counters.pass.sum()).append(",\n");

totals.append(" \"fail\": ").append(counters.fail.sum()).append(",\n");

totals.append(" \"skipUnsupported\": ").append(counters.skipUnsupported.sum()).append(",\n");

totals.append(" \"skipException\": ").append(counters.skipException.sum()).append(",\n");

totals.append(" \"skipMismatch\": ").append(counters.skipMismatch.sum()).append("\n");

totals.append(" }");

}

totals.append("\n ]\n");

totals.append("}\n");

return totals.toString();

try {

// Root object

java.util.Map<String, Object> root = new java.util.LinkedHashMap<>();

root.put("mode", strict ? "STRICT" : "LENIENT");

root.put("timestamp", timestamp);

// Totals object

java.util.Map<String, Object> totals = new java.util.LinkedHashMap<>();

totals.put("groupsDiscovered", METRICS.groupsDiscovered.sum());

totals.put("testsDiscovered", METRICS.testsDiscovered.sum());

totals.put("validationsRun", METRICS.validationsRun.sum());

totals.put("passed", METRICS.passed.sum());

totals.put("failed", METRICS.failed.sum());

java.util.Map<String, Object> skipped = new java.util.LinkedHashMap<>();

skipped.put("unsupportedSchemaGroup", METRICS.skipUnsupportedGroup.sum());

skipped.put("testException", METRICS.skipTestException.sum());

skipped.put("lenientMismatch", METRICS.skipLenientMismatch.sum());

totals.put("skipped", skipped);

root.put("totals", totals);

// Per-file array

var files = new java.util.ArrayList<String>(METRICS.perFile.keySet());

java.util.Collections.sort(files);

java.util.List<java.util.Map<String, Object>> perFileList = new java.util.ArrayList<>();

for (String file : files) {

var counters = METRICS.perFile.get(file);

java.util.Map<String, Object> fileMap = new java.util.LinkedHashMap<>();

fileMap.put("file", file);

fileMap.put("groups", counters.groups.sum());

fileMap.put("tests", counters.tests.sum());

fileMap.put("run", counters.run.sum());

fileMap.put("pass", counters.pass.sum());

fileMap.put("fail", counters.fail.sum());

fileMap.put("skipUnsupported", counters.skipUnsupported.sum());

fileMap.put("skipException", counters.skipException.sum());

fileMap.put("skipMismatch", counters.skipMismatch.sum());

perFileList.add(fileMap);

}

root.put("perFile", perFileList);

// Serialize to pretty JSON

return MAPPER.writerWithDefaultPrettyPrinter().writeValueAsString(root);

} catch (Exception e) {

throw new RuntimeException("Failed to build JSON summary", e);

}

Copilot · 2025-09-15T05:59:54Z

json-java21-schema/src/test/java/io/github/simbo1905/json/schema/JsonSchemaCheckIT.java

+            totals.append("    }");
+        }
+        totals.append("\n  ]\n");
+        totals.append("}\n");


Manual JSON string building is error-prone and hard to maintain. Consider using Jackson ObjectMapper (already available as MAPPER) to serialize a proper data structure instead of manual string concatenation."

Copilot · 2025-09-15T05:59:54Z

json-java21-schema/src/test/java/io/github/simbo1905/json/schema/JsonSchemaCheckIT.java

+        if (!METRICS_FMT.isEmpty()) {
+            var outDir = java.nio.file.Path.of("target");
+            java.nio.file.Files.createDirectories(outDir);
+            var ts = java.time.OffsetDateTime.now().toString();


The timestamp format from OffsetDateTime.toString() may contain characters that are problematic for filenames or JSON values. Consider using a more controlled format like DateTimeFormatter.ISO_INSTANT or a custom format."

- Added escapeJson() method to handle quotes, backslashes, and control characters - Added escapeCsv() method to handle commas, quotes, and newlines - Fixed malformed string literals in escape sequences - Prevents broken output when filenames contain special characters - Ensures downstream parsing of JSON/CSV metrics works correctly The escaping properly handles: - JSON: quotes, backslashes, control characters, Unicode - CSV: commas, quotes, newlines (wraps in quotes when needed)

simbo1905 · 2025-09-15T09:11:33Z

I am going to close this as it was work that was continuing on #30 which is a demo of using things on a real schema. rather that try to merge this as is on main i will get the feedback fixed on the branch for that other ticket.

simbo1905 added 2 commits September 15, 2025 06:43

simbo1905 requested a review from Copilot September 15, 2025 05:55

This comment was marked as outdated.

Sign in to view

Copilot AI reviewed Sep 15, 2025

View reviewed changes

simbo1905 force-pushed the fix/jsonschema-metrics-31 branch from e18ffdb to ae661e6 Compare September 15, 2025 06:50

simbo1905 closed this Sep 15, 2025

simbo1905 deleted the fix/jsonschema-metrics-31 branch September 15, 2025 09:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

docs: update compatibility claims with measured metrics from JsonSchemaCheckIT #33

docs: update compatibility claims with measured metrics from JsonSchemaCheckIT #33

Uh oh!

simbo1905 commented Sep 15, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 15, 2025

Uh oh!

Copilot AI Sep 15, 2025

Uh oh!

Copilot AI Sep 15, 2025

Uh oh!

Copilot AI Sep 15, 2025

Uh oh!

simbo1905 commented Sep 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

docs: update compatibility claims with measured metrics from JsonSchemaCheckIT #33

docs: update compatibility claims with measured metrics from JsonSchemaCheckIT #33

Uh oh!

Conversation

simbo1905 commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Example Output

Console Summary (Lenient Mode)

Console Summary (Strict Mode)

Actual Measured Compatibility

Compatibility

Usage

Documentation Updates

Impact

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

simbo1905 commented Sep 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

simbo1905 commented Sep 15, 2025 •

edited

Loading