JSON Schema: strict coverage headline + docs; single fetch path, compile-session, and logging discipline

.github/workflows/ci.yml

@@ -13,13 +13,36 @@ jobs:
       - name: Checkout
         uses: actions/checkout@v4
 
-      - name: Set up JDK 24
+      - name: Set up JDK 21
         uses: actions/setup-java@v4
         with:
           distribution: temurin
-          java-version: '24'
+          java-version: '21'
           cache: 'maven'
 
       - name: Build and verify
         run: mvn -B -DskipITs=false -DskipTests=false verify
 
+      - name: Assert test count (no tests silently skipped)
+        run: |
+          python3 - <<'PY'
+          import os, xml.etree.ElementTree as ET, sys
+          totals={'tests':0,'failures':0,'errors':0,'skipped':0}
+          for dirpath,_,files in os.walk('.'):
+              if 'target' not in dirpath: continue
+              if 'surefire-reports' not in dirpath and 'failsafe-reports' not in dirpath: continue
+              for fn in files:
+                  if not fn.endswith('.xml'): continue
+                  p=os.path.join(dirpath,fn)
+                  try:
+                      r=ET.parse(p).getroot()
+                      for k in totals: totals[k]+=int(r.get(k,'0'))
+                  except Exception:
+                      pass
+          exp_tests=1908
+          exp_skipped=713
+          if totals['tests']!=exp_tests or totals['skipped']!=exp_skipped:
+              print(f"Unexpected test totals: {totals} != expected tests={exp_tests}, skipped={exp_skipped}")
+              sys.exit(1)
+          print(f"OK totals: {totals}")
+          PY

README.md

@@ -104,7 +104,28 @@ var result = schema.validate(
 // result.valid() => true
 ```
 
-Compatibility: runs the official 2020‑12 JSON Schema Test Suite on `verify`; in strict mode it currently passes about 71% of applicable cases.
+Compatibility: runs the official 2020‑12 JSON Schema Test Suite on `verify`; **strict compatibility is 61.6%** (1024 of 1,663 validations). [Overall including all discovered tests: 56.2% (1024 of 1,822)].
+
+### JSON Schema Test Suite Metrics
+
+The validator now provides defensible compatibility statistics:
+
+```bash
+# Run with console metrics (default)
+./mvn-test-no-boilerplate.sh -pl json-java21-schema
+
+# Export detailed JSON metrics
+./mvn-test-no-boilerplate.sh -pl json-java21-schema -Djson.schema.metrics=json
+
+# Export CSV metrics for analysis
+./mvn-test-no-boilerplate.sh -pl json-java21-schema -Djson.schema.metrics=csv
+```
+
+**Current measured compatibility**:
+- **Strict (headline)**: 61.6% (1024 of 1,663 validations)
+- **Overall (incl. out‑of‑scope)**: 56.2% (1024 of 1,822 discovered tests)
+- **Test coverage**: 420 test groups, 1,663 validation attempts
+- **Skip breakdown**: 65 unsupported schema groups, 0 test exceptions, 647 lenient mismatches
 
 ## Building
 

json-java21-schema/README.md

@@ -2,8 +2,9 @@
 
 Stack-based JSON Schema validator using sealed interface pattern with inner record types.
 
-- Draft 2020-12 subset: object/array/string/number/boolean/null, allOf/anyOf/not, if/then/else, const, $defs and local $ref (including root "#")
+- Draft 2020-12 subset: object/array/string/number/boolean/null, allOf/anyOf/not, if/then/else, const, format (11 validators), $defs and local $ref (including root "#")
 - Thread-safe compiled schemas; immutable results with error paths/messages
+- **Novel Architecture**: This module uses an innovative immutable "compile many documents (possibly just one) into an immutable set of roots using a work stack" compile-time architecture for high-performance schema compilation and validation. See `AGENTS.md` for detailed design documentation.
 
 Quick usage
 
@@ -22,18 +23,29 @@ Compatibility and verify
 
 - The module runs the official JSON Schema Test Suite during Maven verify.
 - Default mode is lenient: unsupported groups/tests are skipped to avoid build breaks while still logging.
-- Strict mode: enable with -Djson.schema.strict=true to enforce full assertions. In strict mode it currently passes about 71% of applicable cases.
+- Strict mode: enable with `-Djson.schema.strict=true` to enforce full assertions.
+- Measured compatibility (headline strictness): 61.6% (1024 of 1,663 validations)
+  - Overall including all discovered tests: 56.2% (1024 of 1,822)
+- Test coverage: 420 test groups, 1,663 validation attempts, 65 unsupported schema groups, 0 test exceptions, 647 lenient mismatches
+- Detailed metrics available via `-Djson.schema.metrics=json|csv`
 
 How to run
 
 ```bash
 # Run unit + integration tests (includes official suite)
-mvn -pl json-java21-schema -am verify
+./mvn-test-no-boilerplate.sh -pl json-java21-schema
 
 # Strict mode
-mvn -Djson.schema.strict=true -pl json-java21-schema -am verify
+./mvn-test-no-boilerplate.sh -pl json-java21-schema -Djson.schema.strict=true
 ```
 
+OpenRPC validation
+
+- Additional integration test validates OpenRPC documents using a minimal, self‑contained schema:
+  - Test: `src/test/java/io/github/simbo1905/json/schema/OpenRPCSchemaValidationIT.java`
+  - Resources: `src/test/resources/openrpc/` (schema and examples)
+  - Thanks to OpenRPC meta-schema and examples (Apache-2.0): https://github.com/open-rpc/meta-schema and https://github.com/open-rpc/examples
+
 ## API Design
 
 Single public interface with all schema types as inner records:
@@ -145,3 +157,30 @@ if (!result.valid()) {
     }
 }
 ```
+
+### Format Validation
+
+The validator supports JSON Schema 2020-12 format validation with opt-in assertion mode:
+
+- **Built-in formats**: uuid, email, ipv4, ipv6, uri, uri-reference, hostname, date, time, date-time, regex
+- **Annotation by default**: Format validation is annotation-only (always passes) unless format assertion is enabled
+- **Opt-in assertion**: Enable format validation via:
+  - `JsonSchema.Options(true)` parameter in `compile()`
+  - System property: `-Djsonschema.format.assertion=true`
+  - Root schema flag: `"formatAssertion": true`
+- **Unknown formats**: Gracefully handled with logged warnings (no validation errors)
+
+```java
+// Format validation disabled (default) - always passes
+var schema = JsonSchema.compile(Json.parse("""
+  {"type": "string", "format": "email"}
+"""));
+schema.validate(Json.parse("\"invalid-email\"")); // passes
+
+// Format validation enabled - validates format
+var schema = JsonSchema.compile(Json.parse("""
+  {"type": "string", "format": "email"}
+"""), new JsonSchema.Options(true));
+schema.validate(Json.parse("\"invalid-email\"")); // fails
+schema.validate(Json.parse("\"[email protected]\"")); // passes
+```

json-java21-schema/mvn-test-no-boilerplate.sh

@@ -0,0 +1,71 @@
+#!/bin/bash
+
+# Strip Maven test boilerplate - show compile errors and test results only
+# Usage: ./mvn-test-no-boilerplate.sh [maven test arguments]
+# 
+# Examples:
+#   ./mvn-test-no-boilerplate.sh -Dtest=RefactorTests
+#   ./mvn-test-no-boilerplate.sh -Dtest=RefactorTests#testList -Djava.util.logging.ConsoleHandler.level=INFO
+#   ./mvn-test-no-boilerplate.sh -Dtest=RefactorTests#testList -Djava.util.logging.ConsoleHandler.level=FINER
+#
+# For running tests in a specific module:
+#   ./mvn-test-no-boilerplate.sh -pl json-java21-api-tracker -Dtest=CompilerApiLearningTest
+#
+# The script automatically detects if mvnd is available, otherwise falls back to mvn
+
+# Detect if mvnd is available, otherwise use mvn
+if command -v mvnd &> /dev/null; then
+    MVN_CMD="mvnd"
+else
+    MVN_CMD="mvn"
+fi
+
+timeout 120 $MVN_CMD test "$@" 2>&1 | awk '
+BEGIN { 
+    scanning_started = 0
+    compilation_section = 0
+    test_section = 0
+}
+
+# Skip all WARNING lines before project scanning starts
+/INFO.*Scanning for projects/ { 
+    scanning_started = 1 
+    print
+    next
+}
+
+# Before scanning starts, skip WARNING lines
+!scanning_started && /^WARNING:/ { next }
+
+# Show compilation errors
+/COMPILATION ERROR/ { compilation_section = 1 }
+/BUILD FAILURE/ && compilation_section { compilation_section = 0 }
+
+# Show test section
+/INFO.*T E S T S/ { 
+    test_section = 1
+    print "-------------------------------------------------------"
+    print " T E S T S"
+    print "-------------------------------------------------------"
+    next
+}
+
+# In compilation error section, show everything
+compilation_section { print }
+
+# In test section, show everything - let user control logging with -D arguments
+test_section {
+    print
+}
+
+# Before test section starts, show important lines only
+!test_section && scanning_started {
+    if (/INFO.*Scanning|INFO.*Building|INFO.*resources|INFO.*compiler|INFO.*surefire|ERROR|FAILURE/) {
+        print
+    }
+    # Show compilation warnings/errors
+    if (/WARNING.*COMPILATION|ERROR.*/) {
+        print
+    }
+}
+'