Skip to content

Commit 2b5bdfe

Browse files
committed
docs: update compatibility claims with measured metrics from JsonSchemaCheckIT
- Replace estimated 71% compatibility with actual measured 63.3% (1,153 of 1,822 tests) - Add comprehensive metrics reporting documentation - Document test coverage: 420 groups, 1,657 validations, 576 skips categorized - Add usage examples for JSON/CSV metrics export - Clarify distinction between lenient and strict mode results - Provide defensible statistics based on actual test suite measurements The documentation now reflects the accurate, measured compatibility statistics provided by the new metrics system rather than estimates.
1 parent 86b73bc commit 2b5bdfe

File tree

3 files changed

+51
-2
lines changed

3 files changed

+51
-2
lines changed

README.md

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,27 @@ var result = schema.validate(
104104
// result.valid() => true
105105
```
106106

107-
Compatibility: runs the official 2020‑12 JSON Schema Test Suite on `verify`; in strict mode it currently passes about 71% of applicable cases.
107+
Compatibility: runs the official 2020‑12 JSON Schema Test Suite on `verify`; **measured compatibility is 63.3%** (1,153 of 1,822 tests pass) with comprehensive metrics reporting.
108+
109+
### JSON Schema Test Suite Metrics
110+
111+
The validator now provides defensible compatibility statistics:
112+
113+
```bash
114+
# Run with console metrics (default)
115+
mvn verify -pl json-java21-schema
116+
117+
# Export detailed JSON metrics
118+
mvn verify -pl json-java21-schema -Djson.schema.metrics=json
119+
120+
# Export CSV metrics for analysis
121+
mvn verify -pl json-java21-schema -Djson.schema.metrics=csv
122+
```
123+
124+
**Current measured compatibility**:
125+
- **Overall**: 63.3% (1,153 of 1,822 tests pass)
126+
- **Test coverage**: 420 test groups, 1,657 validation attempts
127+
- **Skip breakdown**: 70 unsupported schema groups, 2 test exceptions, 504 lenient mismatches
108128

109129
## Building
110130

json-java21-schema/AGENTS.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,32 @@ The project uses `java.util.logging` with levels:
4747
- **JSON Schema Test Suite**: Official tests from json-schema-org
4848
- **Real-world schemas**: Complex nested validation scenarios
4949
- **Performance tests**: Large schema compilation
50+
- **Metrics reporting**: Comprehensive compatibility statistics with detailed skip categorization
51+
52+
### JSON Schema Test Suite Metrics
53+
54+
The integration test now provides defensible compatibility metrics:
55+
56+
```bash
57+
# Run with console metrics (default)
58+
mvnd verify -pl json-java21-schema
59+
60+
# Export detailed JSON metrics
61+
mvnd verify -pl json-java21-schema -Djson.schema.metrics=json
62+
63+
# Export CSV metrics for analysis
64+
mvnd verify -pl json-java21-schema -Djson.schema.metrics=csv
65+
```
66+
67+
**Current measured compatibility** (as of implementation):
68+
- **Overall**: 63.3% (1,153 of 1,822 tests pass)
69+
- **Test coverage**: 420 test groups, 1,657 validation attempts
70+
- **Skip breakdown**: 70 unsupported schema groups, 2 test exceptions, 504 lenient mismatches
71+
72+
The metrics distinguish between:
73+
- **unsupportedSchemaGroup**: Whole groups skipped due to unsupported features (e.g., $ref, anchors)
74+
- **testException**: Individual tests that threw exceptions during validation
75+
- **lenientMismatch**: Expected≠actual results in lenient mode (counted as failures in strict mode)
5076

5177
#### Annotation Tests (`JsonSchemaAnnotationsTest.java`)
5278
- **Annotation processing**: Compile-time schema generation

json-java21-schema/README.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,10 @@ Compatibility and verify
2222

2323
- The module runs the official JSON Schema Test Suite during Maven verify.
2424
- Default mode is lenient: unsupported groups/tests are skipped to avoid build breaks while still logging.
25-
- Strict mode: enable with -Djson.schema.strict=true to enforce full assertions. In strict mode it currently passes about 71% of applicable cases.
25+
- Strict mode: enable with -Djson.schema.strict=true to enforce full assertions.
26+
- **Measured compatibility**: 63.3% (1,153 of 1,822 tests pass in lenient mode)
27+
- **Test coverage**: 420 test groups, 1,657 validation attempts, 70 unsupported schema groups, 2 test exceptions
28+
- Detailed metrics available via `-Djson.schema.metrics=json|csv`
2629

2730
How to run
2831

0 commit comments

Comments
 (0)