Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,27 @@ var result = schema.validate(
// result.valid() => true
```

Compatibility: runs the official 2020‑12 JSON Schema Test Suite on `verify`; in strict mode it currently passes about 71% of applicable cases.
Compatibility: runs the official 2020‑12 JSON Schema Test Suite on `verify`; **measured compatibility is 63.3%** (1,153 of 1,822 tests pass) with comprehensive metrics reporting.

### JSON Schema Test Suite Metrics

The validator now provides defensible compatibility statistics:

```bash
# Run with console metrics (default)
mvn verify -pl json-java21-schema

# Export detailed JSON metrics
mvn verify -pl json-java21-schema -Djson.schema.metrics=json

# Export CSV metrics for analysis
mvn verify -pl json-java21-schema -Djson.schema.metrics=csv
```

**Current measured compatibility**:
- **Overall**: 63.3% (1,153 of 1,822 tests pass)
- **Test coverage**: 420 test groups, 1,657 validation attempts
- **Skip breakdown**: 70 unsupported schema groups, 2 test exceptions, 504 lenient mismatches

## Building

Expand Down
26 changes: 26 additions & 0 deletions json-java21-schema/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,32 @@ The project uses `java.util.logging` with levels:
- **JSON Schema Test Suite**: Official tests from json-schema-org
- **Real-world schemas**: Complex nested validation scenarios
- **Performance tests**: Large schema compilation
- **Metrics reporting**: Comprehensive compatibility statistics with detailed skip categorization

### JSON Schema Test Suite Metrics

The integration test now provides defensible compatibility metrics:

```bash
# Run with console metrics (default)
mvnd verify -pl json-java21-schema

# Export detailed JSON metrics
mvnd verify -pl json-java21-schema -Djson.schema.metrics=json

# Export CSV metrics for analysis
mvnd verify -pl json-java21-schema -Djson.schema.metrics=csv
```

**Current measured compatibility** (as of implementation):
- **Overall**: 63.3% (1,153 of 1,822 tests pass)
- **Test coverage**: 420 test groups, 1,657 validation attempts
- **Skip breakdown**: 70 unsupported schema groups, 2 test exceptions, 504 lenient mismatches

The metrics distinguish between:
- **unsupportedSchemaGroup**: Whole groups skipped due to unsupported features (e.g., $ref, anchors)
- **testException**: Individual tests that threw exceptions during validation
- **lenientMismatch**: Expected≠actual results in lenient mode (counted as failures in strict mode)

#### Annotation Tests (`JsonSchemaAnnotationsTest.java`)
- **Annotation processing**: Compile-time schema generation
Expand Down
5 changes: 4 additions & 1 deletion json-java21-schema/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,10 @@ Compatibility and verify

- The module runs the official JSON Schema Test Suite during Maven verify.
- Default mode is lenient: unsupported groups/tests are skipped to avoid build breaks while still logging.
- Strict mode: enable with -Djson.schema.strict=true to enforce full assertions. In strict mode it currently passes about 71% of applicable cases.
- Strict mode: enable with -Djson.schema.strict=true to enforce full assertions.
- **Measured compatibility**: 63.3% (1,153 of 1,822 tests pass in lenient mode)
- **Test coverage**: 420 test groups, 1,657 validation attempts, 70 unsupported schema groups, 2 test exceptions
- Detailed metrics available via `-Djson.schema.metrics=json|csv`

How to run

Expand Down
Loading