Skip to content

Commit f67129e

Browse files
author
Simon Massey
authored
Convert JSON Test Suite from blocking unit test to standalone reporting tool (#17)
* compat test but got failures * Refactor JSON Test Suite to standalone reporting tool with robust encoding - Convert JsonTestSuiteSummary from blocking unit test to executable tool - Add Maven wrapper and exec plugin configuration - Disable unit test with @disabled to prevent build failures - Implement RobustCharDecoder for comprehensive encoding detection (BOM, UTF-16/32, permissive UTF-8) - Add security documentation warning about undocumented StackOverflowError vulnerabilities - Achieve 0 files skipped, 99.3% conformance across 318 test files - Support both human-readable and JSON output formats Resolves #16
1 parent bf963a4 commit f67129e

File tree

10 files changed

+1122
-1
lines changed

10 files changed

+1122
-1
lines changed
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
wrapperVersion=3.3.2
18+
distributionType=only-script
19+
distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.9/apache-maven-3.9.9-bin.zip

README.md

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,15 @@ This is a simplified backport with the following changes from the original:
2929
- Removed value-based class annotations.
3030
- Compatible with JDK 21.
3131

32+
## Security Considerations
33+
34+
**⚠️ This unstable API contains undocumented security vulnerabilities.** The compatibility test suite (documented below) includes crafted attack vectors that expose these issues:
35+
36+
- **Stack exhaustion attacks**: Deeply nested JSON structures can trigger `StackOverflowError`, potentially leaving applications in an undefined state and enabling denial-of-service attacks
37+
- **API contract violations**: The `Json.parse()` method documentation only declares `JsonParseException` and `NullPointerException`, but malicious inputs can trigger undeclared exceptions
38+
39+
These vulnerabilities exist in the upstream OpenJDK sandbox implementation and are reported here for transparency.
40+
3241
## Building
3342

3443
Requires JDK 21 or later. Build with Maven:
@@ -220,3 +229,40 @@ String formatted = Json.toDisplayString(data, 2);
220229
// ]
221230
// }
222231
```
232+
233+
## JSON Test Suite Compatibility
234+
235+
This backport includes a compatibility report tool that tests against the [JSON Test Suite](https://github.com/nst/JSONTestSuite) to track conformance with JSON standards.
236+
237+
### Running the Compatibility Report
238+
239+
First, build the project and download the test suite:
240+
241+
```bash
242+
# Build project and download test suite
243+
./mvnw clean compile generate-test-resources -pl json-compatibility-suite
244+
245+
# Run human-readable report
246+
./mvnw exec:java -pl json-compatibility-suite
247+
248+
# Run JSON output (dogfoods the API)
249+
./mvnw exec:java -pl json-compatibility-suite -Dexec.args="--json"
250+
```
251+
252+
### Current Status
253+
254+
The implementation achieves **99.3% overall conformance** with the JSON Test Suite:
255+
256+
- **Valid JSON**: 97.9% success rate (93/95 files pass)
257+
- **Invalid JSON**: 100% success rate (correctly rejects all invalid JSON)
258+
- **Implementation-defined**: Handles 35 edge cases per implementation choice (27 accepted, 8 rejected)
259+
260+
The 2 failing cases involve duplicate object keys, which this implementation rejects (stricter than required by the JSON specification). This is an implementation choice that prioritizes data integrity over permissiveness.
261+
262+
### Understanding the Results
263+
264+
- **Files skipped**: Currently 0 files skipped due to robust encoding detection that handles various character encodings
265+
- **StackOverflowError**: Security vulnerability exposed by malicious deeply nested structures - can leave applications in undefined state
266+
- **Duplicate keys**: Implementation choice to reject for data integrity (2 files fail for this reason)
267+
268+
This tool reports status rather than making API design decisions, aligning with the project's goal of tracking upstream development without advocacy.

json-compatibility-suite/pom.xml

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0"
3+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
4+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
5+
<modelVersion>4.0.0</modelVersion>
6+
7+
<parent>
8+
<groupId>io.github.simbo1905.json</groupId>
9+
<artifactId>json-java21-parent</artifactId>
10+
<version>0.1-SNAPSHOT</version>
11+
</parent>
12+
13+
<artifactId>json-compatibility-suite</artifactId>
14+
<packaging>jar</packaging>
15+
16+
<name>JSON Compatibility Suite</name>
17+
18+
<dependencies>
19+
<dependency>
20+
<groupId>io.github.simbo1905.json</groupId>
21+
<artifactId>json-java21</artifactId>
22+
<version>${project.version}</version>
23+
</dependency>
24+
<!-- JUnit 5 for testing -->
25+
<dependency>
26+
<groupId>org.junit.jupiter</groupId>
27+
<artifactId>junit-jupiter-api</artifactId>
28+
<scope>test</scope>
29+
</dependency>
30+
<dependency>
31+
<groupId>org.junit.jupiter</groupId>
32+
<artifactId>junit-jupiter-engine</artifactId>
33+
<scope>test</scope>
34+
</dependency>
35+
<dependency>
36+
<groupId>org.assertj</groupId>
37+
<artifactId>assertj-core</artifactId>
38+
<scope>test</scope>
39+
</dependency>
40+
</dependencies>
41+
42+
<build>
43+
<plugins>
44+
<plugin>
45+
<groupId>com.googlecode.maven-download-plugin</groupId>
46+
<artifactId>download-maven-plugin</artifactId>
47+
<executions>
48+
<execution>
49+
<id>download-json-test-suite</id>
50+
<phase>generate-test-resources</phase>
51+
<goals>
52+
<goal>wget</goal>
53+
</goals>
54+
<configuration>
55+
<url>https://github.com/nst/JSONTestSuite/archive/refs/heads/master.zip</url>
56+
<outputDirectory>${project.build.directory}/test-resources</outputDirectory>
57+
<outputFileName>json-test-suite.zip</outputFileName>
58+
<unpack>true</unpack>
59+
</configuration>
60+
</execution>
61+
</executions>
62+
</plugin>
63+
<plugin>
64+
<groupId>org.codehaus.mojo</groupId>
65+
<artifactId>exec-maven-plugin</artifactId>
66+
<version>3.4.1</version>
67+
<configuration>
68+
<mainClass>jdk.sandbox.compatibility.JsonTestSuiteSummary</mainClass>
69+
<includePluginDependencies>false</includePluginDependencies>
70+
</configuration>
71+
</plugin>
72+
</plugins>
73+
</build>
74+
</project>
Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
package jdk.sandbox.compatibility;
2+
3+
import jdk.sandbox.java.util.json.Json;
4+
import jdk.sandbox.java.util.json.JsonArray;
5+
import jdk.sandbox.java.util.json.JsonObject;
6+
import jdk.sandbox.java.util.json.JsonString;
7+
import jdk.sandbox.java.util.json.JsonNumber;
8+
import jdk.sandbox.java.util.json.JsonParseException;
9+
10+
import java.nio.charset.MalformedInputException;
11+
import java.nio.charset.StandardCharsets;
12+
import java.nio.file.Files;
13+
import java.nio.file.Path;
14+
import java.nio.file.Paths;
15+
import java.util.ArrayList;
16+
import java.util.List;
17+
import java.util.logging.Logger;
18+
19+
/**
20+
* Generates a conformance summary report.
21+
* Run with: mvn exec:java -pl json-compatibility-suite
22+
*/
23+
public class JsonTestSuiteSummary {
24+
25+
private static final Logger LOGGER = Logger.getLogger(JsonTestSuiteSummary.class.getName());
26+
private static final Path TEST_DIR = Paths.get("json-compatibility-suite/target/test-resources/JSONTestSuite-master/test_parsing");
27+
28+
public static void main(String[] args) throws Exception {
29+
boolean jsonOutput = args.length > 0 && "--json".equals(args[0]);
30+
JsonTestSuiteSummary summary = new JsonTestSuiteSummary();
31+
if (jsonOutput) {
32+
summary.generateJsonReport();
33+
} else {
34+
summary.generateConformanceReport();
35+
}
36+
}
37+
38+
void generateConformanceReport() throws Exception {
39+
TestResults results = runTests();
40+
41+
System.out.println("\n=== JSON Test Suite Conformance Report ===");
42+
System.out.println("Repository: java.util.json backport");
43+
System.out.printf("Test files analyzed: %d%n", results.totalFiles);
44+
System.out.printf("Files skipped (could not read): %d%n%n", results.skippedFiles);
45+
46+
System.out.println("Valid JSON (y_ files):");
47+
System.out.printf(" Passed: %d%n", results.yPass);
48+
System.out.printf(" Failed: %d%n", results.yFail);
49+
System.out.printf(" Success rate: %.1f%%%n%n", 100.0 * results.yPass / (results.yPass + results.yFail));
50+
51+
System.out.println("Invalid JSON (n_ files):");
52+
System.out.printf(" Correctly rejected: %d%n", results.nPass);
53+
System.out.printf(" Incorrectly accepted: %d%n", results.nFail);
54+
System.out.printf(" Success rate: %.1f%%%n%n", 100.0 * results.nPass / (results.nPass + results.nFail));
55+
56+
System.out.println("Implementation-defined (i_ files):");
57+
System.out.printf(" Accepted: %d%n", results.iAccept);
58+
System.out.printf(" Rejected: %d%n%n", results.iReject);
59+
60+
double conformance = 100.0 * (results.yPass + results.nPass) / (results.yPass + results.yFail + results.nPass + results.nFail);
61+
System.out.printf("Overall Conformance: %.1f%%%n", conformance);
62+
63+
if (!results.shouldPassButFailed.isEmpty()) {
64+
System.out.println("\n⚠️ Valid JSON that failed to parse:");
65+
results.shouldPassButFailed.forEach(f -> System.out.println(" - " + f));
66+
}
67+
68+
if (!results.shouldFailButPassed.isEmpty()) {
69+
System.out.println("\n⚠️ Invalid JSON that was incorrectly accepted:");
70+
results.shouldFailButPassed.forEach(f -> System.out.println(" - " + f));
71+
}
72+
73+
if (results.shouldPassButFailed.isEmpty() && results.shouldFailButPassed.isEmpty()) {
74+
System.out.println("\n✅ Perfect conformance!");
75+
}
76+
}
77+
78+
void generateJsonReport() throws Exception {
79+
TestResults results = runTests();
80+
JsonObject report = createJsonReport(results);
81+
System.out.println(Json.toDisplayString(report, 2));
82+
}
83+
84+
private TestResults runTests() throws Exception {
85+
if (!Files.exists(TEST_DIR)) {
86+
throw new RuntimeException("Test suite not downloaded. Run: ./mvnw clean compile generate-test-resources -pl json-compatibility-suite");
87+
}
88+
89+
List<String> shouldPassButFailed = new ArrayList<>();
90+
List<String> shouldFailButPassed = new ArrayList<>();
91+
List<String> skippedFiles = new ArrayList<>();
92+
93+
int yPass = 0, yFail = 0;
94+
int nPass = 0, nFail = 0;
95+
int iAccept = 0, iReject = 0;
96+
97+
var files = Files.walk(TEST_DIR)
98+
.filter(p -> p.toString().endsWith(".json"))
99+
.sorted()
100+
.toList();
101+
102+
for (Path file : files) {
103+
String filename = file.getFileName().toString();
104+
String content = null;
105+
char[] charContent = null;
106+
107+
try {
108+
content = Files.readString(file, StandardCharsets.UTF_8);
109+
charContent = content.toCharArray();
110+
} catch (MalformedInputException e) {
111+
LOGGER.warning("UTF-8 failed for " + filename + ", using robust encoding detection");
112+
try {
113+
byte[] rawBytes = Files.readAllBytes(file);
114+
charContent = RobustCharDecoder.decodeToChars(rawBytes, filename);
115+
} catch (Exception ex) {
116+
throw new RuntimeException("Failed to read test file " + filename + " - this is a fundamental I/O failure, not an encoding issue: " + ex.getMessage(), ex);
117+
}
118+
}
119+
120+
// Test with char[] API (always available)
121+
boolean parseSucceeded = false;
122+
try {
123+
Json.parse(charContent);
124+
parseSucceeded = true;
125+
} catch (JsonParseException e) {
126+
parseSucceeded = false;
127+
} catch (StackOverflowError e) {
128+
LOGGER.warning("StackOverflowError on file: " + filename);
129+
parseSucceeded = false; // Treat as parse failure
130+
}
131+
132+
// Update counters based on results
133+
if (parseSucceeded) {
134+
if (filename.startsWith("y_")) {
135+
yPass++;
136+
} else if (filename.startsWith("n_")) {
137+
nFail++;
138+
shouldFailButPassed.add(filename);
139+
} else if (filename.startsWith("i_")) {
140+
iAccept++;
141+
}
142+
} else {
143+
if (filename.startsWith("y_")) {
144+
yFail++;
145+
shouldPassButFailed.add(filename);
146+
} else if (filename.startsWith("n_")) {
147+
nPass++;
148+
} else if (filename.startsWith("i_")) {
149+
iReject++;
150+
}
151+
}
152+
}
153+
154+
return new TestResults(files.size(), skippedFiles.size(),
155+
yPass, yFail, nPass, nFail, iAccept, iReject,
156+
shouldPassButFailed, shouldFailButPassed, skippedFiles);
157+
}
158+
159+
private JsonObject createJsonReport(TestResults results) {
160+
double ySuccessRate = 100.0 * results.yPass / (results.yPass + results.yFail);
161+
double nSuccessRate = 100.0 * results.nPass / (results.nPass + results.nFail);
162+
double conformance = 100.0 * (results.yPass + results.nPass) / (results.yPass + results.yFail + results.nPass + results.nFail);
163+
164+
return JsonObject.of(java.util.Map.of(
165+
"repository", JsonString.of("java.util.json backport"),
166+
"filesAnalyzed", JsonNumber.of(results.totalFiles),
167+
"filesSkipped", JsonNumber.of(results.skippedFiles),
168+
"validJson", JsonObject.of(java.util.Map.of(
169+
"passed", JsonNumber.of(results.yPass),
170+
"failed", JsonNumber.of(results.yFail),
171+
"successRate", JsonNumber.of(Math.round(ySuccessRate * 10) / 10.0)
172+
)),
173+
"invalidJson", JsonObject.of(java.util.Map.of(
174+
"correctlyRejected", JsonNumber.of(results.nPass),
175+
"incorrectlyAccepted", JsonNumber.of(results.nFail),
176+
"successRate", JsonNumber.of(Math.round(nSuccessRate * 10) / 10.0)
177+
)),
178+
"implementationDefined", JsonObject.of(java.util.Map.of(
179+
"accepted", JsonNumber.of(results.iAccept),
180+
"rejected", JsonNumber.of(results.iReject)
181+
)),
182+
"overallConformance", JsonNumber.of(Math.round(conformance * 10) / 10.0),
183+
"shouldPassButFailed", JsonArray.of(results.shouldPassButFailed.stream()
184+
.map(JsonString::of)
185+
.toList()),
186+
"shouldFailButPassed", JsonArray.of(results.shouldFailButPassed.stream()
187+
.map(JsonString::of)
188+
.toList())
189+
));
190+
}
191+
192+
private record TestResults(
193+
int totalFiles, int skippedFiles,
194+
int yPass, int yFail, int nPass, int nFail, int iAccept, int iReject,
195+
List<String> shouldPassButFailed, List<String> shouldFailButPassed, List<String> skippedFiles2
196+
) {}
197+
}

0 commit comments

Comments
 (0)