-
Notifications
You must be signed in to change notification settings - Fork 189
Fix #1176 JSON Decoder and Encoder limit disagreement #1242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
@YOCKOW Can you please take a look? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems fine. The limit is somewhat arbitrary but it does make sense that we should be able to round trip to exactly the limit.
OpenStepPlist.swift also has a limit of 512 - does that one need to be updated as well? @kperryua
You can't round-trip an OpenStep plist, so that's less of a concern. To make the symmetry more obvious, would it make more sense to start the counting at the same value instead? This 513 value begs for a comment explaining the choice. |
@kperryua I considered about that. However, that would involve some much more significant code logic refactoring, rewriting most of JSONWriter.swift to be like JSONScanner.swift. Another possibility is make depth start from -1, so it is at 0 for the first iteration. But I think that needs similar comment just like increasing the maximum limit.
But if you still prefer to see an alternate appraoch that refactor more code, lmk and I can look into that. |
Edit: I noticed that I hadn't submitted my review. I'm sorry for having just requested reviews from others before my review...🙇 @kperryua @jevonmao It may look like: mutating func serializeJSON(_ value: JSONEncoderValue, depth: Int = 0) throws {
switch value {
case .string(let str):
serializeString(str)
case .bool(let boolValue):
writer(boolValue ? "true" : "false")
case .number(let numberStr):
writer(contentsOf: numberStr.utf8)
case .array(let array):
guard depth < Self.maximumRecursionDepth else {
throw JSONError.tooManyNestedArraysOrDictionaries()
}
try serializeArray(array, depth: depth + 1)
case .nonPrettyDirectArray(let arrayRepresentation):
writer(contentsOf: arrayRepresentation)
case let .directArray(bytes, lengths):
guard depth < Self.maximumRecursionDepth else {
throw JSONError.tooManyNestedArraysOrDictionaries()
}
try serializePreformattedByteArray(bytes, lengths, depth: depth + 1)
case .object(let object):
guard depth < Self.maximumRecursionDepth else {
throw JSONError.tooManyNestedArraysOrDictionaries()
}
try serializeObject(object, depth: depth + 1)
case .null:
writer("null")
}
} |
I think it is possible to further move the guard statements to top of the function? mutating func serializeJSON(_ value: JSONEncoderValue, depth: Int = 0) throws {
guard depth < Self.maximumRecursionDepth else {
throw JSONError.tooManyNestedArraysOrDictionaries()
}
switch value {
case .string(let str):
serializeString(str)
case .bool(let boolValue):
writer(boolValue ? "true" : "false")
case .number(let numberStr):
writer(contentsOf: numberStr.utf8)
case .array(let array):
try serializeArray(array, depth: depth + 1)
case .nonPrettyDirectArray(let arrayRepresentation):
writer(contentsOf: arrayRepresentation)
case let .directArray(bytes, lengths):
try serializePreformattedByteArray(bytes, lengths, depth: depth + 1)
case .object(let object):
try serializeObject(object, depth: depth + 1)
case .null:
writer("null")
}
} |
@YOCKOW Additionally, while debugging I discovered another issue with the way error is propagated in ![]() |
Yeah, I think it's also ok if it doesn't affect perf.
I'm not sure if it's intended or a bug, but I agree that it can be misleading. |
@swift-ci Please smoke test |
@swift-ci Please test |
Jenkins hasn't reported results on macOS and on Linux (why?), but they seem to be passed:
However, it seems that |
@YOCKOW I'm not too sure either, looking at the Windows test outputs it says:
All the test also pass without issues on my local development environment in Xcode. Can you give some help on this? |
That message just means that no tests did run with I guess the problem might lie in On macOS: Test Case '-[FoundationEssentialsTests.JSONEncoderTests test_JSONPassTests]' started.
Test Case '-[FoundationEssentialsTests.JSONEncoderTests test_JSONPassTests]' passed (0.218 seconds). On Linux: Test Case 'JSONEncoderTests.test_JSONPassTests' started at 2025-04-10 06:14:47.294
Test Case 'JSONEncoderTests.test_JSONPassTests' passed (0.197 seconds) On the other hand, on Windows: Test Case 'JSONEncoderTests.test_JSONPassTests' started at 2025-04-10 06:24:12.054 ...Neither @compnerd Would you give us some hints why this change fails only on Windows? |
It is difficult to say, but my guess would be stack exhaustion as it seems that you are removing some of the recursion limits. Note that Windows has a much tighter bounds on the stack than Linux or macOS, so unless you are ensuring that the call is performing tail call elimination, you would exhaust the stack sooner on Windows than on other platforms. |
Do you know if it is possible to debug the Windows issue if I only have Mac device? |
Thank you for your response. That makes sense. By the way, I did some coarse experiments1. From the results of them, program compiled with (My questions is: Do some build configs of the compiler affect the results?) 🤔 Then, what do you think we should do? @jevonmao @kperryua @parkera
Footnotes |
@YOCKOW Personally I'm more on the side of keeping this PR within the scope of the issue it addresses, and postpone the fix for another future PR. But if you have other thoughts, just let me know and I can work on it. |
The bottom line is that we can't merge this if it breaks Windows. Maybe the limit needs to be platform-specific? |
@parkera I believe as pointed out by @YOCKOW , the new limits added isn't what caused the Windows crash, but rather the test case added just revealed it by testing a deep JSON recursion case. |
Fix the JSON Decoder and Encoder limit disagreement bug described in issue #1176.
Due to the
JSONWriter
uses passed parameter to track the depth limit, its depth starts counting from 1 for the 1st iteration. ButJSONScanner
starts counting from 0. This result in large deep nested JSONs being able to decode without issues, but crashes when encoded in reverse.I added additional JSON test file and test case to reproduce the error, and then incremented the maximum recursion depth variable for encoding to account for the off-by-1.