Skip to content

Commit 6894115

Browse files
grynspangrynspan
committed
Rethink how we capture expectation conditions and their subexpressions.
This PR completely rewrites how we capture expectation conditions. For example, given the following expectation: ```swift ``` We currently detect that there is a binary operation and emit code that calls the binary operator as a closure and passes the left-hand value and right-hand value, then checks that the result of the operation is `true`. This is sufficient for simpler expressions like that one, but more complex ones (including any that involve `try` or `await` keywords) cannot be expanded correctly. With this PR, such expressions _can_ generally be expanded correctly. The change involves rewriting the macro condition as a closure to which is passed a local, mutable "context" value. Subexpressions of the condition expression are then rewritten by walking the syntax tree of the expression (using typical swift-syntax API) and replacing them with calls into the context value that pass in the value and related state. If the expectation ultimately fails, the collected data is transformed into an instance of the SPI type `Expression` that contains the source code of the expression and interesting subexpressions as well as the runtime values of those subexpressions. Nodes in the syntax tree are identified by a unique ID which is composed of the swift-syntax ID for that node as well as all its parent nodes in a compact bitmask format. These IDs can be transformed into graph/trie key paths when expression/subexpression relationships need to be reconstructed on failure, meaning that a single rewritten node doesn't otherwise need to know its "place" in the overall expression. There remain a few caveats (that also generally affect the current implementation): - Mutating member functions are syntactically indistinguishable from non-mutating ones and miscompile when rewritten; - Expressions involving move-only types are also indistinguishable, but need lifetime management to be rewritten correctly; and - Expressions where the `try` or `await` keyword is _outside_ the `#expect` macro cannot be expanded correctly because the macro cannot see those keywords during expansion. The first issue might be resolvable in the future using pointer tricks, although I don't hold a lot of hope for it. The second issue is probably resolved by non-escaping types. The third issue is an area of active exploration for us and the macros/swift-syntax team.
1 parent 981aa1c commit 6894115

40 files changed

+2529
-1716
lines changed

‎Package.swift

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,13 @@ let package = Package(
130130
],
131131
swiftSettings: .packageSettings
132132
),
133+
.testTarget(
134+
name: "SubexpressionShowcase",
135+
dependencies: [
136+
"Testing",
137+
],
138+
swiftSettings: .packageSettings
139+
),
133140

134141
.macro(
135142
name: "TestingMacros",
@@ -266,6 +273,7 @@ extension Array where Element == PackageDescription.SwiftSetting {
266273

267274
result += [
268275
.enableUpcomingFeature("ExistentialAny"),
276+
.enableExperimentalFeature("NonescapableTypes"),
269277

270278
.enableExperimentalFeature("AccessLevelOnImport"),
271279
.enableUpcomingFeature("InternalImportsByDefault"),

‎Sources/Testing/ABI/Encoded/ABI.EncodedExpectation.swift

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
//
2+
// This source file is part of the Swift.org open source project
3+
//
4+
// Copyright (c) 2025 Apple Inc. and the Swift project authors
5+
// Licensed under Apache License v2.0 with Runtime Library Exception
6+
//
7+
// See https://swift.org/LICENSE.txt for license information
8+
// See https://swift.org/CONTRIBUTORS.txt for Swift project authors
9+
//
10+
11+
extension ABI {
12+
/// A type implementing the JSON encoding of ``Expectation`` for the ABI entry
13+
/// point and event stream output.
14+
///
15+
/// This type is not part of the public interface of the testing library. It
16+
/// assists in converting values to JSON; clients that consume this JSON are
17+
/// expected to write their own decoders.
18+
///
19+
/// - Warning: Expectations are not yet part of the JSON schema.
20+
struct EncodedExpectation<V>: Sendable where V: ABI.Version {
21+
/// The expression evaluated by this expectation.
22+
///
23+
/// - Warning: Expressions are not yet part of the JSON schema.
24+
var _expression: EncodedExpression<V>
25+
26+
init(encoding expectation: borrowing Expectation, in eventContext: borrowing Event.Context) {
27+
_expression = EncodedExpression<V>(encoding: expectation.evaluatedExpression, in: eventContext)
28+
}
29+
}
30+
}
31+
32+
// MARK: - Codable
33+
34+
extension ABI.EncodedExpectation: Codable {}

‎Sources/Testing/ABI/Encoded/ABI.EncodedExpression.swift

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
//
2+
// This source file is part of the Swift.org open source project
3+
//
4+
// Copyright (c) 2025 Apple Inc. and the Swift project authors
5+
// Licensed under Apache License v2.0 with Runtime Library Exception
6+
//
7+
// See https://swift.org/LICENSE.txt for license information
8+
// See https://swift.org/CONTRIBUTORS.txt for Swift project authors
9+
//
10+
11+
extension ABI {
12+
/// A type implementing the JSON encoding of ``Expression`` for the ABI entry
13+
/// point and event stream output.
14+
///
15+
/// This type is not part of the public interface of the testing library. It
16+
/// assists in converting values to JSON; clients that consume this JSON are
17+
/// expected to write their own decoders.
18+
///
19+
/// - Warning: Expressions are not yet part of the JSON schema.
20+
struct EncodedExpression<V>: Sendable where V: ABI.Version {
21+
/// The source code of the original captured expression.
22+
var sourceCode: String
23+
24+
/// A string representation of the runtime value of this expression.
25+
///
26+
/// If the runtime value of this expression has not been evaluated, the
27+
/// value of this property is `nil`.
28+
var runtimeValue: String?
29+
30+
/// The fully-qualified name of the type of value represented by
31+
/// `runtimeValue`, or `nil` if that value has not been captured.
32+
var runtimeTypeName: String?
33+
34+
/// Any child expressions within this expression.
35+
var children: [EncodedExpression]?
36+
37+
init(encoding expression: borrowing __Expression, in eventContext: borrowing Event.Context) {
38+
sourceCode = expression.sourceCode
39+
runtimeValue = expression.runtimeValue.map(String.init(describingForTest:))
40+
runtimeTypeName = expression.runtimeValue.map(\.typeInfo.fullyQualifiedName)
41+
if !expression.subexpressions.isEmpty {
42+
children = expression.subexpressions.map { [eventContext = copy eventContext] subexpression in
43+
Self(encoding: subexpression, in: eventContext)
44+
}
45+
}
46+
}
47+
}
48+
}
49+
50+
// MARK: - Codable
51+
52+
extension ABI.EncodedExpression: Codable {}

‎Sources/Testing/ABI/Encoded/ABI.EncodedIssue.swift

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,11 @@ extension ABI {
4545
/// - Warning: Errors are not yet part of the JSON schema.
4646
var _error: EncodedError<V>?
4747

48+
/// The expectation associated with this issue, if applicable.
49+
///
50+
/// - Warning: Expectations are not yet part of the JSON schema.
51+
var _expectation: EncodedExpectation<V>?
52+
4853
init(encoding issue: borrowing Issue, in eventContext: borrowing Event.Context) {
4954
_severity = switch issue.severity {
5055
case .warning: .warning
@@ -58,6 +63,9 @@ extension ABI {
5863
if let error = issue.error {
5964
_error = EncodedError(encoding: error, in: eventContext)
6065
}
66+
if case let .expectationFailed(expectation) = issue.kind {
67+
_expectation = EncodedExpectation(encoding: expectation, in: eventContext)
68+
}
6169
}
6270
}
6371
}

‎Sources/Testing/ABI/Encoded/ABI.EncodedMessage.swift

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,11 +64,19 @@ extension ABI {
6464
/// The symbol associated with this message.
6565
var symbol: Symbol
6666

67+
/// How much to indent this message when presenting it.
68+
///
69+
/// - Warning: This property is not yet part of the JSON schema.
70+
var _indentation: Int?
71+
6772
/// The human-readable, unformatted text associated with this message.
6873
var text: String
6974

7075
init(encoding message: borrowing Event.HumanReadableOutputRecorder.Message) {
7176
symbol = Symbol(encoding: message.symbol ?? .default)
77+
if message.indentation > 0 {
78+
_indentation = message.indentation
79+
}
7280
text = message.conciseStringValue ?? message.stringValue
7381
}
7482
}

‎Sources/Testing/CMakeLists.txt

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ add_library(Testing
1717
ABI/Encoded/ABI.EncodedBacktrace.swift
1818
ABI/Encoded/ABI.EncodedError.swift
1919
ABI/Encoded/ABI.EncodedEvent.swift
20+
ABI/Encoded/ABI.EncodedExpectation.swift
21+
ABI/Encoded/ABI.EncodedExpression.swift
2022
ABI/Encoded/ABI.EncodedInstant.swift
2123
ABI/Encoded/ABI.EncodedIssue.swift
2224
ABI/Encoded/ABI.EncodedMessage.swift
@@ -41,6 +43,8 @@ add_library(Testing
4143
Expectations/Expectation.swift
4244
Expectations/Expectation+Macro.swift
4345
Expectations/ExpectationChecking+Macro.swift
46+
Expectations/ExpectationContext.swift
47+
Expectations/ExpectationContext+Pointers.swift
4448
Issues/Confirmation.swift
4549
Issues/ErrorSnapshot.swift
4650
Issues/Issue.swift
@@ -63,7 +67,7 @@ add_library(Testing
6367
SourceAttribution/Backtrace+Symbolication.swift
6468
SourceAttribution/CustomTestStringConvertible.swift
6569
SourceAttribution/Expression.swift
66-
SourceAttribution/Expression+Macro.swift
70+
SourceAttribution/ExpressionID.swift
6771
SourceAttribution/SourceContext.swift
6872
SourceAttribution/SourceLocation.swift
6973
SourceAttribution/SourceLocation+Macro.swift

‎Sources/Testing/Events/Recorder/Event.ConsoleOutputRecorder.swift

Lines changed: 25 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,22 @@ private let _ansiEscapeCodePrefix = "\u{001B}["
129129
private let _resetANSIEscapeCode = "\(_ansiEscapeCodePrefix)0m"
130130

131131
extension Event.Symbol {
132+
/// Get the string value to use for a message with no associated symbol.
133+
///
134+
/// - Parameters:
135+
/// - options: Options to use when writing the symbol.
136+
///
137+
/// - Returns: A string representation of "no symbol" appropriate for writing
138+
/// to a stream.
139+
fileprivate static func placeholderStringValue(options: Event.ConsoleOutputRecorder.Options) -> String {
140+
#if os(macOS) || (os(iOS) && targetEnvironment(macCatalyst))
141+
if options.useSFSymbols {
142+
return " "
143+
}
144+
#endif
145+
return " "
146+
}
147+
132148
/// Get the string value for this symbol with the given write options.
133149
///
134150
/// - Parameters:
@@ -171,7 +187,7 @@ extension Event.Symbol {
171187
case .attachment:
172188
return "\(_ansiEscapeCodePrefix)94m\(symbolCharacter)\(_resetANSIEscapeCode)"
173189
case .details:
174-
return symbolCharacter
190+
return "\(symbolCharacter)"
175191
}
176192
}
177193
return "\(symbolCharacter)"
@@ -305,18 +321,12 @@ extension Event.ConsoleOutputRecorder {
305321
/// - Returns: Whether any output was produced and written to this instance's
306322
/// destination.
307323
@discardableResult public func record(_ event: borrowing Event, in context: borrowing Event.Context) -> Bool {
308-
let messages = _humanReadableOutputRecorder.record(event, in: context)
309-
310-
// Padding to use in place of a symbol for messages that don't have one.
311-
var padding = " "
312-
#if os(macOS) || (os(iOS) && targetEnvironment(macCatalyst))
313-
if options.useSFSymbols {
314-
padding = " "
315-
}
316-
#endif
324+
let symbolPlaceholder = Event.Symbol.placeholderStringValue(options: options)
317325

326+
let messages = _humanReadableOutputRecorder.record(event, in: context)
318327
let lines = messages.lazy.map { [test = context.test] message in
319-
let symbol = message.symbol?.stringValue(options: options) ?? padding
328+
let symbol = message.symbol?.stringValue(options: options) ?? symbolPlaceholder
329+
let indentation = String(repeating: " ", count: message.indentation)
320330

321331
if case .details = message.symbol {
322332
// Special-case the detail symbol to apply grey to the entire line of
@@ -325,17 +335,17 @@ extension Event.ConsoleOutputRecorder {
325335
// to the indentation provided by the symbol.
326336
var lines = message.stringValue.split(omittingEmptySubsequences: false, whereSeparator: \.isNewline)
327337
lines = CollectionOfOne(lines[0]) + lines.dropFirst().map { line in
328-
"\(padding) \(line)"
338+
"\(indentation)\(symbolPlaceholder) \(line)"
329339
}
330340
let stringValue = lines.joined(separator: "\n")
331341
if options.useANSIEscapeCodes, options.ansiColorBitDepth > 1 {
332-
return "\(_ansiEscapeCodePrefix)90m\(symbol) \(stringValue)\(_resetANSIEscapeCode)\n"
342+
return "\(_ansiEscapeCodePrefix)90m\(symbol) \(indentation)\(stringValue)\(_resetANSIEscapeCode)\n"
333343
} else {
334-
return "\(symbol) \(stringValue)\n"
344+
return "\(symbol) \(indentation)\(stringValue)\n"
335345
}
336346
} else {
337347
let colorDots = test.map { self.colorDots(for: $0.tags) } ?? ""
338-
return "\(symbol) \(colorDots)\(message.stringValue)\n"
348+
return "\(symbol) \(indentation)\(colorDots)\(message.stringValue)\n"
339349
}
340350
}
341351

‎Sources/Testing/Events/Recorder/Event.HumanReadableOutputRecorder.swift

Lines changed: 18 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,15 @@ extension Event {
2525
/// The symbol associated with this message, if any.
2626
var symbol: Symbol?
2727

28+
/// How much to indent this message when presenting it.
29+
///
30+
/// The way in which this additional indentation is rendered is
31+
/// implementation-defined. Typically, the greater the value of this
32+
/// property, the more whitespace characters are inserted.
33+
///
34+
/// Rendering of indentation is optional.
35+
var indentation = 0
36+
2837
/// The human-readable message.
2938
var stringValue: String
3039

@@ -463,20 +472,18 @@ extension Event.HumanReadableOutputRecorder {
463472
additionalMessages.append(_formattedComment(knownIssueComment))
464473
}
465474

466-
if verbosity > 0, case let .expectationFailed(expectation) = issue.kind {
475+
if verbosity >= 0, case let .expectationFailed(expectation) = issue.kind {
467476
let expression = expectation.evaluatedExpression
468-
func addMessage(about expression: __Expression) {
469-
let description = expression.expandedDebugDescription()
470-
additionalMessages.append(Message(symbol: .details, stringValue: description))
471-
}
472-
let subexpressions = expression.subexpressions
473-
if subexpressions.isEmpty {
474-
addMessage(about: expression)
475-
} else {
476-
for subexpression in subexpressions {
477-
addMessage(about: subexpression)
477+
func addMessage(about expression: __Expression, depth: Int) {
478+
let description = expression.expandedDescription(verbose: verbosity > 0)
479+
if description != expression.sourceCode {
480+
additionalMessages.append(Message(symbol: .details, indentation: depth, stringValue: description))
481+
}
482+
for subexpression in expression.subexpressions {
483+
addMessage(about: subexpression, depth: depth + 1)
478484
}
479485
}
486+
addMessage(about: expression, depth: 0)
480487
}
481488

482489
let atSourceLocation = issue.sourceLocation.map { " at \($0)" } ?? ""

‎Sources/Testing/ExitTests/ExitTest.swift

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -421,7 +421,7 @@ func callExitTest(
421421
encodingCapturedValues capturedValues: [ExitTest.CapturedValue],
422422
processExitsWith expectedExitCondition: ExitTest.Condition,
423423
observing observedValues: [any PartialKeyPath<ExitTest.Result> & Sendable],
424-
expression: __Expression,
424+
sourceCode: @escaping @autoclosure @Sendable () -> [__ExpressionID: String],
425425
comments: @autoclosure () -> [Comment],
426426
isRequired: Bool,
427427
isolation: isolated (any Actor)? = #isolation,
@@ -479,10 +479,14 @@ func callExitTest(
479479
}
480480

481481
// Plumb the exit test's result through the general expectation machinery.
482-
return __checkValue(
482+
let expectationContext = __ExpectationContext(
483+
sourceCode: sourceCode(),
484+
runtimeValues: [.root: { Expression.Value(reflecting: result.exitStatus) }]
485+
)
486+
return check(
483487
expectedExitCondition.isApproximatelyEqual(to: result.exitStatus),
484-
expression: expression,
485-
expressionWithCapturedRuntimeValues: expression.capturingRuntimeValues(result.exitStatus),
488+
expectationContext: expectationContext,
489+
mismatchedErrorDescription: nil,
486490
mismatchedExitConditionDescription: String(describingForTest: expectedExitCondition),
487491
comments: comments(),
488492
isRequired: isRequired,

‎Sources/Testing/Expectations/Expectation+Macro.swift

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -66,10 +66,10 @@
6666
/// running in the current task and an instance of ``ExpectationFailedError`` is
6767
/// thrown.
6868
@freestanding(expression) public macro require<T>(
69-
_ optionalValue: T?,
69+
_ optionalValue: consuming T?,
7070
_ comment: @autoclosure () -> Comment? = nil,
7171
sourceLocation: SourceLocation = #_sourceLocation
72-
) -> T = #externalMacro(module: "TestingMacros", type: "RequireMacro")
72+
) -> T = #externalMacro(module: "TestingMacros", type: "UnwrapMacro") where T: ~Copyable
7373

7474
/// Unwrap an optional boolean value or, if it is `nil`, fail and throw an
7575
/// error.
@@ -124,10 +124,10 @@ public macro require(
124124
@freestanding(expression)
125125
@_documentation(visibility: private)
126126
public macro require<T>(
127-
_ optionalValue: T,
127+
_ optionalValue: consuming T,
128128
_ comment: @autoclosure () -> Comment? = nil,
129129
sourceLocation: SourceLocation = #_sourceLocation
130-
) -> T = #externalMacro(module: "TestingMacros", type: "NonOptionalRequireMacro")
130+
) -> T = #externalMacro(module: "TestingMacros", type: "NonOptionalRequireMacro") where T: ~Copyable
131131

132132
// MARK: - Matching errors by type
133133

0 commit comments

Comments
 (0)