Skip to content

Allow tests to differentiate between iterations when ran repeatedly #1041

@Stannieman

Description

@Stannieman

Motivation

When running a test repeatedly, it is currently impossible for code inside the test to differentiate between different iterations within the same test run.

A concrete example where this poses an issue is with the SnapshotTesting library. It generates a unique name for each snapshot used in a test. E.g. when a test checks 3 snapshots it generates 3 unique names for those. When a test is ran multiple times it needs to "reset" these names between test iterations for this to work. However, it currently has no way to know the difference between 3 snapshots within a single test or 1 snapshot in a test that's ran 3 times repeated.

To solve this it needs to be able to detect that a test it has seen before is actually a different iteration. The issue is being tracked on there side here: pointfreeco/swift-snapshot-testing#963.

Proposed solution

A solution could be to add an iteration: UInt property to Test that gets incremented for each iteration. Something like this is actually already printed to the console in Xcode. SnapshotTesting could use that to "reset" the unique snapshot names when the iteration changes for the same test.

I feel like this somewhat relates to #671, where uniquely identifying parametrized cases are discussed. They are not entirely the same though.
Snapshot names should not be reset for different parametrized cases, only for different iterations when running repeatedly. Similarly, you probably want to be able to still identify a parametrized case across different iterations.
So while both problems and solutions are very similar, they do have different use cases and probably ask for separate fields.

Alternatives considered

No response

Additional information

No response

Activity

stmontgomery

stmontgomery commented on Mar 28, 2025

@stmontgomery
Contributor

Can you clarify what mechanism you are using to run a test repeatedly? Are you using the Xcode feature "Run Test Repeatedly…", for example, or do you have a parameterized test function, or is it using some other technique?

Stannieman

Stannieman commented on Mar 28, 2025

@Stannieman
Author

It is indeed through the Xcode UI.

Checking "Relaunch Tests for Each Repetition" does not change the behavior either. (Is this a bug? I'd expect global variables to be reset when this is checked, but they are not.)

grynspan

grynspan commented on Mar 28, 2025

@grynspan
Contributor

The testing library reports which iteration is running (and it can be computed trivially in the event stream handler anyway), so I suspect this needs to be tracked on Apple's side.

stmontgomery

stmontgomery commented on Mar 28, 2025

@stmontgomery
Contributor

The request here is to introduce a mechanism by which an actively-running test could inspect which iteration it's running. I do think that would require internal modeling and likely API additions in the testing library itself. One way this could be conveyed is via a property on Test.Case; then, querying Test.Case.current would allow a test function to access it. We'd need to think through whether Test.Case or Test is the right place, though; it's possible suites may want access to the iteration count, too.

Then separately, Xcode would able to adopt that information once available (perhaps replacing the mechanism it currently uses), and that adoption task should be tracked separately.

grynspan

grynspan commented on Mar 28, 2025

@grynspan
Contributor

Fair enough.

Stannieman

Stannieman commented on Apr 1, 2025

@Stannieman
Author

The issue in SnapshotTesting was fixed in a different way. This now becomes a bit less relevant, but if it's something that will still be useful in the grand scheme of things it can remain open of course.

stephencelis

stephencelis commented on Apr 1, 2025

@stephencelis

For context, we simply require a new test scoping trait to reset this state at a task-local level.

It still would be nice for libraries to be able to detect this kind of thing and provide a nice default behavior (or warning that a necessary trait isn't applied).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestpublic-apiAffects public APItools integration🛠️ Integration of swift-testing into tools/IDEs

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @stephencelis@stmontgomery@Stannieman@grynspan

        Issue actions

          Allow tests to differentiate between iterations when ran repeatedly · Issue #1041 · swiftlang/swift-testing