Inheriting a suite should inherit its tests

[soumyamahunt]

Motivation

In XCTest, inheriting a XCTestCase class inherits all its tests creating new test case. This is especially useful when testing behaviors with different configurations. i.e.

class FeatureAWithConfigX: XCTestCase {
    override setup() {
        // setup config X
    }

    // Test behavior
    func testStuff() {}
}

class FeatureAWithConfigY: FeatureAWithConfigX {
    override setup() {
        // setup config Y
    }

    // No need to write tests for same behaviors
}

but this kind of usage is not possible with swift testing:

@Suite
class FeatureAWithConfigX {
    init() {
        // setup config X
    }

    // Test behavior
    @Test
    func stuff() {}
}

@Suite
class FeatureAWithConfigY: FeatureAWithConfigX {
    override init() {
        // setup config Y
    }

    // Write the tests again?
}

Proposed solution

Inheriting a suite should inherit all its tests as well, or any other functionality that could allow reusing all the tests defined in a test suite (may be parameterization at @suite level?)

@Suite
class FeatureAWithConfigX {
    init() {
        // setup config X
    }

    // Test behavior
    @Test
    func stuff() {}
}

@Suite
class FeatureAWithConfigY: FeatureAWithConfigX {
    override init() {
        // setup config Y
    }

    // Reuse same tests
}

Alternatives considered

Couldn't find any alternatives going through documentation.

Additional information

A real world usage of such pattern will be let's suppose you have written some logic available in ModuleA and you have already written tests for ModuleA. Now you are refactoring this into a ModuleB, but to avoid any breaking functionality you want to ship both ModuleA and ModuleB while ModuleB usage is controlled by some flag. To test both ModuleA and ModuleB have same behaviors you will need such functionality.

[grynspan]

I think we've had an issue about this filed previously but I don't see it.

This is something we'd like to support but it would be a source-breaking change at this time, so it would need to either be opt-in or wait until the next Swift language mode.

It might be appropriate to express this relationship via a whole new suite trait such as .inherited, though I'd prefer to infer this from language-level features like open vs. final.

There are also some unfortunate constraints around algorithmic complexity. The naïve/comprehensive approach (looking for subclasses) is at least O(n^2) across all types in the system. A more optimized approach (looking for superclasses) requires that subclasses explicitly opt in somehow, which may not always be feasible.

[grynspan]

Tracked internally as rdar://154663240.

[soumyamahunt]

This is something we'd like to support but it would be a source-breaking change at this time, so it would need to either be opt-in or wait until the next Swift language mode.

Do you mean this would be source breaking change for this kind of scenario, or do you have any other scenario in mind:

@Suite
class FeatureAWithConfigX {
    init() {
        // setup config X
    }

    // Test behavior
    @Test
    func stuff() {}
}

@Suite
class FeatureAWithConfigY: FeatureAWithConfigX {
    override init() {
        // setup config Y
    }

    // Test behavior
    @Test
    func stuff() {} // this will break
}

It might be appropriate to express this relationship via a whole new suite trait such as .inherited, though I'd prefer to infer this from language-level features like open vs. final.

I think this would be acceptable as well. For such a niche case introducing new language mode might not be ideal.

There are also some unfortunate constraints around algorithmic complexity. The naïve/comprehensive approach (looking for subclasses) is at least O(n^2) across all types in the system. A more optimized approach (looking for superclasses) requires that subclasses explicitly opt in somehow, which may not always be feasible.

Wouldn't the same be applicable to XCTest as well? If this is applicable to XCTest then this shouldn't cause any performance regression when migrating to swift-testing.

[grynspan]

Do you mean this would be source breaking change for this kind of scenario, or do you have any other scenario in mind:

It would mean that if you've already used classes to arrange your code, suddenly your subclasses would have additional test functions that weren't present previously. It would also pose challenges for Xcode/VSCode integration since class hierarchies aren't fully visible to the IDE prior to compilation.

[grynspan]

Wouldn't the same be applicable to XCTest as well? If this is applicable to XCTest then this shouldn't cause any performance regression when migrating to swift-testing.

No. XCTest test functions are enumerated by the Objective-C runtime (or, when using corelibs-xctest, magical gunk in between the compiler and SwiftPM that has deep tendrils.) This operation is transparent to us. Swift Testing is pure Swift and is implemented using Swift macros. Swift does not maintain a runtime-visible list of functions/methods/selectors that a class implements, so there's no automatic discovery mechanism for us to rely on.

[soumyamahunt]

@grynspan what about parameterization support at @Suite level?

@Suite(arguments: [configA, configB])
class FeatureA {
    init(config: Config) {
        // setup config passed
    }

    // Test behavior
    @Test
    func stuff() {}
}

I think that could be used to achieve such use-cases and this doesn't break any existing cases. Do you think achieving parameterization will be simpler than test methods inheritance?