Support throwing errors from within `@Test` attributes

[Johannes Auer]

Description

The argument parameter of Test(_:_:arguments:) is currently very limited.

Adding a version of Test(_:_:arguments: () async throws -> C) would greatly improve the parameterisation possibilities (e.g. async loading of parameters from a file).

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

swift-testing version/commit hash

No response

Swift & OS version (output of swift --version && uname -a)

No response

[Jonathan Grynspan]

The @Test macro supports async arguments. It doesn't currently support throwing arguments because that's somewhat harder to model in a reasonable way and we need to figure out what it means to throw an error from within an attribute.

Does that help at all?

[Johannes Auer]

The @Test macro supports async arguments.

I didn't know that - great!

It doesn't currently support throwing arguments because that's somewhat harder to model in a reasonable way and we need to figure out what it means to throw an error from within an attribute.

Wouldn't it be easy to allow () async throws -> C as an argument and let the test fail if an error is thrown? The advantage of a closure would be that a test could be more self-contained, without helper functions providing the arguments. Allowing a throwing closure would also reduce the boilerplate required by wrapping throwing code in do catch blocks and letting the error fail (or crash). So instead of:

func myArguments() async -> [String] {
    do {
        try await ...
    } catch {
        ...
    }
}

@Test("test all strings", arguments: await myArguments())
func allStrings(stirng: String) {
    ...
}

we could write

@Test("test all strings") {
    try await ...
}
func allStrings(stirng: String) {
    ...
}

I have not spent any time with macros in Swift, so there may be a limitation I am not aware of.

[Jonathan Grynspan]

It is not hard to allow the developer to express try in that position—what's hard is figuring out how to actually handle the error if it occurs, and there are multiple answers that seem reasonable.

A failure to generate an arguments collection for a test means we cannot fully instantiate that test as specified by the developer. We've been leaning toward saying that we should instead instantiate a monomorphic Test instance that fails immediately with the thrown error, however this results in a test with a physical structure that does not match what the developer intended, and that could be confusing. We already do something similar for tests with unavailable arguments (i.e. @available prevented them from being resolved) but such tests are skipped at runtime anyway, so it has less of a visible impact on the test output.

we could write

@test("test all strings") {
try await ...
}
func allStrings(stirng: String) {
...
}

This would not be grammatically correct Swift, so you'd have to write something like @Test("test all strings", arguments: { try await ... }) instead. Supporting a closure in that position would require additional overloads to the Test macro. While this is technically feasible, I think we'd need some strong justification for adding those overloads when your initial example (minus the do/catch, as we are planning to support try here in the future) is equivalent.

[Johannes Auer]

Ok, that makes sense. Adding try would be a great addition to the api.

Thanks for your time!