Problems with async test
BrunoMazzo opened this issue · comments
Hi, I'm having problems when using an async test with KIF. I was able to create a simple app with one tests that always pass if the test is a sync function, but always fails if it is an async func:
struct ContentView: View {
@State var showAlert = false
@State var showSheet = false
var body: some View {
NavigationView {
VStack {
Button {
showAlert = true
} label: {
Text("Show alert")
}
.alert("Alert", isPresented: $showAlert) {
Button {
showSheet = true
} label: {
Text("Show sheet")
}
}
.sheet(isPresented: $showSheet) {
Text("Hello world")
}
}
.padding()
}
}
}
final class kifasyncTests: KIFTestCase {
@MainActor
func testAsyncMethod() async throws {
// iOS 15.5 and 16.2 Always fails here. Nothing happens
tester().tapView(withAccessibilityLabel: "Show alert")
// iOS 15.4 Always fails here. I cannot see the alert button press
tester().tapView(withAccessibilityLabel: "Show sheet")
// iOS 16.0 Always fails here. I can see the alert button being press but the alert never dismiss
try! tester().tryFindingView(withAccessibilityLabel: "Hello world")
}
//Always pass without any warning
func testSyncMethod() {
tester().tapView(withAccessibilityLabel: "Show alert")
tester().tapView(withAccessibilityLabel: "Show sheet")
try! tester().tryFindingView(withAccessibilityLabel: "Hello world")
}
}
I'm able to reproduce the error with iOS 15.4, 15.5, 16.0 and 16.2 simulators (didn't test early versions) and with a real device with iOS 16.2. I'm using Xcode 14.2 and the latests KIF version from master.
Facing similar error as the UI does not update when the KIF perform a step.
With some investigation, I could mitigate the failures by adding tester().wait(forTimeInterval: 0.01) to before an action that causes the freezing.
Example test is as follow:
import Nimble
import Quick
import KIF
final class HomeSpec: QuickSpec {
override func spec() {
describe("The app") {
describe("its open") {
it("it shows its ui components") {
self.tester().wait(forTimeInterval: 0.01)
await self.checkHello()
await self.showAlert()
await self.closeAlert()
self.tester().wait(forTimeInterval: 1.01)
await self.checkNoAlert()
}
}
}
}
@MainActor
func checkHello() {
self.tester().waitForView(withAccessibilityLabel: "Hello, world!")
}
@MainActor
func showAlert() {
self.tester().tapView(withAccessibilityLabel: "Show")
}
@MainActor
func closeAlert() {
self.tester().tapView(withAccessibilityLabel: "Close")
}
@MainActor
func checkNoAlert() {
self.tester().waitForAbsenceOfView(withAccessibilityLabel: "OK")
}
}
extension QuickSpec {
func tester(file: String = #file, _ line: Int = #line) -> KIFUITestActor {
return KIFUITestActor(inFile: file, atLine: line, delegate: self)
}
func system(file: String = #file, _ line: Int = #line) -> KIFSystemTestActor {
return KIFSystemTestActor(inFile: file, atLine: line, delegate: self)
}
}
Not sure why KIF blocks the UI thread from updating or is there a better way to wait for UI updates.
I tried to add some wait but in some more complex cases I have it was still failing. I did some digging and I think the problem is that CFRunLoopRunInMode doesn't work if you are in an async context. Found in the CFRunLoop.h file CF_SWIFT_UNAVAILABLE_FROM_ASYNC("CFRunLoopRunInMode cannot be used from async contexts."). So I try to make a wrapper to switch out of the async context and it worked for now. I end up with something like this:
class AsyncKIFUITestActor {
let kitTestActor: KIFUITestActor
init(inFile file: String = #file, atLine line: Int = #line, delegate: KIFTestActorDelegate) {
kitTestActor = KIFUITestActor(inFile: file, atLine: line, delegate: delegate)
}
func tapView(withAccessibilityLabel accessibilityLabel: String) async {
return await withCheckedContinuation { continuation in
DispatchQueue.main.async {
self.kitTestActor.tapView(withAccessibilityLabel: accessibilityLabel)
continuation.resume()
}
}
}
func tryFindingView(withAccessibilityLabel accessibilityLabel: String) async throws {
return try await withCheckedThrowingContinuation { continuation in
DispatchQueue.main.async {
do {
try self.kitTestActor.tryFindingView(withAccessibilityLabel: accessibilityLabel)
continuation.resume()
} catch {
continuation.resume(throwing: error)
}
}
}
}
...
}
extension XCTestCase {
func asyncTester(file : String = #file, _ line : Int = #line) -> AsyncKIFUITestActor {
return AsyncKIFUITestActor(inFile: file, atLine: line, delegate: self)
}
}
and then changed the test to use it:
final class KIFAsyncTests: XCTestCase {
@MainActor
func testAsyncMethod() async throws {
await asyncTester().tapView(withAccessibilityLabel: "Show alert")
await asyncTester().tapView(withAccessibilityLabel: "Show sheet")
try await asyncTester().tryFindingView(withAccessibilityLabel: "Hello world")
}
}I had to create a wrapper for every function that I used, but it appears to work fine. I need to test in more complex projects.
Thanks for digging in on this! Would this change work for the existing KIVUIViewTestActor tester, or does it represent a change in behavior for places that aren't using async? I'm wondering if it makes sense to replace this in place instead of setting up a new duplicated actor?
I think it is a change in behaviour. Non async functions can't call it. It is possible to overload the current actors to have both functions, one sync and another async, but it will have a lot of duplicated code.
It may be easier extending viewTester than tester, as that has the action and predicate builder methods decoupled. We could probably add a usingAsync: property setter that flips the behavior between sync and async waiting. We could expose a static property setter to modify the defaults for tests that primarily expect the async behavior.
I'm not actively working on KIF at the moment, but I'm definitely available for guidance and can review changes if this is something you wanted to work on. Our codebase doesn't have SwiftUI, so there hasn't been a need to solve this problem for ourselves. Thanks for looking into this!