Very long time to execute `SBTUITestTunnelServer.takeOff()`
acecilia opened this issue · comments
Hi đŸ‘‹
I am experiencing that executing SBTUITestTunnelServer.takeOff() takes very long time, between 20 to 60 seconds. I am using latest version 8.3.0. This happens with SBTUITestTunnelDisableIPC true or false.
Is this expected?
Thanks
Did you try running tests disabling the debugger in the scheme? Are you using Xcode 13 or 14?
error build: The package product 'SBTUITestTunnelServer' cannot be used as a dependency of this target because it uses unsafe build flags.
@tcamin Thanks for the quick response.
Did you try running tests disabling the debugger in the scheme?
I did unmark Debug executable from the Xcode scheme. I see that takeOff takes variable times of at least 10 seconds.
Are you using Xcode 13 or 14?
I am using Xcode 13.4.1
Do you know, how long is takeOff supposed to take?
Do you know, how long is takeOff supposed to take?
Consider that takeOff completes once both runner app and app under tests are launched, so the time that is logged to console (Tunnel ready after 3.760287s) doesn't simply account to the library implementation. For our app on an M1 we see times in the range of 3-6s.
completes once both runner app and app under tests are launched
I do not fully understand. I am executing takeOff from inside the delegate, so the moment takeOff is invoked, I think both app and app runner are already launched. No?
so the time that is logged to console (Tunnel ready after 3.760287s) doesn't simply account to the library implementation
I did not rely on the Tunnel ready after 3.760287s text. The way I am measuring the delay is by doing this:
print("\(Date().timeIntervalSince1970) - before takeoff")
SBTUITestTunnelServer.takeOff()
print("\(Date().timeIntervalSince1970) - after takeoff")
I am also running on M1
For our app on an M1 we see times in the range of 3-6s.
Thanks for that info. I would say that is also a considerable amount of time. I think a reasonable amount of time for a big UITests is 30 seconds: 6 seconds delay is 20% of the total test execution time
completes once both runner app and app under tests are launched
I do not fully understand. I am executing
takeOfffrom inside the delegate, so the momenttakeOffis invoked, I think both app and app runner are already launched. No?
I didn't understand you were referring to the sole takeOff method but instead were mentioning the time logged on the runner app side (Tunnel ready after ...)
so the time that is logged to console (Tunnel ready after 3.760287s) doesn't simply account to the library implementation
I did not rely on the Tunnel ready after 3.760287s text. The way I am measuring the delay is by doing this:
print("\(Date().timeIntervalSince1970) - before takeoff") SBTUITestTunnelServer.takeOff() print("\(Date().timeIntervalSince1970) - after takeoff")I am also running on M1
For our app on an M1 we see times in the range of 3-6s.
Thanks for that info. I would say that is also a considerable amount of time. I think a reasonable amount of time for a big UITests is 30 seconds: 6 seconds delay is 20% of the total test execution time
The takeOff() method is taking around 1.5s.
I use the completion block to setup network mocks like follows:
app.launchTunnel {
// Here I add network stubs
}
I can confirm that removing all stubbing from the completion block results in a takeOff duration of around 1.5s. So the delay is caused by the stubbing
Anything in the launchTunnel closure will synchronously block the takeOff method. Can you share the launch block that is causing the unusually long delay? We have pretty elaborated launch setup blocks and are not experiencing this problem
Can you share the launch block that is causing the unusually long delay?
It is not one specific, I see it in many of the blocks in my tests. Sorry cant share, but can explain a bit more.
The amount of stubs inside the block is around 150, and are encoding Encodable structs into data that gets passed to the stubRequests method:
app.stubRequests(
matching: SBTRequestMatch(url: "/myurl*", method: "GET"),
response: SBTStubResponse(encodable: encodableStruct, returnCode: 200)
)I already made sure that the encoding is not the part taking time (I removed the encoding code and passed Data() to SBTStubResponse). I also checked that is not one stub taking all the time, but instead is the accumulation of them.
Since I already checked that the delay is still there when passing Data() to SBTStubResponse, seems like the amount of data being transferred is not the problem.
I could confirm that the amount of stubs is the root cause of the failure. Explanation here: #145
Does this happen both with and without using IPC?
I am trying to check, but at the moment I cannot launch the app with IPC and am getting Shutting down with error: '[SBTUITestTunnel] Failed getting IPC proxy, Error Domain=DTXIPCErrorDomain Code=1 "Connection <DTXIPCConnection: 0x600002922d60> is invalid." UserInfo={NSLocalizedDescription=Connection <DTXIPCConnection: 0x600002922d60> is invalid.}' with or without zscaler đŸ˜¢
I noticed though that when that happens, the app closes down without leaving a crash report or any message in the xcresult (only leaves a message in the test log). When running in CI, I just see the app closing, and without any message I can use for debugging đŸ’€ Maybe crashing is a better option đŸ¤”
Are you testing on physical devices or simulators?
Simulator :)