iphone/ipad Sauce Labs tests stall forever
feross opened this issue · comments
Using the latest version of zuul (3.9.0), remote Sauce Labs tests for iPhone and iPad stall forever locally, even though they're actually passing on the remote browser. This just started happening recently for me.
t.js:
var test = require('tape')
test('hi', function (t) {
t.ok('hi there')
t.end()
}).zuul.yml:
ui: tape
browsers:
- name: iphone
version: latest
- name: ipad
version: latestRunning: zuul -- t.js just stalls forever:
- testing: iphone @ Mac 10.10: 9.2
- testing: ipad @ Mac 10.10: 9.2
- queuing: <iphone 9.2 on Mac 10.10>
- queuing: <ipad 9.2 on Mac 10.10>
- starting: <iphone 9.2 on Mac 10.10>
- starting: <ipad 9.2 on Mac 10.10>
- waiting: <iphone 9.2 on Mac 10.10>
- waiting: <ipad 9.2 on Mac 10.10>
- waiting: <iphone 9.2 on Mac 10.10>
- waiting: <ipad 9.2 on Mac 10.10>
- waiting: <iphone 9.2 on Mac 10.10>
- waiting: <ipad 9.2 on Mac 10.10>
- waiting: <iphone 9.2 on Mac 10.10>
- waiting: <ipad 9.2 on Mac 10.10>
- restarting: <ipad 9.2 on Mac 10.10>
- waiting: <iphone 9.2 on Mac 10.10>
...
Even though the tests actually pass on Sauce Labs, as this screenshot shows:
Here's an example from a real-world build if you want a full log.
Any ideas?
I should also mention that zuul --local -- t.js with the local iOS simulator works just fine for me, so it's something about the remote Sauce iOS browsers that must have changed.
Sauce just started using Appium 1.5 (according to this post) so maybe that's the root cause?
I have what looks like a similar issue, but notably it works fine for iOS <=6 and fails for iOS >= 7.
Here's a passing test on iOS 6.1: https://saucelabs.com/beta/tests/6296015c9aab4dbe824a60681becba70/logs
And here's the same test failing on iOS 7.1: https://saucelabs.com/beta/tests/4018b9e8c8a642b5bfe03f0a920b3b64/logs
If you look at the logs tab, the successful logs are selenium output and the failing ones are from appium. I'm not sure whether this is something zuul or sauce labs is doing.
I ran into this in my repos as well (nolanlawson/fruitdown#5). Noticed it about 2 weeks ago; downgrading to Zuul 3.5.0 fixes the problem.
The error I'm seeing from Zuul is:
Error: Cannot find module 'node_modules/is-buffer/index.js' from '/Users/nolan/workspace/fruitdown'
at /Users/nolan/workspace/fruitdown/node_modules/resolve/lib/async.js:46:17
at process (/Users/nolan/workspace/fruitdown/node_modules/resolve/lib/async.js:173:43)
at ondir (/Users/nolan/workspace/fruitdown/node_modules/resolve/lib/async.js:188:17)
at load (/Users/nolan/workspace/fruitdown/node_modules/resolve/lib/async.js:69:43)
at onex (/Users/nolan/workspace/fruitdown/node_modules/resolve/lib/async.js:92:31)
at /Users/nolan/workspace/fruitdown/node_modules/resolve/lib/async.js:22:47
at FSReqWrap.oncomplete (fs.js:82:15)
...but I don't know if it's related.
Downgrading to 3.8.0 also works. I can confirm this bug was introduced in 3.9.0.
Sorry, I'm wrong. 3.8.0 is failing.
Whoops nevermind, completely misdiagnosed it. 3.5.0 is also failing. Seems to be a Saucelabs issue, because that same version of Zuul was fine a few months ago.
Could be an issue of the small code taking care of knowing when a tape test is really finished. https://github.com/defunctzombie/zuul/blob/279f1a1c13f15e5ba107868eb7aac96e0b0476c5/frameworks/tape/client.js
@feross Also have a look at the saucelab selenium log. It may be some commands were failing.
I'm seeing this issue both on repos of mine that use tape (nolanlawson/fruitdown#5) and repos that use mocha (nolanlawson/promise-worker#2 nolanlawson/blob-util#23). /cc @filmaj
@jhiesey appium shows this in the log:
2016-03-02 00:55:27:851 - [MJSONWP] Encountered internal error running command: Error: Tried to use an iOS simulator with xcode version 5.1.1 but only Xcode version 6.0.0 and up are supported
My guess is, since Zuul specifies an appium version of just "1" when it builds up Selenium desired capabilities to POST to Sauce, Sauce assigns it the "latest" version of appium 1.x. Recently, Appium 1.5 was released. Perhaps this is related, as per @feross' point. I have Appium locally, so I can try to reproduce on my mac w/ different versions of Appium using the test cases shown here.
One thing that you might be able to try to at least isolate the Appium version bug is to specify an appium-version of 1.4.16 in Zuul - the last stable version before Appium 1.5. 1.5 was a pretty big rewrite.
Yes I purposely set appium "1" to avoid having to follow the appium releases on saucelabs and benefit from changes automatically. Maybe not the best idea (was a comment from @sebv)
One more note: the capabilities format used by Appium is different than by the desktop-browser "older" Selenium servers. In particular, Appium 1.0+ uses capabilities like platformName and platformVersion. On Sauce's end, if we detect a capability set that includes platformName, we assume it to be an Appium-specific (iOS or Android) test. Otherwise, we assume it to be an old-school desktop-specific Selenium server. We might need additional logic in the sauce-specific bits of the test runner.
There are some confusing edge cases between appium and selenium... I'm happy to help nail this down!
Any pointers on how I could test some of the Zuul interactions locally against my own setup of Appium? Just running npm test w/ latest zuul fails most of the Sauce tests, and hangs indefinitely 😬
I think I found it.. gonna replace talking to Sauce's selenium endpoint (ondemand.saucelabs.com) and make it point to my local appium instance running on localhost. I'll keep digging.
@filmaj thanks for looking into this!
I've got a branch of zuul on my fork that I'm hacking around with to test different mobile environments locally with a local instance of appium. https://github.com/filmaj/zuul/tree/local_appium in case anyone's interested.
I can reproduce the iOS-simulator-spins-forever issue locally w/ appium 1.5.0. When I move appium to 1.4.16 locally, I cannot reproduce the issue.
Suggestion for a quick workaround: explicitly change the appium version used in the SauceBrowser class from "1" to "1.4.16"?
In case y'all are curious, appium-boneyard/appium-ios-driver#108 seems to be the root of the problem.
@filmaj Nice debugging work. Any ETA on when that PR will be merged?
One more thing I highly recommend: we add some more sophistication to SauceBrowser.js to properly set up the mobile desired caps to adhere to Appium's requirements (i.e. platformName instead of platform, platformVersion instead of version, adding a specification for deviceName, using iOS or Android as values for platformName). My branch has the basic changes in SauceBrowser laid out - although that would need more work to catch the edge cases.
The benefits to doing so is a bit nuanced. Using the old-style, desktop-like capabilities gets you something like a browser in some cases - for example, on iOS 6.1, you are actually running your web tests in a WebView inside a native application on iOS; you are not running in Mobile Safari on iOS. Switching to the new-style caps gets the tests running in, likely, a more representative/realistic environment than a WebView-inside-a-native-app. The same thing happens in Android: using old-style capabilities runs your tests in a WebView inside a native app. By changing the Android capabilities to:
{
"platformName":"Android",
"platformVersion":"4.4",
"browserName":"Browser",
"deviceName":"Android Emulator"
}
... you'll get the tests running in the stock Android browser, rather than the WebView thing. The devil is in the details, though: the stock Android browser is not automatable on Android 4.3 or lower.
Perhaps a discussion / pull request / issue separate from this one, but I wanted to highlight it.
@feross I'm not sure about the ETA on the pull req. @imurchie might have more information. Note, though, that even once that PR lands, it will necessitate a new release of Appium, so we'll have to wait until Appium 1.5.1, at a minimum, I think.
A quicker workaround would be to specify Appium 1.4.16 in the SauceBrowser.js class in zuul, as per my comment above. I tried doing that on my fork of zuul but most other tests were failing :(
I will try that right now thx @filmaj
see #272
forcing appium to 1.4 was released as 3.10.0, let me know
3.10.0 is passing in all SauceLabs browsers for me now. Thanks a ton!
Small nit: it looks like 3.10.0 requires phantomjs-prebuilt instead of phantomjs. In my mind that's a breaking change, but I can see how it might not be considered breaking. Also: not really a big deal.
Yes I was unsure phatomjs-prebuilt was a breaking change since it should in most cases bring more features than before? (Like the infamous "bind is not defined" from phantomjs 1).
If that's really an issue for someone, open a new issue, thanks!
We can close this I believe, reopen when needed.
\o/
@vvo the phantomjs 2.0.0 package actually already shipped with those fixes (like the "bind" one); it was only around 2.0.3 iirc that they deprecated in favor of phantomjs-prebuilt.
I think you could merely add a try/catch inside of zuul to fix this. I'll open a PR.
I'll open a PR.
😍
@filmaj the master branch now forces appium 1.5. This works well except for iOS 7.x:
iphone@7.0: [init({"name":"eventemitter3","tags":[],"browserName":"iphone","version":"7.0","platform":"Mac 10.9","appium-version":"1","record-screenshots":false,"record-video":false})] The environment you requested was unavailable.: Appium 1.5.2 does not support Mac 10.9. Please check our platform configurator (https://saucelabs.com/docs/platforms)
What strikes me odd is that older versions of iOS (5.x and 6.x) work fine. Is this expected?
Thanks.