defunctzombie / zuul

I was trying to test with Microsoft Edge in Sauce (and mocha-bdd ui, babelify transform), but while I could watch my tests passing in Sauce, Zuul would crash with:

/Users/mc/cloned/zuul/lib/SauceBrowser.js:205
                            res.filter(Boolean).forEach(function(msg) {
                                ^

TypeError: res.filter is not a function

After I patched that error by switching res.filter to lodash's filter function, I would then, upon a test assertion, get:

/Users/mc/cloned/zuul/bin/zuul:275
                assertion.frames.forEach(function(frame) {
                                 ^

TypeError: assertion.frames.forEach is not a function

I threw some debugs in SauceBrowser, and it turns out the responses for Edge (and all arrays nested in the responses) are objects with number keys rather than arrays:

  zuul:sauce:microsoftedge:20 { '0': { type: 'test', ...} } +0ms

I was able to get some success by switching Array.filter in SauceBrowser.js and Array.forEach in zuul with lodash's functions, which can handle objects and arrays (and lodash is already a project dependency).

However, even with these fixes, if there's a failing test, some of the time the suite will get stuck, both in the actual browser and in Zuul, waiting for the "test_end" for the test directly after the failing test. I'm also frequently (but not always) getting failed tests with: Error: global leak detected: msWDfn. I might be imagining things, but it seems like this causes a frozen suite more often than normal failed tests do.

That's as much progress as I was able to make. I could use some direction to continue digging. I haven't been able to reproduce this problem on my local Windows 10 VM (using a localtunnel).

Are you using tape? Maybe have a look at https://github.com/defunctzombie/zuul/blob/master/frameworks/tape/client.js

I haven't been able to reproduce this problem on my local Windows 10 VM (using a localtunnel)

Wo that's strange.

I have been fighting a good number of issues lately with saucelabs using edge browsers (chrome dev/beta, firefox dev). Maybe that's a temporarly issue.

But the fact that it works locally is really weird because the setup should be identical.

Otherwise it may be a difference in selenium version, also try to comment this when running with sauce:

zuul/lib/SauceBrowser.js

Line 65 in 7721fd0

'appium-version': '1'

It may be triggering some weird things.

Using mocha. Been rooting around in the mocha framework and client files, and I'm thinking it could be a problem trying to do an Array.splice on a non-array, but I'm not super familiar with the code there.

Yeah the local thing was super unexpected. The failure is intermittent, so it's possible it just didn't manifest in the dozen times I tried it locally. I've been noticing complete failures with Firefox beta and dev, but for me the problem appears to be completely different (in that MS Edge at least starts the tests, and sometimes finishes them).

No luck commenting out appium-version. Microsoft's WebDriver support for Edge looks pretty recent, so I'm sure that's not helping things.

Just had the exact same issue, did you solve it somehow?

I had limited success by replacing some native array method calls with lodash functions (I can open a PR with those changes if you'd like).

After that, though, I could not diagnose the client-side frozen tests and msWDfn global leaks. I just updated my Bootcamp to Windows 10 and installed the MS WebDriver server, so my current plan is:

Add msWDfn to Mocha's acceptable globals and see if that helps
Connect Zuul to my local WD server instead of Sauce and try to find where the freezing problem is happening

Don't know when I'll be able to get to that, though. Did you encounter the res.filter is not a function error, the frozen test error, or both? And were you using Tape or Mocha?

Yes I encountered the res.filter is not a function .. with tape!

Ok so I do think that somehow the microsoft edge selenium driver is not sending data back to the client like others, Maybe it's sending an "array like". For now I just do not test it until I have time investigatin myself..

I encountered this a few months ago but forgot to open an issue/send a PR. The issue is that there's only one version of Edge on Sauce Labs at the moment, so the API returns an object instead of an array of objects to describe the browser versions.

zuul should check that the response is an array and, if not, make it a single-element array.

Does it make sense to add globals: [ 'msWDfn' ] here?
I've been hit by this issue and I wonder if the above workaround will fix it.

Woah @mcous can you confirm?

👍

Hey sorry haven't had time to get back to this since #242 was merged. The only thing that confuses me is that the global leak always happened inconsistently for me, where I'd expect it to happen every time. It clearly looks to be a global added by Microsoft's WebDriver, so whitelisting it can't hurt as long as it's appearance isn't indicative of some sort of problem.

The issue is that there's only one version of Edge on Sauce Labs at the moment, so the API returns an object instead of an array of objects to describe the browser versions.

What about this @mcous ?

I've never seen any problems when asking for Edge browser versions, just the actual test results. Zuul has been able to start a Edge browser on Sauce whenever I've tried, and it seems like #242 made it so Zuul can handle Edge's array-like test results.

Running some (Mocha) tests right now, and I still get occasional "msWDfn" global leak (ran a ~100 test suite 4 times, got 3 suite passes and 1 suite with one failed test because of the global leak). This global leak seems to be the only issue left with Edge on Zuul.

Starting an Edge browsers works for me now, so the issue I referenced doesn't exist anymore.

I get the msWDfn global leak almost every time when I run this test suite unshiftio/recovery#4.
With other projects I only get it occasionally.

I ran into the msWDfn in my unit tests (not using zuul).
Just thought I'd pop in to say this is the first time I've hit the issue.

I'm eagerly waiting for a new zuul release to work around the msWDfn issue.

I'm using QUnit so I just leak the variable early root.msWDfn = undefined.
I did something similar when an older version of Node use to leak a variable.

released 3.9.0 which will ignore msWDfn global with mocha, dunno for qunit thought.

this issue is not relevant to this problem also, closing.

Cannot test Microsoft Edge on Sauce (response is an object instead of an array)