wait-on command missing in Orb version 3
Phonesis opened this issue · comments
The wait-on command seems to have been removed from version 3 of the orb. This was very useful in previous versions as you could wait for the localhost instance of a test page to be ready and responsive before proceeding.
What is the suggested way to do this now using the orb?
It seems odd this was removed as it is vital... I am having to resort to using start-server-and-test now using my own custom NPM script. It defeats the purpose of using the orb as it previously had these features baked in.
Tests no longer wait for localhost to be ready so time out.
Can this be re added? What does the orb do differently on start?
@Phonesis, I'm going to forward this request to add back the wait-on functionality to our product team for discussion.
Ok great thanks
@mschile Any updates on this one yet? I have done more playing around and it seems to render the usage of the orb obsolete without this feature for any team using Cypress to test localhost instances.
The only option is to use start-server-and-test. The problem with that though is it only works if you include your test command as a param for the start-server function call. So in other words, it renders the orb completely obsolete because you end up calling scripts from package.json which contain every cypress specific param such as browser name etc
As well as this, if you run tests in parallel, you will be calling the same start-server-and-test script for each sharded machine. So you are in effect spinning the app up many times when before it was only required once in a pre-script hook
Without wait-on, v3 of the orb is basically a non starter.
- I also noted in cypress-io/cypress-realworld-app#1341 that the Cypress Real-World App (RWA) has become flaky, and the prime suspect is the removal of the wait-on feature.
It is not enough that Cypress waits for the webserver. In the RWA there is a database server which needs to fire up. Cypress does not know about this and carries on regardless, with varying levels of success. 🤔
I am also seeing chrome crash now, which we never used to. I have a feeling the two issues are linked. Tests are not waiting for the site to get served and become available and this is somehow causing chrome renderer issues. Perhaps because the machines are trying to build the app at the same time as running the tests.
I'm going to forward this request to add back the
wait-onfunctionality to our product team for discussion.
It looks like this issue is a regression bug rather than a pure feature request, since apparently the version changes have broken something which used to work. If that categorization is correct, does it change the way it will be handled by your product team?
@Phonesis, I don't have any updates at this time but this request is in their queue to look at.
@MikeMcC399, it wouldn't be a bug per se, but it would definitely influence our decision to add back the functionality.
Hello @mschile
I have extensively tested this now and it is clear to me that the orb in its current state is basically unusable if you are spinning up a localhost instance of a site to run Cypress tests on.
wait-on is vital. I have tried using start-server-and-test but it seems to cause browsers to crash so it seems incompatible with the orb.
I have also tried using wait-on via a package.json script but that does not seem to work either.
wait-on worked perfectly in version 2.2.0 of the orb and ensured tests did not get triggered for any parallel machine until the site was ready.
Hi @Phonesis, I'm surprised using the wait-on package did not work for you. That is the package we used in v2 of the orb and simply did npx wait-on <param>. Adding a wait-on to the package.json script called from the cypress-command should've done the same thing.
I created a test project that delays the server start-up by 10 seconds and runs a simple Cypress test against it. Here is the successful run using v3 of the CircleCi orb. You can see it waiting for the server to start and then running the test. We may need to investigate your configuration further.
@mschile can you provide a working example where you are using v3 of the orb, wait-on via a package.json script, and localhost?
Every variant of config I tried failed to work and cypress tests executed before the site was served
Also, in the example you have provided above I have noticed you are calling wait-on in the cypress-command within the orb as opposed to the start-command? I am calling wait on within the start-command
@Phonesis, here is the test project I created to demonstrate using the wait-on package:
v3 Orb
wait-on via package.json script called from cypress-command
localhost server
Successful CircleCI run that shows waiting
I am calling wait on within the start-command
You would want to wait in the cypress-command since you want to wait before running the actual Cypress tests. Waiting in the start-command wouldn't do anything. Sounds like that is where your issue lies. Let me know if changing it to the cypress-command fixes your issue.
hi @mschile thanks for the example
So implementing wait-on in the way you have demonstrated works. So good news there.
I guess there are two takeaways here. I think you need to update the orb documentation with an example of how to use wait-on with the latest version of the orb because it isn't clear.
Also, I think it is clear start-server-and-test is incompatible with the orb because you could call that in the start-command and it doesn't seem to work properly (you get browser crashes / mem leaks perhaps)
I still maintain it would be cleaner for you to re add the wait-on orb param though. It was a nice feature and made code more readable in the config.