Speed up test warmup time when large imports are being used in tests; reduce the time for "Your Tests Are Starting..."
muratkeremozcan opened this issue · comments
What would you like?
Can cypress have a mini internal build step for its support file and plugins (config) file. They don’t change often and Cypress could bundle and tree shake them and have one internal file to load until the next time the file changes.
While tuning up a front end app tests, I realized that large files (ex: a network helpers with many cy.intercepts) that import from other large files (ex: a test utils package that can generate many kinds of domain objects) that import from other large files (ex: faker with all locales vs faker with english only) significantly increase test warmup time (Your Tests Are Loading…). We are talking 5-10 seconds before a test starts.
This is a side issue, related to inefficient usage of Cypress test plugins, but together they are what slow down e2e test start up time by 10-20 seconds in our front-end apps.
Mind that if these large imports occur in files that run before any tests; commands.ts, e2e.ts, component.tsx, they will slow down each test.
Currently I have fine tuned the plugin imports, and I'm tackling the large files with inefficient imports
Even when we do our best though, and decrease the test warm up from 10-20 to 5-10, we will still hit a bottle neck at some point and can't optimize further.
Why is this needed?
Saving 5-10 seconds in each test from "Your Tests Are Starting..." would be a significant CI cost savings, and huge improvement for the developer experience in any test .
Other
Mind that that time does not count towards Cypress test duration, but it's real when you run a test locally, and it's real CI cost.
I have some questions and thoughts.
Are we talking E2E or CT (or both)? They both work a bit differently, even when using webpack for both. Also, is this primarily motivated as an optimization for local usage (open mode) CI usage (run mode)?
Since CT is using a dev server, the common files (eg supportFile) should only be compiled once, at the start. Webpack Dev Server (WDS) keeps a cache of modules, it should be able to reuse one that has already been compiled. This means you should only see "Your Tests Are Starting..." once, for the first test run (well, it should only be taking a few seconds the first time - when changing to other specs, it should only need to bundle the spec file, not re-bundle the supportFile.
pluginsFile is only executed once, but that could be adding to the slowness - the double whammy of compiling a complex supportFile on top of executing a large pluginsFile.
E2E uses webpack-preprocessor, which is still webpack, but does not use a dev server - each spec is compiled "just in time" when it runs. I am not sure what this does with supportFile, but I think we have some caching logic.
I think the first thing we should do to improve performance here is find out what exactly is the bottleneck. I have done some exploration here, one thing not mentioned here is the cost of parsing large JS files. See this comment. Another idea might be not using a dev-server on CI at all, and just compiling everything and serving static files via regular HTTP. Startup time would actually be slower, but overall execution time might be faster.
It can impact both e2e or CT. For our setup, E2e is the pain point. CT could initially start faster, but after the initial
start, switching betweetn tests is fast as you have described.
Let me see what I can do about giving you access to the repository. Since we have an NDA, this should be possible.