e2e tests fail if external license website is unreachable
jotaen4tinypilot opened this issue · comments
We have started to automatically check that license links in the <about-dialog> are valid, by resolving the link and checking the response status.
For license links that forward to an external website, this means we are depending on whether that external website is online. If an external license page is down, our e2e CI job fails and can block PRs.
For example, at the time of that writing, the git server of libwebsockets appears to be down, i.e. everything under https://libwebsockets.org/git/ is unresponsive. The URL path itself still seems valid, as it’s used for links on the libwebsockets homepage (which is up). Any e2e CI job currently fails after a 30s timeout.
Solutions
Ideas:
- (a) We just accept that this happens from time to time.
- In this case, we could consider making the e2e test so that it fails immediately, without 30s timeout, and outputs clear logging to explain this corner-case.
- (b) We stop following redirects when resolving the license links.
- This, however, can produce false positives, because if e.g. a remote address is moved, we wouldn’t notice.
- (c) We don’t make use of external redirects anymore, but manually copy and distribute the license copy as local file.
- We would have to repeat this process whenever we bump a dependency version, however.
We currently rely on 10 external license URLs (in Pro).
Another aspect to consider is whether we strictly comply with legal requirements by just linking to external websites for the licenses, as we can’t ensure that they will be reachable at all times forever.
I think if it's down for more than a day, we switch to the Github version of the license:
https://github.com/warmcat/libwebsockets/blob/v3.2.0/LICENSE
And in general, I'm fine with sticking with checking every link unless we find that this comes up more than once per year or so.
Update: libwebsocket’s git server is up again, so our e2e tests have recovered for the time being.