Support for CI (reruns) with waveform output
oscargus opened this issue · comments
The new open source waveform viewer Surfer can be executed in the browser: https://app.surfer-project.org/ There is a bit of ongoing integration in #1002 (basically supporting arbitrary viewers).
A nice feature when running in CI (and maybe even in the local terminal) is to generate a link that opens the output from a simulation run in the web-based version of Surfer. Basically, just click the link in the log file and the trace will open in Surfer.
However, as storing the waveforms may require quite a bit of storage, one may also only want to do this for failing runs, then requiring a rerun of the failing tests (which may be a bit of a challenge if there are randomized components, but apart from that is straightforward). One may also consider storing the waveform and if the test passes, one removes it.
Opening this issue to possibly get a bit of feedback how it should be implemented etc.
(One may also consider adding something similar for enhanced terminals: generate a clickable link for each test which opens up the waveform in either Surfer or GTKWave locally. But maybe not as useful as the above.)
Maybe parts of this also affects https://github.com/VUnit/vunit_action (it would maybe be nice to configure this from the action?), but still there should be basic support for it in VUnit itself.
Storage is not critical, because CI environments are removed after execution, so we can generate them unconditionally in CI and then decide explicitly whether we want to keep the waveforms. In fact, it'd be interesting to save some of the executions of the examples and have them linked from the docs, so that users can see the sources and the waveform without having to execute the examples themselves.
Locally, users decide whether to use --gui.
However, I'm not sure about how does Surfer get the data/waveform. Does it get a URL and it then fetches the content? If that is the case:
- In CI, Surfer will not be able to access the artifacts, because GitHub requires login. So, apart from the URL to the VCD/GHW/FST file, it will need a token.
- Locally, as far as I am aware, browsers cannot access the local filesystem unless the user explicitly loads a file, or unless the JS app is executed on Electron rather than a regular browser.
I need better understanding to suggest whether to implement this in VUnit itself, in the action, in the test files of VUnit (essentially pytest) or as some example code block in the docs.
Locally, I was primarily referring to starting the native version of Surfer/GTKWave. I checked with my PhD student for a bit more details and he has, as part of a build tool that he is developing, set things up so that there is a special xdg-mime-handler for swim://-links (which installs itself if possible(?)). This is the code: https://gitlab.com/spade-lang/swim/-/blob/master/src/links.rs?ref_type=heads and this is how to encode the "links" https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda
Didn't know that artifacts required login, but then, probably the easiest thing is that the CI hosts the Surfer instance to get rid of the cross-referencing thing. Could probably be quite easily integrated with a suitable Docker-container, although I do not really know how GitHub handles this (we are deploying WASM-instances of Surfer for every PR/MR, so GitLab can do it).
Still a bit worried about artifact sizes (and slowing down the CI by writing to file), but then I am not really sure what the storage limitations are. Of course it is easier to just always generate it, and then, if required later, one can add more features.
Just putting this one here for later reference as it may be useful: https://github.com/marketplace/actions/deploy-pr-preview