Shortly run SQLancer on the individual DBMS as part of the CI
mrigger opened this issue · comments
It would be desirable to add smoke testing to the CI, to verify that the testing implementation for one of the DBMS is not obviously broken.
This seems to be more tricky than initially thought. For DuckDB and SQLite, running SQLancer might cause the JVM to crash when a memory error is triggered (see #1). Also for the other DBMS, it might be that bugs are still triggered on their latest version, either since a new bug was introduced or because an old one has not yet been fixed. An option would be to disable checking whether SQLancer finds a bug, but then we would also not detect any regression bugs in SQLancer.
I think the optimal setup would be if the DBMS under test would, for each commit to it, pull the latest (release) version of SQLancer and run it for a while (e.g., 10 minutes), to prevent that any new bugs are introduced to the DBMS. SQLancer could, in turn, test the latest (release) version of the DBMS to check that SQLancer is not obviously broken.
Closing this, since SQLancer now runs tests for the DBMS it supports.