Cockroach tests are pinned to historic version
mhutchinson opened this issue · comments
See #2971 for history. The cockroach tests run as part of the full trillian test suite kicked off during the GCB tests. When running, these tests secretly downloaded the latest tagged version of cockroach in the background and started it up. This breaks hermetic builds, which means that a test that passes today can break tomorrow because some external factor has changed. This is considered an anti-pattern, and thus #2971 pins this to a specific version to a) fix the current breakage, and b) bring us a step closer to hermetic builds.
What isn't clear to me is whether this crdb tests running as part of GCB is even intentional; there are GitHub actions that run specific Cockroach tests. Are these intended to be additional, or instead of testing Cockroach in the main test suite body?
If we do continue with running Cockroach tests in the main GCB test suite then we should have some mechanism that allows the version it tests with not pinned to an history build as #2971 introduces. This is implemented with dependabot elsewhere; we commit a fixed version in the code (which makes it hermetic), but also regularly update it using dependabot to make sure it doesn't get stale. Can we do anything like this for CRDB?
I actually don't have a particularly strong opinion on how to run these tests. We could try to make it so they don't run as part of GCB (that actually wasn't the intention) or we could pin the version through an environment variable that dependabot could track. I'm open to suggestions.
@mhutchinson is there a possibility of switching dependabot for Renovate? I've been researching dependabot and the functionality it provides seems quite limited. I'd like to be able to set the CRDB version via an environment variable or a variable in a file, and updating versions from custom variables like that is not supported by dependabot as far as I can tell. This is something I could trivially do with Renovate though.
I'm on leave for some time, but I'll tag @AlCutter to look at this in the meantime (or assign to someone else).