support test *sets*
vcunat opened this issue · comments
When you specify an attrset containing tests instead of concrete tests (possibly by mistake), you only get a confusing error message and it succeeds (!):
warning: ignoring the user-specified setting 'restrict-eval', because it is a restricted setting and you are not a trusted user
Perhaps it wouldn't be too hard to run all the tests in the set. From tests already merged usage could be e.g. @ofborg test boot
This appears to be because borg runs tests with nix-build ./nixos/release.nix tests.<name> -- if you try that yourself with boot (or hostname), you'll notice it also succeeds immediately. (That warning isn't an error; it appears in all of the logs I've ever seen.) Running tests with nix-build ./default.nix -A nixosTests.hostname runs all the checks, as you'd expect.
I've got a patch that would switch borg to nixosTests, but I'm gonna let it cook a little to make sure I'm not missing anything.
I've also been tripped up by this: it's pretty disconcerting to get a tick when it actually ran nothing
I just ran into this (@ofborg test networking), but didn't even see an error. The log is just a confusing 404 html page.
@cole-h do you still have that patch handy?