share rebuilds recording nondeterminism
raboof opened this issue · comments
When using nix-build --check or nix build --rebuild, successful rebuilds will be passed to the post-build-hook, but unsuccessful rebuilds are not.
Perhaps we could define a diff-hook that shares the rebuild result in this case?
Do you mean unsuccessful rebuilds as build failed or as build gave a different output than the one in cache.nixos.org ?
builds that gave a different output than the one already on the system
I agree that having a diff-hook is a good idea.
The difficulty with the diff-hook is that if I am not mistaken the diff-hook is called only with the first output that failed reproducibility check. The post-build-hook is not called for succeeding outputs, the process is stopped as soon as nix detects one non-reproducible output.
Good point - and yes, that matches what I see.
So this means:
- matching rebuilds are under-counted (successful builds that also have a failed output are missed)
- mismatching rebuilds are under-counted (in the case that a derivation has multiple mismatching outputs, only one is reported)
While not ideal, I think those are limitations we can live with for now?
I think we can experiment and see how skewed our results are. I'll also try to see if we can try to improve things on Nix's side.