`maxrss` is skewed if `bench.zig` needs to be compiled
squeek502 opened this issue · comments
Ryan Liptak commented
I've been trying to make gotta-go-fast
more consistent, and have been testing by running the benchmarks for a single commit multiple times. I noticed that on the first run, if bench.zig
needs to be compiled, maxrss
will be much larger than it will on subsequent runs:
benchmark | maxrss run 1 | maxrss run 2 | maxrss run 3 |
---|---|---|---|
self-hosted-parser | 534872 | 95520 | 95284 |
zig-fmt | 406304 | 97664 | 97744 |
translate-c-windows-h | 408728 | 69344 | 69692 |
arena-allocator | 461844 | 95512 | 94976 |
std-rand | 461384 | 97816 | 97820 |
std-hash-map | 460512 | 96332 | 95488 |
insert-10M-int | 452404 | 221248 | 221248 |
random-distinct | 453284 | 110736 | 110728 |
random-find | 460872 | 99680 | 95232 |
ast-check-os | 404396 | 69016 | 68724 |
ast-check-AstGen | 404984 | 68940 | 69092 |
ast-check-Sema | 405624 | 69432 | 69164 |
One extremely simple but quite wasteful fix would be to add
{
const warmup_run = try execCapture(gpa, main_argv.items, .{ .cwd = bench_cwd });
defer gpa.free(warmup_run);
}
before the const main_stdout = try execCapture(...)
call.