ziglang / gotta-go-fast

Performance Tracking for Zig

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

`maxrss` is skewed if `bench.zig` needs to be compiled

squeek502 opened this issue · comments

I've been trying to make gotta-go-fast more consistent, and have been testing by running the benchmarks for a single commit multiple times. I noticed that on the first run, if bench.zig needs to be compiled, maxrss will be much larger than it will on subsequent runs:

benchmark maxrss run 1 maxrss run 2 maxrss run 3
self-hosted-parser 534872 95520 95284
zig-fmt 406304 97664 97744
translate-c-windows-h 408728 69344 69692
arena-allocator 461844 95512 94976
std-rand 461384 97816 97820
std-hash-map 460512 96332 95488
insert-10M-int 452404 221248 221248
random-distinct 453284 110736 110728
random-find 460872 99680 95232
ast-check-os 404396 69016 68724
ast-check-AstGen 404984 68940 69092
ast-check-Sema 405624 69432 69164

One extremely simple but quite wasteful fix would be to add

{
    const warmup_run = try execCapture(gpa, main_argv.items, .{ .cwd = bench_cwd });
    defer gpa.free(warmup_run);
}

before the const main_stdout = try execCapture(...) call.