Hook that fires after each cycle
froehlichA opened this issue · comments
I'd really like a hook cycle that is executed after each run cycle, supplied in the constructor just like the setup & teardown hooks. Some benchmarking code needs to reset state after the benchmarking test function itself is called.
const bench = new Bench({
cycle: () => {
[...]
}
});I can implement this myself if you give me the go-ahead.
Sure, I'd like you to implement it, but the thing is where it should be used. And could you give a strong example? The reason I'm asking this is that I want to put the minimum amount of features into tinylibs so they don't get out of the Minimal stage.
But from my side, I love this feature. We just need to discuss it more.
I need this feature because I'm benchmarking a database (more specifically, BlinkDB) where state must be reset after an insertMany, else the operation fails because of clashing primary keys.
In order to prevent duplicate primary keys, current code has to look like this:
...
let users: User[] = ...;
const bench = new Bench()
.add("lokijs", () => {
const newUsers = users.map(u => ({ ...u, id: 10000 * lokiIndex + u.id }));
lokiIndex++;
lokiUserTable.insert(newUsers);
})
.add("blinkdb", async () => {
const newUsers = users.map(u => ({ ...u, id: 10000 * blinkIndex + u.id }));
blinkIndex++;
await insertMany(blinkUserTable, newUsers);
});This is brittle, obviously skews the results, and doesn't look all that great. (There are other ways to do it, but afaik no way that doesn't include code I don't want to benchmark in the task.)
It'd be much cleaner with a simple cycle hook:
...
let users: User[] = ...;
const bench = new Bench({
cycle: async () => {
lokiUserTable.clear();
await clear(blinkUserTable);
}
})
.add("lokijs", () => {
lokiUserTable.insert(users);
})
.add("blinkdb", async () => {
await insertMany(blinkUserTable, users);
});In my case, it doesn't matter - But I'd prefer after each cycle. Generally speaking, if you have code that specifically needs to run before the first task cycle, you can just call it before starting the benchmark. Also, Benchmark.js handles it similarly.
@froehlichA See #33, it's not complete and i'll review it later. but we can have:
beforeAll: which fallbacks to setup
afterAll: which fallbacks to teardown
beforeEach: like vitest/jest beforeEach, which runs before each time a case starts
afterEach: like `cycle`, but with different name, and similar to beforeEach, but runs after each case
cc @achingbrain
Landed in #33, Let me know your opinion.