Log best move and score changes at specific evaluation thresholds?

[kinchahoy]

I'm trying to use Katago with a GPU to find very interesting / clever moves and games played by pros (or even Katago).

One approach is to search through modern games with Katago and highlight games that skew closest to playing Katago's recommended move for as long as possible (KaTrain does something like this, and I don't think this is too hard to do).

Another thing I'd love to do is find situations where Katago's evaluation / recommendations change a lot between pure evaluation (i.e. 1 visit), 10 visits, 100 visits, 1K visits, 10K visits. The idea is to find positions or moves where Katago with 1K visits evaluates the position / best move much differently than Katago with 10K visits (I suspect there are relatively few situations where a human pro has played this move). I'm looking for big swings in evaluation not +-0.2 evaluation shifts.

Is there a way to log this sort of interim evaluation from Katago?

[David J Wu]

That's neat! If you come up with interesting results to post/publish somewhere I'd definitely enjoy reading about it.

So as for how to do it - how about just doing separate queries for each one? Make sure you have very few or only 1 search thread per analysis thread (multithreaded search will greatly harm tiny visit searches like 10 visits). Compensate for having no search parallelism by having many analysis threads doing a whole game, or whole ranges of a game at once, to get better batching. For example, analyze the whole game at 10 visits, then 100, and so on.

Because you're suggesting to use exponentially spaced visits, even if there were no caching, the cost of doing 10k + 1k + 100 + 10 + 1 = 11111 visits is only barely more expensive than just doing 10k visits. And in reality, there is caching of neural net evaluations of the same position, so the increase in cost for doing all the queries will be even less than that.