Train without computing benchmarks before N-th iteration

[Stéphane Martin]

During a training, I would like to start computing benchmarks only after several iterations were executed.
Even potentially not computing benchmarks at all, for some experiments.
I am not quite sure how to modify AlphaZero.jl without breaking anything.
Would you have some guidance, please?
NB. Setting benchmark=[] in params.jl doesn't work:

ERROR: LoadError: MethodError: no method matching (Vector{<:AlphaZero.Benchmark.Evaluation})(::Vector{Any})
Stacktrace:
  [1] convert(#unused#::Type{Vector{<:AlphaZero.Benchmark.Evaluation}}, a::Vector{Any})
    @ Base .\array.jl:617
  [2] AlphaZero.Experiments.Experiment(name::String, gspec::AlphaZero.Examples.BonbonRectangle.GameSpec, params::AlphaZero.Params, mknet::Type, netparams::AlphaZero.FluxLib.ResNetHP, benchmark::Vector{Any})
    @ AlphaZero.Experiments C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\experiments.jl:27
  [3] top-level scope
    @ C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\games\bonbon-rectangle\params.jl:115
  [4] include(mod::Module, _path::String)
    @ Base .\Base.jl:419
  [5] include(x::String)
    @ AlphaZero.Examples.BonbonRectangle.Training C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\games\bonbon-rectangle\main.jl:4
  [6] top-level scope
    @ C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\games\bonbon-rectangle\main.jl:7
  [7] include(mod::Module, _path::String)
    @ Base .\Base.jl:419
  [8] include(x::String)
    @ AlphaZero.Examples C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\examples.jl:1
  [9] top-level scope
    @ C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\examples.jl:5
 [10] include(mod::Module, _path::String)
    @ Base .\Base.jl:419
 [11] include(x::String)
    @ AlphaZero C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\AlphaZero.jl:6
 [12] top-level scope
    @ C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\AlphaZero.jl:172
 [13] include
    @ .\Base.jl:419 [inlined]
 [14] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base .\loading.jl:1554
 [15] top-level scope
    @ stdin:1

[gwario]

You might be able to do it by overriding the following function in src/ui/session.jl:

function Handlers.iteration_finished(session::Session, report)
  bench = run_benchmark(session)
  save_increment!(session, bench, report)
  flush(Log.logfile(session.logger))
end

Session should contain the current iteration number

[Stéphane Martin]

Thank you!
I commented the 2 first instructions of this function (as well as the same instructions in function zeroth_iteration!(session::Session)) and it works, at least to skip all benchmarks.

[gwario]

Thank you! I commented the 2 first instructions of this function (as well as the same instructions in function zeroth_iteration!(session::Session)) and it works, at least to skip all benchmarks.

Just be careful with save_increment, you still want to be able to continue the session if you interrupted the script or something... so you probably need some of the code in save_increment...

[Stéphane Martin]

Oh you are right, thank you again! I amended my changes, reverting function Handlers.iteration_finished(session::Session, report) and function zeroth_iteration!(session::Session) back to their original state, but changing function run_benchmark(session::Session) as below and also allowing benchmark to equal nothing, in src/ui/session.jl and src/ui/experiments.jl:

function run_benchmark(session::Session)
  report = Report.Benchmark()
  if !isnothing(session.benchmark)
    for duel in session.benchmark
      outcome = run_duel(session.env, duel, logger=session.logger)
      push!(report, outcome)
    end
  end
  return report
end