Stack overflow with lots of RAM still available

Question

Stack overflow with lots of RAM still available

smart-fr opened this issue a year ago · comments

Hello,

Here's the error message and the stacktrace I get when I dummy_run my game.
When this occurs, neither the RAM of my PC nor the RAM of the GPU are fully used, even far from this.
Moreover, I have reduced my game's dimensions and, consequently, model, so much that I doubt the game tree complexity should be of an issue.

Are there some settings that I should check in order to make sure Julia isn't artificially limiting its access to the hardware resources?

[ Info: Using the Flux implementation of AlphaZero.NetLib.

Initializing a new AlphaZero environment

Initial report
Number of network parameters: 1,627,841
Number of regularized network parameters: 1,623,168
Memory footprint per MCTS node: 3242 bytes
Running benchmark: AlphaZero against MCTS (2 rollouts)
Average reward: -1.00 (0% won, 0% draw, 100% lost), redundancy: 0.6%
Running benchmark: Network Only against MCTS (2 rollouts)
Average reward: +1.00 (100% won, 0% draw, 0% lost), redundancy: 0.2%
Starting iteration 1

Starting self-play

StackOverflowError:
Stacktrace:
[1] Array
@ .\boot.jl:459 [inlined]
[2] Array
@ .\boot.jl:478 [inlined]
[3] vect
@ .\array.jl:125 [inlined]
[4] compute_actions_masks_actions_hooks(board::StaticArraysCore.SMatrix{4, 4, UInt8, 16})
@ AlphaZero.Examples.BonbonRectangle C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\games\bonbon-rectangle\game.jl:467
[5] update_status!(g::AlphaZero.Examples.BonbonRectangle.GameEnv)
@ AlphaZero.Examples.BonbonRectangle C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\games\bonbon-rectangle\game.jl:493
[6] play!(g::AlphaZero.Examples.BonbonRectangle.GameEnv, action::UInt16)
@ AlphaZero.Examples.BonbonRectangle C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\games\bonbon-rectangle\game.jl:972
[7] run_simulation!(env::AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.Batchifier.BatchedOracle{AlphaZero.Batchifier.var"#8#9"}}, game::AlphaZero.Examples.BonbonRectangle.GameEnv; η::Vector{Float64}, root::Bool)
@ AlphaZero.MCTS C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\mcts.jl:214
[8] run_simulation!(env::AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.Batchifier.BatchedOracle{AlphaZero.Batchifier.var"#8#9"}}, game::AlphaZero.Examples.BonbonRectangle.GameEnv; η::Vector{Float64}, root::Bool) (repeats 5430 times)
@ AlphaZero.MCTS C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\mcts.jl:218
[9] explore!(env::AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.Batchifier.BatchedOracle{AlphaZero.Batchifier.var"#8#9"}}, game::AlphaZero.Examples.BonbonRectangle.GameEnv, nsims::Int64)
@ AlphaZero.MCTS C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\mcts.jl:243
[10] think(p::MctsPlayer{AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.Batchifier.BatchedOracle{AlphaZero.Batchifier.var"#8#9"}}}, game::AlphaZero.Examples.BonbonRectangle.GameEnv)
@ AlphaZero C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\play.jl:198
[11] play_game(gspec::AlphaZero.Examples.BonbonRectangle.GameSpec, player::MctsPlayer{AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.Batchifier.BatchedOracle{AlphaZero.Batchifier.var"#8#9"}}}; flip_probability::Float64)
@ AlphaZero C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\play.jl:308
[12] (::AlphaZero.var"#simulate_game#70"{MctsPlayer{AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.Batchifier.BatchedOracle{AlphaZero.Batchifier.var"#8#9"}}}, AlphaZero.var"#remote_game_simulated#78"{Distributed.RemoteChannel{Channel{Nothing}}}, Simulator{AlphaZero.var"#159#164"{Env{AlphaZero.Examples.BonbonRectangle.GameSpec, ResNet, NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}}, SelfPlayParams}, AlphaZero.var"#make_oracle#163"{Env{AlphaZero.Examples.BonbonRectangle.GameSpec, ResNet, NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}}, SelfPlayParams}, typeof(AlphaZero.self_play_measurements)}, AlphaZero.Examples.BonbonRectangle.GameSpec, SimParams})(sim_id::Int64)
@ AlphaZero C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\simulations.jl:232
[13] macro expansion
@ C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\util.jl:189 [inlined]
[14] (::AlphaZero.Util.var"#9#10"{AlphaZero.var"#68#69"{AlphaZero.var"#remote_game_simulated#78"{Distributed.RemoteChannel{Channel{Nothing}}}, Simulator{AlphaZero.var"#159#164"{Env{AlphaZero.Examples.BonbonRectangle.GameSpec, ResNet, NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}}, SelfPlayParams}, AlphaZero.var"#make_oracle#163"{Env{AlphaZero.Examples.BonbonRectangle.GameSpec, ResNet, NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}}, SelfPlayParams}, typeof(AlphaZero.self_play_measurements)}, AlphaZero.Examples.BonbonRectangle.GameSpec, SimParams, AlphaZero.var"#48#49"{Channel{Any}}, AlphaZero.var"#make#65"{Channel{Any}}}, UnitRange{Int64}, typeof(vcat), ReentrantLock})()
@ AlphaZero.Util C:\Users\smart.julia\packages\ThreadPools\ANo2I\src\macros.jl:261ERROR: MethodError: no method matching zero(::Type{Any})
Closest candidates are:
zero(::Type{Union{Missing, T}}) where T at missing.jl:105
zero(::Union{Type{P}, P}) where P<:Dates.Period at C:\Program Files\Julia-1.8.4\share\julia\stdlib\v1.8\Dates\src\periods.jl:53
zero(::ColorTypes.Colorant) at C:\Users\smart.julia\packages\ColorTypes\1dGw6\src\traits.jl:477
...
Stacktrace:
[1] zero(#unused#::Type{Any})
@ Base .\missing.jl:106
[2] reduce_empty(#unused#::typeof(+), #unused#::Type{Any})
@ Base .\reduce.jl:338
[3] reduce_empty(#unused#::typeof(Base.add_sum), #unused#::Type{Any})
@ Base .\reduce.jl:347
[4] mapreduce_empty(#unused#::typeof(identity), op::Function, T::Type)
@ Base .\reduce.jl:367
[5] reduce_empty(op::Base.MappingRF{typeof(identity), typeof(Base.add_sum)}, #unused#::Type{Any})
@ Base .\reduce.jl:356
[6] reduce_empty_iter
@ .\reduce.jl:379 [inlined]
[7] mapreduce_empty_iter(f::Function, op::Function, itr::Vector{Any}, ItrEltype::Base.HasEltype)
@ Base .\reduce.jl:375
[8] _mapreduce(f::typeof(identity), op::typeof(Base.add_sum), #unused#::IndexLinear, A::Vector{Any})
@ Base .\reduce.jl:427
[9] _mapreduce_dim
@ .\reducedim.jl:365 [inlined]
[10] #mapreduce#765
@ .\reducedim.jl:357 [inlined]
[11] mapreduce
@ .\reducedim.jl:357 [inlined]
[12] #_sum#775
@ .\reducedim.jl:999 [inlined]
[13] _sum
@ .\reducedim.jl:999 [inlined]
[14] #sum#773
@ .\reducedim.jl:995 [inlined]
[15] _mean(f::typeof(identity), A::Vector{Any}, dims::Colon)
@ Statistics C:\Program Files\Julia-1.8.4\share\julia\stdlib\v1.8\Statistics\src\Statistics.jl:170
[16] #mean#2
@ C:\Program Files\Julia-1.8.4\share\julia\stdlib\v1.8\Statistics\src\Statistics.jl:164 [inlined]
[17] mean(A::Vector{Any})
@ Statistics C:\Program Files\Julia-1.8.4\share\julia\stdlib\v1.8\Statistics\src\Statistics.jl:164
[18] self_play_step!(env::Env{AlphaZero.Examples.BonbonRectangle.GameSpec, ResNet, NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}}, handler::Session{Env{AlphaZero.Examples.BonbonRectangle.GameSpec, ResNet, NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}}})
@ AlphaZero C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\training.jl:293
[19] macro expansion
@ .\timing.jl:463 [inlined]
[20] macro expansion
@ C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\report.jl:267 [inlined]
[21] train!(env::Env{AlphaZero.Examples.BonbonRectangle.GameSpec, ResNet, NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}}, handler::Session{Env{AlphaZero.Examples.BonbonRectangle.GameSpec, ResNet, NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}}})
@ AlphaZero C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\training.jl:325
[22] resume!(session::Session{Env{AlphaZero.Examples.BonbonRectangle.GameSpec, ResNet, NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}}})
@ AlphaZero.UserInterface C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\ui\session.jl:316
[23] dummy_run(experiment::Experiment; dir::Nothing, nostdout::Bool)
@ AlphaZero.Scripts C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\scripts\dummy_run.jl:72
[24] dummy_run
@ C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\scripts\dummy_run.jl:67 [inlined]
[25] #dummy_run#2
@ C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\scripts\scripts.jl:9 [inlined]
[26] dummy_run(s::String)
@ AlphaZero.Scripts C:\Projets\BonbonRectangle\IA\dev\AlphaZero.jl\src\scripts\scripts.jl:9
[27] top-level scope
@ none:1

Stéphane Martin · Answer 1 · Tue Jan 24 2023 02:17:37 GMT+0800 (China Standard Time)

Insisting a little to get knowledgeable guidance, since I pursued my testing and am a little puzzled with the results.
I built a VM on Jarvislabs.ai with 56 computing core, 256GB RAM and 8 Nvidia RTX A6000 GPU, each with 48GB memory.
I launched AlphaZero.jl/scripts/profile/mcts.jl in order to profile the MCTS for my game.
Unfortunately, the outcome is a Stack Overflow error. It comes way quicker than on my personal PC, but sadly it is the same outcome.
Given the VM characteristics, I doubt my model is so complex it incurs a shortage of resources. Where should I look for the real bottleneck?

Illustration and full feedback below.

Executing task: /home/bonbonrectangle/julia-1.8.5/bin/julia --color=yes --project=/home/.julia/environments/v1.8 /home/bonbonrectangle/dev/AlphaZero.jl/scripts/profile/mcts.jl 

ERROR: LoadError: TaskFailedException
Stacktrace:
 [1] wait
   @ ./task.jl:345 [inlined]
 [2] threading_run(fun::var"#8#threadsfor_fun#11"{var"#8#threadsfor_fun#5#12"{Int64, Experiment, Vector{MctsPlayer{AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.MCTS.RandomOracle{AlphaZero.Examples.BonbonRectangle.GameSpec}}}}}}, static::Bool)
   @ Base.Threads ./threadingconstructs.jl:38
 [3] macro expansion
   @ ./threadingconstructs.jl:89 [inlined]
 [4] macro expansion
   @ ./timing.jl:463 [inlined]
 [5] profile_mcts(exp::Experiment; batched::Bool, nrep::Int64, nworkers::Int64)
   @ Main ~/bonbonrectangle/dev/AlphaZero.jl/scripts/profile/mcts.jl:51
 [6] top-level scope
   @ ~/bonbonrectangle/dev/AlphaZero.jl/scripts/profile/mcts.jl:61

    nested task error: StackOverflowError:
    Stacktrace:
      [1] println(::Base.TTY, ::String, ::Vararg{String})
        @ Base ./strings/io.jl:75
      [2] println(xs::String)
        @ Base ./coreio.jl:4
      [3] compute_actions_masks_actions_hooks(board::StaticArraysCore.SMatrix{4, 4, UInt8, 16})
        @ AlphaZero.Examples.BonbonRectangle ~/bonbonrectangle/dev/AlphaZero.jl/games/bonbon-rectangle/game.jl:485
      [4] update_status!(g::AlphaZero.Examples.BonbonRectangle.GameEnv)
        @ AlphaZero.Examples.BonbonRectangle ~/bonbonrectangle/dev/AlphaZero.jl/games/bonbon-rectangle/game.jl:493
      [5] play!(g::AlphaZero.Examples.BonbonRectangle.GameEnv, action::UInt16)
        @ AlphaZero.Examples.BonbonRectangle ~/bonbonrectangle/dev/AlphaZero.jl/games/bonbon-rectangle/game.jl:972
      [6] run_simulation!(env::AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.MCTS.RandomOracle{AlphaZero.Examples.BonbonRectangle.GameSpec}}, game::AlphaZero.Examples.BonbonRectangle.GameEnv; η::Vector{Float64}, root::Bool)
        @ AlphaZero.MCTS ~/bonbonrectangle/dev/AlphaZero.jl/src/mcts.jl:214
      [7] run_simulation!(env::AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.MCTS.RandomOracle{AlphaZero.Examples.BonbonRectangle.GameSpec}}, game::AlphaZero.Examples.BonbonRectangle.GameEnv; η::Vector{Float64}, root::Bool) (repeats 6485 times)
        @ AlphaZero.MCTS ~/bonbonrectangle/dev/AlphaZero.jl/src/mcts.jl:218
      [8] explore!(env::AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.MCTS.RandomOracle{AlphaZero.Examples.BonbonRectangle.GameSpec}}, game::AlphaZero.Examples.BonbonRectangle.GameEnv, nsims::Int64)
        @ AlphaZero.MCTS ~/bonbonrectangle/dev/AlphaZero.jl/src/mcts.jl:243
      [9] think(p::MctsPlayer{AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.MCTS.RandomOracle{AlphaZero.Examples.BonbonRectangle.GameSpec}}}, game::AlphaZero.Examples.BonbonRectangle.GameEnv)
        @ AlphaZero ~/bonbonrectangle/dev/AlphaZero.jl/src/play.jl:198
     [10] macro expansion
        @ ~/bonbonrectangle/dev/AlphaZero.jl/scripts/profile/mcts.jl:53 [inlined]
     [11] (::var"#8#threadsfor_fun#11"{var"#8#threadsfor_fun#5#12"{Int64, Experiment, Vector{MctsPlayer{AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.MCTS.RandomOracle{AlphaZero.Examples.BonbonRectangle.GameSpec}}}}}})(tid::Int64; onethread::Bool)
        @ Main ./threadingconstructs.jl:84
     [12] #8#threadsfor_fun
        @ ./threadingconstructs.jl:51 [inlined]
     [13] (::Base.Threads.var"#1#2"{var"#8#threadsfor_fun#11"{var"#8#threadsfor_fun#5#12"{Int64, Experiment, Vector{MctsPlayer{AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.MCTS.RandomOracle{AlphaZero.Examples.BonbonRectangle.GameSpec}}}}}}, Int64})()
        @ Base.Threads ./threadingconstructs.jl:30
    
    caused by: StackOverflowError:
    Stacktrace:
      [1] compute_cell_legal_actions_update_actions_hook!(board::StaticArraysCore.SMatrix{4, 4, UInt8, 16}, column::Int64, row::Int64, actions_hook::Matrix{Tuple{UInt8, UInt8}})
        @ AlphaZero.Examples.BonbonRectangle ~/bonbonrectangle/dev/AlphaZero.jl/games/bonbon-rectangle/game.jl:374
      [2] compute_actions_masks_actions_hooks(board::StaticArraysCore.SMatrix{4, 4, UInt8, 16})
        @ AlphaZero.Examples.BonbonRectangle ~/bonbonrectangle/dev/AlphaZero.jl/games/bonbon-rectangle/game.jl:473
      [3] update_status!(g::AlphaZero.Examples.BonbonRectangle.GameEnv)
        @ AlphaZero.Examples.BonbonRectangle ~/bonbonrectangle/dev/AlphaZero.jl/games/bonbon-rectangle/game.jl:493
      [4] play!(g::AlphaZero.Examples.BonbonRectangle.GameEnv, action::UInt16)
        @ AlphaZero.Examples.BonbonRectangle ~/bonbonrectangle/dev/AlphaZero.jl/games/bonbon-rectangle/game.jl:972
      [5] run_simulation!(env::AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.MCTS.RandomOracle{AlphaZero.Examples.BonbonRectangle.GameSpec}}, game::AlphaZero.Examples.BonbonRectangle.GameEnv; η::Vector{Float64}, root::Bool)
        @ AlphaZero.MCTS ~/bonbonrectangle/dev/AlphaZero.jl/src/mcts.jl:214
      [6] run_simulation!(env::AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.MCTS.RandomOracle{AlphaZero.Examples.BonbonRectangle.GameSpec}}, game::AlphaZero.Examples.BonbonRectangle.GameEnv; η::Vector{Float64}, root::Bool) (repeats 6485 times)
        @ AlphaZero.MCTS ~/bonbonrectangle/dev/AlphaZero.jl/src/mcts.jl:218
      [7] explore!(env::AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.MCTS.RandomOracle{AlphaZero.Examples.BonbonRectangle.GameSpec}}, game::AlphaZero.Examples.BonbonRectangle.GameEnv, nsims::Int64)
        @ AlphaZero.MCTS ~/bonbonrectangle/dev/AlphaZero.jl/src/mcts.jl:243
      [8] think(p::MctsPlayer{AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.MCTS.RandomOracle{AlphaZero.Examples.BonbonRectangle.GameSpec}}}, game::AlphaZero.Examples.BonbonRectangle.GameEnv)
        @ AlphaZero ~/bonbonrectangle/dev/AlphaZero.jl/src/play.jl:198
      [9] macro expansion
        @ ~/bonbonrectangle/dev/AlphaZero.jl/scripts/profile/mcts.jl:53 [inlined]
     [10] (::var"#8#threadsfor_fun#11"{var"#8#threadsfor_fun#5#12"{Int64, Experiment, Vector{MctsPlayer{AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.MCTS.RandomOracle{AlphaZero.Examples.BonbonRectangle.GameSpec}}}}}})(tid::Int64; onethread::Bool)
        @ Main ./threadingconstructs.jl:84
     [11] #8#threadsfor_fun
        @ ./threadingconstructs.jl:51 [inlined]
     [12] (::Base.Threads.var"#1#2"{var"#8#threadsfor_fun#11"{var"#8#threadsfor_fun#5#12"{Int64, Experiment, Vector{MctsPlayer{AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer), Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}}, AlphaZero.MCTS.RandomOracle{AlphaZero.Examples.BonbonRectangle.GameSpec}}}}}}, Int64})()
        @ Base.Threads ./threadingconstructs.jl:30
in expression starting at /home/bonbonrectangle/dev/AlphaZero.jl/scripts/profile/mcts.jl:61

 *  The terminal process "/home/bonbonrectangle/julia-1.8.5/bin/julia '--color=yes', '--project=/home/.julia/environments/v1.8', '/home/bonbonrectangle/dev/AlphaZero.jl/scripts/profile/mcts.jl'" terminated with exit code: 1. 
 *  Terminal will be reused by tasks, press any key to close it.

Stéphane Martin · Answer 2 · Wed Jan 25 2023 01:52:04 GMT+0800 (China Standard Time)

Focusing on the error:

[6] run_simulation!(env::AlphaZero.MCTS.Env{NamedTuple{(:board, :actions_hook, :curplayer),
Tuple{StaticArraysCore.SMatrix{4, 4, UInt8, 16}, StaticArraysCore.SMatrix{4, 4, Tuple{UInt8, UInt8}, 16}, UInt8}},
AlphaZero.MCTS.RandomOracle{AlphaZero.Examples.BonbonRectangle.GameSpec}},
game::AlphaZero.Examples.BonbonRectangle.GameEnv; η::Vector{Float64}, root::Bool) (repeats 6485 times)

@ AlphaZero.MCTS ~/bonbonrectangle/dev/AlphaZero.jl/src/mcts.jl:218

I can see that the Stack Overflow crash occured on the 6485th simulation, meaning that several thousands of simulations have been run without problem.

Interestingly, I get the very same number running on the cheapest VM using 1 computing core and 2GB RAM, no GPU at all... On my personal PC, this number was 5430 (see the first Stacktrace of this post).
So, definitely, there's no hardware resource problem here.

The code of my game.jl interface should be optimized. On the (few thousands)-th simulation, the play! function throws a Stack Overflow exception, while there's a lot of available RAM left. Do you have an idea where I should look?

Jonathan Laurent · Answer 3 · Wed Jan 25 2023 18:30:38 GMT+0800 (China Standard Time)

Is there a way your issue is related to #47?

Stéphane Martin · Answer 4 · Wed Jan 25 2023 20:34:25 GMT+0800 (China Standard Time)

It is, thank you!
I introduced a moves count limit in my game, which makes total sense actually.
Problem solved.