load model retraining consumes exponential time
ccccccc17 opened this issue · comments
I am doing a continuing learning framework with three tasks. And I noticed that each time I load the model and keep training in new tasks, it takes much longer than training from scratch. I am wondering why this happens and how to accelerate it.
Thanks!
That's a good question... Are you training for a given number of steps, or episodes? If the latter, one hypothesis: when you load a pretrained model, the agent does a better job at avoiding collisions (compared to a newly initialised model), which means that episodes will last longer and take more time to simulate. Wdyt?
I use the model. learn() function to pass the hyper-parameters, where the total learning steps are 2e5 for each subtask.
Btw, I got the user warning saying that 'a Box observation space has an unconventional shape'. Do you think I should be worried? I am using the dqn_cnn scripts to run a highway-fast env.
The GrayscaleImage observation has so-called "unconventional shape" because it's [W, H] only and SB3 expects [W, H, C] for an image. I don't think it is worrying as long as it's handled correctly in SB3, but I cant say I remember exactly what they do. If they do nothing and feed the observation to a CNN directly it's probably fine.
I got a new one. Sorry about occupying so much time. I am trying to retrieve evaluation statistics, for instance, episode rewards. What are some hints about that? Or maybe you have a code snippet I could learn from.
the rewards are obtained at every environment step:
reward, obs, done, truncated, info = env.step(action)
You can sum them across timesteps until a terminal state is reached to get the episode rewards.