How to set the "nb_steps_warmup" and "nb_steps" properly?
Beliefuture opened this issue · comments
Beliefuture commented
I got the following warning while using keras-rl
to train a DQN agent.
The batch_size
argument is a small value i.e. 50.
According to the warning information below, it seems that there are not enough experience to sample.
I am not so sure about how to balance the arguments nb_steps_warmup
and nb_steps
.
Moreover, does the nb_steps_warmup
is a part of nb_steps
, i.e. if nb_steps_warmup
=10, nb_steps
=50, there remain 50-10=40 steps used to train?
/lib/python3.7/site-packages/rl/memory.py:37: UserWarning: Not enough entries to sample
without replacement. Consider increasing your warm-up phase to avoid oversampling!
warnings.warn('Not enough entries to sample without replacement. Consider increasing your warm-up phase to avoid overs
ampling!')
Traceback (most recent call last):
File "main.py", line 124, in <module>
history = agent.fit(env, nb_steps=args.nb_steps_train, visualize=False, verbose=args.verbose)
File "/lib/python3.7/site-packages/rl/core.py", line 193, in fit
metrics = self.backward(reward, terminal=done)
File "/lib/python3.7/site-packages/rl/agents/dqn.py", line 250, in backward
experiences = self.memory.sample(self.batch_size)
File "/lib/python3.7/site-packages/rl/memory.py", line 263, in sample
batch_idxs = sample_batch_indexes(0, self.nb_entries, size=batch_size)
File "/lib/python3.7/site-packages/rl/memory.py", line 38, in sample_batch_indexes
batch_idxs = np.random.random_integers(low, high - 1, size=size)
File "mtrand.pyx", line 1328, in numpy.random.mtrand.RandomState.random_integers
File "mtrand.pyx", line 746, in numpy.random.mtrand.RandomState.randint
File "_bounded_integers.pyx", line 1254, in numpy.random._bounded_integers._rand_int64
ValueError: low >= high
Beliefuture commented
Another question is the meaning of train_interval
on this issue #109 but no answer yet. The official documentation is not so detailed.