keras-rl / keras-rl

Deep Reinforcement Learning for Keras.

Home Page:http://keras-rl.readthedocs.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

How to set the "nb_steps_warmup" and "nb_steps" properly?

Beliefuture opened this issue · comments

I got the following warning while using keras-rl to train a DQN agent.
The batch_size argument is a small value i.e. 50.

According to the warning information below, it seems that there are not enough experience to sample.
I am not so sure about how to balance the arguments nb_steps_warmup and nb_steps.

Moreover, does the nb_steps_warmup is a part of nb_steps, i.e. if nb_steps_warmup=10, nb_steps=50, there remain 50-10=40 steps used to train?

/lib/python3.7/site-packages/rl/memory.py:37: UserWarning: Not enough entries to sample
 without replacement. Consider increasing your warm-up phase to avoid oversampling!
  warnings.warn('Not enough entries to sample without replacement. Consider increasing your warm-up phase to avoid overs
ampling!')
Traceback (most recent call last):
  File "main.py", line 124, in <module>
    history = agent.fit(env, nb_steps=args.nb_steps_train, visualize=False, verbose=args.verbose)
  File "/lib/python3.7/site-packages/rl/core.py", line 193, in fit
    metrics = self.backward(reward, terminal=done)
  File "/lib/python3.7/site-packages/rl/agents/dqn.py", line 250, in backward
    experiences = self.memory.sample(self.batch_size)
  File "/lib/python3.7/site-packages/rl/memory.py", line 263, in sample
    batch_idxs = sample_batch_indexes(0, self.nb_entries, size=batch_size)
  File "/lib/python3.7/site-packages/rl/memory.py", line 38, in sample_batch_indexes
    batch_idxs = np.random.random_integers(low, high - 1, size=size)
  File "mtrand.pyx", line 1328, in numpy.random.mtrand.RandomState.random_integers
  File "mtrand.pyx", line 746, in numpy.random.mtrand.RandomState.randint
  File "_bounded_integers.pyx", line 1254, in numpy.random._bounded_integers._rand_int64
ValueError: low >= high

Another question is the meaning of train_interval on this issue #109 but no answer yet. The official documentation is not so detailed.