Running the brain demo gets warning every iteration

[Benjamin Hill]

I get a warning every loop when running the brain demo code:

  if(this.regression && y.constructor !== Array)
    console.log("Warning: a regression net requires an array as training output vector.");

This is the basic code from demo/rldemo.html

function start() {
    // var brain = new deepqlearn.Brain(num_inputs, num_actions);
    var brain = new deepqlearn.Brain(3, 2); // 3 inputs, 2 possible outputs (0,1)
    var state = [Math.random(), Math.random(), Math.random()];
    for(var k=0;k<10000;k++) {
        // var action = brain.forward(array_with_num_inputs_numbers);
        var action = brain.forward(state); // returns index of chosen action
        var reward = action === 0 ? 1.0 : 0.0;
        brain.backward(reward); // <-- learning magic happens here
        state[Math.floor(Math.random()*3)] += Math.random()*2-0.5;
    }
    brain.epsilon_test_time = 0.0; // don't make any more random choices
    brain.learning = false;
    // get an optimal action from the learned policy
    var action = brain.forward([.9, .3, .1]);
    console.log(action);
}

[Nima Gardideh]

@salamanders – were you able to solve this issue?

[Glennn]

I'm also experiencing this. @salamanders, @nemo - did you figure this out?
I note that the comment for "RegressionLayer" in convnet.js says that:

// y is a list here of size num_inputs or... it can be a struct {dim: i, val: x}

so it seems to me that warning might be out-of-date, as y can either be an Array or a structure (like this) for the regression layer?

[Andrej]

Gah, sorry about this. Do brain.backward([reward])

edit: also fixed on site.

[Glennn]

@karpathy - That didn't work for me. This is because the "y" variable in the train function is a structure ({dim: i, val:x}, not an Array, in the case of the deep-q-learning example.

[Nima Gardideh]

hey @karpathy – that actually doesn't fix the problem.

You can get rid of the warning by changing this line in deepqlearn.js#254:

          var loss = this.tdtrainer.train(x, ystruct);

to

          var loss = this.tdtrainer.train(x, [ystruct]);

Though, I'm not sure if this breaks anything or not.

[Glennn]

@nemo - While that removes the logged error (because you're now sending in an Array), I think the logic in the RegressionLayer "backward" function is no longer correct. The this.out_depth variable is 5, but y is a 1-dimensional array - so when you loop i from 0 to 5, the loss function becomes undefined (NAN) for i>0. I didn't see any errors in the output, but (to me at least) I don't think this will actually train correctly.

On reviewing the code, I think the correct solution is, probably, to change the warning so it checks for conditions where y is either (a) an Array, (b) a number of (c) a struct with entries .dim and .val - as these are the three cases that RegressionLayer actually handles. The deepQLearning example seems to be configured as an example of (c).

That being said, I'm still trying to understand fully how this works, so am not 100% confident about that assessment of the issue.

[Nima Gardideh]

@DrGlennn – I thought the backward function is the one in the RegressionLayer class (which accounts for an arrayed y value.

if(y instanceof Array || y instanceof Float64Array) {
        for(var i=0;i<this.out_depth;i++) {
          var dy = x.w[i] - y[i];
          x.dw[i] = dy;
          loss += 0.5*dy*dy;
        }
}

[Benjamin Hill]

I am not getting the error from the latest hosted build - but I may not be running it the same way, I had to whip together what I think I did last time.

[Glennn]

@nemo - Yes. The backward function is the one in the RegressionLayer and, in general, it will handle an Array (as per the code you included). However, this won't work correctly when the array you are sending in is an array of a single element like:
[{dim: 0, val:2.356]

If you just convert {dim:0, val:2.356} into an array (by putting [] around it) then it will use that code, but will loop from i=0 to i=this.out_depth (here out.depth=5). After the first element, this will produce an overall loss of NAN.

Anyway, that's my understanding but I'm not an expert by any means, so happy to receive clarification if you think I've misunderstood what's going on.

@salamanders - I still get the logging error, when running the rldemo.html example. As I see it, the basic issue is that thedeepqlearn.jscode (line 253) is sending in the following structure to the trainer:

      var ystruct = {dim: e.action0, val: r};
      var loss = this.tdtrainer.train(x, ystruct);

As such the test in the train method:

  if(this.regression && y.constructor !== Array)
   console.log("Warning: a regression net requires an array as training output vector.");

is correctly (at least as written) logging a warning that y is not an Array. However, despite the warning, the Regression method does not seem to require an array as a training output vector. It can also handle this {dim: ?, val: ?} structure.

Happy to hear if I'm misunderstanding what is happening here. I'm still trying to really understand, so happy to be corrected if my understanding is way off base.

That being said, my guess is that the "if statement" just needs to include the condition where y is of the form {dim: e.action0, val: r}

Thoughts?