[question] Printing model summary
rfali opened this issue · comments
I was going through this Colab and trying to make a custom agent on the pattern of MyRandomDQNAgent(dqn_agent.DQNAgent). The DQNAgent's network is specified here which is in turn the NatureDQNNetwork specified here.
Now this is going to sound stupid, but I have a similar object (a multi headed DQN built in Dopamine) that I am trying to recreate in another library (RLlib). I wanted to print the model.summary() of this Keras.Model to make sure both are equivalent, but I am really struggling with it. If someone can help point out a solution. Thanks!
Small post but the pictures make it look bigger. Question at the end.
I started with this Tensorboard Tutorial and as I wanted to see a keras model summary, I was interested in seeing the Conceptual Graph. I ran their Colab too to make sure. It works well and I could see the Conceptual Graph under the keras Tag. This is how it looks like
and here is a screenshot from the Tensorboard Tutorial highlighting the Keras Tag.
I ran the Dopmaine Cartpole Colab and put the tensorboard callback as well.
tensorboard_callback = keras.callbacks.TensorBoard(log_dir=DQN_PATH)
I was able to see the Graph but not the Conceptual Graph, as their was no Keras Tag :(
I was able to run the agents Colab and visualize the Graph, but not the Conceptual Graph (same as above). Here is that image
Finally, I made a Colab myself (sharing it here) (sourced from here) to experiment with a Pong agent (to check the atari_lib.NatureDQNNetwork). Unfortunately this happened
I checked the callback API to see if there is an option to enable Tags, but didn't any.
Is there a way I can play Pong or the custom agent colab and see the Keras Tag in Tensorboard?
I know this is not by itself any issue with Dopamine, but I would greatly appreciate any help for the question above. Thanks.
I added the debug_mode=True in the Dopamine agents colab, as follows
def create_random_dqn_agent(sess, environment, summary_writer, debug_mode=True):
and decreased the number of iterations to 10, still the Tensorboard output was like this
sorry, that was a typo at that moment, but I had run it with debug=True. Here is the Colab I am working on with just that single modification to the agents colab. Still no tensorboard event files. Perhaps you can point out what I am doing wrong in this colab or the agents colab?
@psc-g sorry, I thought I had made the Colab public, but it wasn't. You can try it now here for a glance.
To cut to the chase, I am trying to modify this Dopamine based DQN agent for use in another library(RLlib) since I wanted to run this in a multi-agent environment. Please see this issue for more details and a figure of the network architecture.
Due to your familiarity with Dopamine, can you please answer the following if you can:
- The network here is outputting multiple Q-heads, one for each gamma. Is that correct? I was actually trying to print this model's summary to confirm the model outputs and shape when i posted my question.
- Since there is a single tensor output of Q-values (one for each action) in a vanilla DQN, is the model above outputting multiple tensors or a list of tensors? I think for
ngammas it is outputtingntensors of Q-values, and thehyp_q_valueis also a tensor, which is calculated through the integral function here. So the output of the model still only has Q-heads equal to num_gammas. - The action is selected according to the
acting_policy(last q-head or integral of all q-heads), but a separate loss is calculated for each head (using current_Q and target_Q), individual loss is aggregated and scaled, and then minimized through gradient descent. Is my thinking correct?
Thank you for your time, I would appreciate your help.