encounter tf retracing warning while training cobblestone Agent
Smu-Tan opened this issue · comments
Hi, I got an error when training the Hieagent (cobblestone):
WARNING:tensorflow:5 out of the last 370 calls to <function ClassicCnn.call at 0x000002A240927C18> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loo p, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes tha t can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
And it only happens when training the cobblestone agent, do you have any idea of this error? Literally, this might just be a warning rather than an error since the program still continued, however, for cobblestone, during the training process, it has very low rewards(almost all episodes are 0).
Here's the whole log:
log []
log agent started
2021-09-23 00:09:19.598061: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-09-23 00:09:19.620535: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-09-23 00:09:20.197291: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8100
2021-09-23 00:09:20.796600: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-09-23 00:09:21.224352: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
Building model
:[log:6] [cobblestone:0] [planks:0] [stick:0] [wooden_pickaxe:0] [crafting_table:0] [stone_pickaxe:0] [iron_ore:0] [furnace:0] [iron_pickaxe:0]
planks [{'craft': 'planks'}]
planks agent started
Building model
:[log:0] [cobblestone:0] [planks:24] [stick:0] [wooden_pickaxe:0] [crafting_table:0] [stone_pickaxe:0] [iron_ore:0] [furnace:0] [iron_pickaxe:0]
log []
log agent started
next step run....
:[log:6] [cobblestone:0] [planks:24] [stick:0] [wooden_pickaxe:0] [crafting_table:0] [stone_pickaxe:0] [iron_ore:0] [furnace:0] [iron_pickaxe:0]
planks [{'craft': 'planks'}]
planks agent started
crafting_table [{'craft': 'crafting_table'}]
crafting_table agent started
Building model
:[log:5] [cobblestone:0] [planks:24] [stick:0] [wooden_pickaxe:0] [crafting_table:1] [stone_pickaxe:0] [iron_ore:0] [furnace:0] [iron_pickaxe:0]
stick [{'craft': 'stick'}]
stick agent started
Building model
:[log:5] [cobblestone:0] [planks:16] [stick:8] [wooden_pickaxe:0] [crafting_table:2] [stone_pickaxe:0] [iron_ore:0] [furnace:0] [iron_pickaxe:0]
wooden_pickaxe [{'place': 'crafting_table'}, {'nearbyCraft': 'wooden_pickaxe'}]
wooden_pickaxe agent started
Building model
:[log:5] [cobblestone:0] [planks:11] [stick:10] [wooden_pickaxe:1] [crafting_table:1] [stone_pickaxe:0] [iron_ore:0] [furnace:0] [iron_pickaxe:0]
crafting_table [{'craft': 'crafting_table'}]
crafting_table agent started
:[log:5] [cobblestone:0] [planks:7] [stick:10] [wooden_pickaxe:1] [crafting_table:1] [stone_pickaxe:0] [iron_ore:0] [furnace:0] [iron_pickaxe:0]
cobblestone [{'equip': 'wooden_pickaxe'}]
cobblestone agent started
WARNING:tensorflow:5 out of the last 370 calls to <function ClassicCnn.call at 0x000002A240927C18> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loo
p, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes tha
t can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
Building model
WARNING:tensorflow:5 out of the last 370 calls to <function DuelingModel.call at 0x000002A2409EC168> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a l
oop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes t
hat can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
:[log:5] [cobblestone:0] [planks:3] [stick:10] [wooden_pickaxe:1] [crafting_table:3] [stone_pickaxe:0] [iron_ore:0] [furnace:0] [iron_pickaxe:0] Failed to delete the temporary minecraft directory.
@Smu-Tan Could you specify your tensorflow version? Do you use docker?
@Smu-Tan Could you specify your tensorflow version? Do you use docke?
TF version is TensorFlow-gpu 2.5.1 and unfortunately I don't use docker.
I'll investigate this, even if that is only a warning, the reward for the cobblestone task shouldn't be zero.
I'll investigate this, even if that is only a warning, the reward for the cobblestone task shouldn't be zero.
Thx! I guess this could be something wrong with the pretraining of cobblestone. Because I tested if running the pre-trained cobblestone agent with the hierarachical structure, it did un-reasonable actions all the time for only cobblestone agent(camera on sky and attack). Perhaps there're some bugs when slicing the trajectory data(Hie Agent) since I converted the data structures when using the newest Minerl dataset.
I'll investigate this, even if that is only a warning, the reward for the cobblestone task shouldn't be zero.
Hi, I've checked the sliced_trajectory data part, and it is correct. May I ask you what chain you used in the pretraining and training part for forger? Most likely this one？MineRLObtainIronPickaxe-v0/v3_rigid_mustard_greens_monster-11_878-4825
If so, the chain should be: ["log:6", "craft:0:planks", "planks:24", "log:1", "craft:1:planks", "planks:28", "craft:0:crafting_table", "crafting_table:1", "craft:0:stick", "stick:8", "place:0:crafting_table", "nearbyCraft:0:wooden_pickaxe", "wooden_pickaxe:1", "crafting_table:1", "equip:0:wooden_pickaxe", "cobblestone:11", "place:1:crafting_table", "nearbyCraft:0:stone_pickaxe", "stone_pickaxe:1", "wooden_pickaxe:1", "crafting_table:1", "equip:0:stone_pickaxe", "iron_ore:3", "place:2:crafting_table", "nearbyCraft:0:furnace", "furnace:1", "place:0:furnace", "nearbySmelt:0:iron_ingot", "iron_ingot:3", "nearbyCraft:0:iron_pickaxe", "iron_pickaxe:1"], right?
Coz I noticed that the reason why cobblestone agent cannot get reward probably is the agent is still “on the ground” rather than "under the ground". Because I checked the sliced data for the cobblestone agent (just eyeballing), most of the states are "under the ground". Therefore I tried to change the chain using the function extract_chain.generate_final_chain(), the chain I got is below:
["log:5", "craft:0:planks", "planks:20", "craft:0:crafting_table", "crafting_table:1", "place:0:crafting_table", "craft:0:stick", "stick:8", "nearbyCraft:0:wooden_pickaxe", "wooden_pickaxe:1", "equip:0:wooden_pickaxe", "dirt:12", "cobblestone:14", "nearbyCraft:0:stone_pickaxe", "stone_pickaxe:1", "nearbyCraft:0:furnace", "furnace:1", "stick:6", "cobblestone:3", "equip:0:stone_pickaxe", "place:0:furnace", "iron_ore:1", "cobblestone:5", "iron_ore:3", "equip:1:wooden_pickaxe", "place:0:dirt", "nearbySmelt:0:iron_ingot", "iron_ingot:3", "place:1:dirt", "nearbyCraft:0:iron_pickaxe", "iron_pickaxe:1"],
Now the performance is slightly better, at least the after dirt agent to dig down, the cobblestone agent was trying to get rewards instead of staring at the sky. But there still is a huge gap between the ideal performance and reality.
What's more, I noticed every game terminated with at most 1:40 minutes. Is that normal?
@Smu-Tan I can't reproduce reported behavior with the docker version. Also, I tried to reproduce it without docker and got errors in malmo and MineRL... Could you create a pull request with your implementation? It will be a welcome contribution to make it work with the current version of MineRL.
Yes, it was the v1_rigid_mustard_greens_monster-11_878-4825 trajectory, but I see some problems with your chain. For example, the agent does not place an additional crafting table for creating a stone pickaxe, so he will have to look for the first one (it's hard).
@Smu-Tan I can't reproduce reported behavior with the docker version. Also, I tried to reproduce it without docker and got errors in malmo and MineRL... Could you create a pull request with your implementation? It will be a welcome contribution to make it work with the current version of MineRL.
Yes, it was the
v1_rigid_mustard_greens_monster-11_878-4825trajectory, but I see some problems with your chain. For example, the agent does not place an additional crafting table for creating a stone pickaxe, so he will have to look for the first one (it's hard).
Yes, sure. But it seems I do not have access to this repo, how do you wanna me to upload my implementation?
I believe you can fork Forger, add your code, and then create a pull request.
As I see you have already created a fork with your code, so I can try to reproduce it and fix some stuff. I think I will be able to do this only in a couple of days. Please be patient
I believe you can fork Forger, add your code, and then create a pull request.
As I see you have already created a fork with your code, so I can try to reproduce it and fix some stuff. I think I will be able to do this only in a couple of days. Please be patient
Thanks! I just pulled the request.
Thx!
Hi @Tviskaron, Any updates from your side?
Sorry, still working on it
@Smu-Tan I’m sorry this update took too long, there were a bunch of reasons. Especially, with reproducing results: the 0.4 version of MineRL has problems with headless servers. Thus, we make a version of the model capable with original weights (tf1 version from paper) and now provide them using wandb artifacts. Hope this update fully addresses your issue.