about gpu and memory use
woltium opened this issue · comments
Thanks for your code!
but i have meet a flash back while running the code,which took up more than 20GB of memory
I have 16GB of ram and 16GB of swap space,besides my gpu is 3060,is that enough to run this code?
Namespace(ablation=0, action_loss_coef=1.0, baseline='none', centralized=1, clip_param=0.2, config_file='basic.yaml', critic_lr_coef=0.05, cuda=True, depth_noise_rate=0, depth_obs=0, du_scale=2, dump_location='train', entropy_coef=0.0001, env_frame_height=192, env_frame_width=192, eps=1e-05, eval=0, eval_eps_freq=0, exp_name='ma3_history', frame_height=192, frame_width=192, g_input_content='gaussian_dist_map', gamma=0.99, global_downscaling=2, global_lr=0.0005, hfov=90.0, load_global='0', load_global_critic='0', log_interval=250, map_size_cm=4800, max_batch_size=0, max_collisions_allowed=-1, max_episode_length=3000, max_grad_norm=0.5, noise_level=0.1, noisy_actions=0, noisy_odometry=0, num_episodes=10000, num_global_steps=120, num_gnn_layer=3, num_local_steps=25, num_mini_batch=12, num_processes=12, num_robots=3, object_randomization_freq=0.0, obs_threshold=1, obstacle_boundary=20, ppo_epoch=4, ppo_sample_eps=1, print_images=False, reset_floor=0, reset_orientation=1, restore_eps=0, reward_bias=4, reward_scale=0.005, robot_scale=1, rotation_augmentation=1, save_interval=10000, save_periodic=500000, scene_per_process=1, scenes_file='scenes/train.scenes', seed=1, short_goal_dist=1, snp_location='./snp', tau=0.95, texture_randomization_freq=0.0, train_global=1, unit_size_cm=10, use_clipped_value_loss=1, use_distance_field=1, use_gae=1, use_history=1, use_single_gpu=0, value_loss_coef=3.0, vis_type=0, vision_range=32, z_offset=0.4)
SNP allocated 47.3 MB
Device 0 is available for rendering
Command '['/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/render/mesh_renderer/build/test_device', '1']' returned non-zero exit status 1.
Device 1 is not available for rendering
Using device 0 for rendering
IndoorScene model: Beach
StaticIndoorScene scene: Beach
Loading traversable graph
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/g_dataset/Beach/mesh_z_up.obj
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/main_body.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/wheel.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/sensors/kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Device 0 is available for rendering
Command '['/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/render/mesh_renderer/build/test_device', '1']' returned non-zero exit status 1.
Device 1 is not available for rendering
Using device 0 for rendering
IndoorScene model: Colebrook
StaticIndoorScene scene: Colebrook
Loading traversable graph
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/g_dataset/Colebrook/mesh_z_up.obj
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/main_body.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/wheel.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/sensors/kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Device 0 is available for rendering
Command '['/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/render/mesh_renderer/build/test_device', '1']' returned non-zero exit status 1.
Device 1 is not available for rendering
Using device 0 for rendering
IndoorScene model: Connellsville
StaticIndoorScene scene: Connellsville
Loading traversable graph
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/g_dataset/Connellsville/mesh_z_up.obj
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/main_body.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/wheel.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/sensors/kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Device 0 is available for rendering
Command '['/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/render/mesh_renderer/build/test_device', '1']' returned non-zero exit status 1.
Device 1 is not available for rendering
Using device 0 for rendering
IndoorScene model: Edgemere
StaticIndoorScene scene: Edgemere
Loading traversable graph
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/g_dataset/Edgemere/mesh_z_up.obj
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/main_body.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/wheel.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/sensors/kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Device 0 is available for rendering
Command '['/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/render/mesh_renderer/build/test_device', '1']' returned non-zero exit status 1.
Device 1 is not available for rendering
Using device 0 for rendering
IndoorScene model: Delton
StaticIndoorScene scene: Delton
Loading traversable graph
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/g_dataset/Delton/mesh_z_up.obj
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/main_body.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/wheel.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/sensors/kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Device 0 is available for rendering
Command '['/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/render/mesh_renderer/build/test_device', '1']' returned non-zero exit status 1.
Device 1 is not available for rendering
Using device 0 for rendering
IndoorScene model: Dunmor
StaticIndoorScene scene: Dunmor
Loading traversable graph
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/g_dataset/Dunmor/mesh_z_up.obj
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/main_body.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/wheel.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/sensors/kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Device 0 is available for rendering
Command '['/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/render/mesh_renderer/build/test_device', '1']' returned non-zero exit status 1.
Device 1 is not available for rendering
Using device 0 for rendering
IndoorScene model: Ewansville
StaticIndoorScene scene: Ewansville
Loading traversable graph
Loading traversable graph
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/g_dataset/Ewansville/mesh_z_up.obj
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/main_body.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/wheel.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/sensors/kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Device 0 is available for rendering
Command '['/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/render/mesh_renderer/build/test_device', '1']' returned non-zero exit status 1.
Device 1 is not available for rendering
Using device 0 for rendering
IndoorScene model: Goffs
StaticIndoorScene scene: Goffs
Loading traversable graph
Loading traversable graph
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/g_dataset/Goffs/mesh_z_up.obj
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading robot model file: turtlebot/turtlebot.urdf
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/main_body.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/wheel.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_bottom.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_middle.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_top.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/sensors/kinect.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
Device 0 is available for rendering
Command '['/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/render/mesh_renderer/build/test_device', '1']' returned non-zero exit status 1.
Device 1 is not available for rendering
Using device 0 for rendering
IndoorScene model: Maryhill
StaticIndoorScene scene: Maryhill
Loading traversable graph
Loading traversable graph
Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/g_dataset/Maryhill/mesh_z_up.obj
here is my log
This is most likely a GPU memory overflow and you need more GPUs for training.
Or do this: edit scenes/train.scenes and delete the first three lines and the last three lines. (Doing so will slightly reduce the training effect)
thanks a lot,that problem is fixed after changed as you said,but there are aother problem:
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/main_body.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/kobuki_description/meshes/wheel.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_bottom.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_bottom.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_middle.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_middle.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_top.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/pole_kinect.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/stacks/hexagons/plate_top.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/turtlebot/turtlebot_description/meshes/sensors/kinect.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
INFO:root:Loading /home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/gibson2/data/assets/models/mjcf_primitives/cube.obj
2023-02-23 12:26:43,023 Computing map for Maryhill
2023-02-23 12:26:43,023 Computing map for Ewansville
2023-02-23 12:26:43,023 Computing map for Goffs
2023-02-23 12:26:43,061 Computing map for Edgemere
2023-02-23 12:26:43,221 Computing map for Dunmor
2023-02-23 12:26:43,457 Computing map for Delton
Traceback (most recent call last):
File "main.py", line 606, in <module>
main()
File "main.py", line 191, in main
deterministic=False
File "/home/ubuntu/stuff/NeuralCoMapping/model.py", line 121, in act
value, actor_features, rnn_hxs = self(inputs, rnn_hxs, masks, extras)
File "/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
result = self.forward(*input, **kwargs)
File "/home/ubuntu/stuff/NeuralCoMapping/model.py", line 117, in forward
return self.network(inputs, rnn_hxs, masks, extras)
File "/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
result = self.forward(*input, **kwargs)
File "/home/ubuntu/stuff/NeuralCoMapping/utils/gnn.py", line 393, in forward
value = self.critic(inputs[:, :6, :, :]).squeeze(-1) - 1.5
File "/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
result = self.forward(*input, **kwargs)
File "/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
result = self.forward(*input, **kwargs)
File "/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 343, in forward
return self.conv2d_forward(input, self.weight)
File "/home/ubuntu/anaconda3/envs/neural/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 340, in conv2d_forward
self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR
I think there is some version mismatch with cuDNN or CUDA. May you can copy the description and google it.
thanks again for your reply
i have changed my cuda and cudnn version and i still got errors
the result i got from google is still that the GPU memory usage is too large.
is there any way to further reduce the memory usage?I just hope to run this code successfully for my graduation project and i dont particularly care about the final effect
thanks again!
Try editing the scene file, only leaving a single line "Maryhill". In that case, you are training with minimal GPU memory usage.
Or, if you do not want to train your own model, just try eval mode with our provided pretrained model.
finally i found the cuda version that matchs this project（cudaversion==10.2） is not supported by RTX3060 ,which needs cuda>=11
and i solved it by changing my gpu to 2080ti