Reproduce the result in paper

[Tsaisplus]

Hi! Thank you for the work!

I was trying to reproduce the TABLE II you illustrated in the paper. Especially the predator capture prey scenario in TABLE III. However, i fail to obtain the similar result maybe due to the hyperparameter setting.

Could you please provide the full script of

• environment config.yaml
• epymarl default.yaml
• gymma.yaml
• especially VDN.yaml (as in it is the best result)

Thank you and looking forward to hear back from you!

Best regards

[Reza Torbati]

Hello,

For VDN on predator capture prey, this was the config.json for our final VDN model that we logged which should contain the combination of all yaml files for one of the seeds (the only difference between seeds is the value for seed). All of the hyperparameters we used for each algorithm can be found in section III of the supplementary material. Please let me know if you need anything else!

{
  "action_selector": "epsilon_greedy",
  "add_value_last_step": true,
  "agent": "rnn",
  "agent_output_type": "q",
  "batch_size": 32,
  "batch_size_run": 1,
  "buffer_cpu_only": true,
  "buffer_size": 5000,
  "checkpoint_path": "",
  "double_q": true,
  "env": "gymma",
  "env_args": {
    "key": "robotarium_gym:PredatorCapturePrey-v0",
    "pretrained_wrapper": null,
    "time_limit": 200
  },
  "epsilon_anneal_time": 50000,
  "epsilon_finish": 0.05,
  "epsilon_start": 1.0,
  "evaluate": false,
  "evaluation_epsilon": 0.0,
  "gamma": 0.99,
  "grad_norm_clip": 10,
  "hidden_dim": 128,
  "hypergroup": null,
  "label": "default_label",
  "learner": "q_learner",
  "learner_log_interval": 10000,
  "load_step": 0,
  "local_results_path": "results",
  "log_interval": 50000,
  "lr": 0.0003,
  "mac": "basic_mac",
  "mixer": "vdn",
  "name": "vdn_final",
  "obs_agent_id": true,
  "obs_individual_obs": false,
  "obs_last_action": false,
  "optim_alpha": 0.99,
  "optim_eps": 1e-05,
  "render": false,
  "repeat_id": 1,
  "runner": "episode",
  "runner_log_interval": 10000,
  "save_model": true,
  "save_model_interval": 50000,
  "save_replay": false,
  "seed": 338356864,
  "standardise_returns": false,
  "standardise_rewards": false,
  "t_max": 5050000,
  "target_update_interval_or_tau": 200,
  "test_greedy": true,
  "test_interval": 50000,
  "test_nepisode": 100,
  "use_cuda": true,
  "use_rnn": true,
  "use_tensorboard": false
}

[Tsaisplus]

Thank you for the response!

I have another question. When i run the epymarl/src/main.py with the provided hyperparameters (batch_size i change to 16 instead of 32), for 5050000 timesteps. It occurs that in the middle of the training, at around 600000+ steps, the computer is crashed/freezed and the training process is stopped. I thought it was my computer's issue at first but afterwards i tried on 2 different desktops which remains the same issue, stopped or crashed at 60w+ and 80w+ steps respectively.

For example:
[INFO 11:39:20] my_main t_env: 600925 / 5050000
[INFO 11:39:20] my_main Estimated time left: 10 hours, 37 minutes, 40 seconds. Time passed: 1 hours, 4 minutes, 34 seconds
[INFO 11:39:30] my_main Saving models to results/models/vdn_seed25874876_robotarium_gym:PredatorCapturePrey-v0_2023-09-12 10:34:44.042057/600925
[INFO 11:40:08] my_main Recent Stats | t_env: 605475 | Episode: 4515
ep_length_mean: 115.9117 epsilon: 0.0500 grad_norm: 2.6975 loss: 1.4832
q_taken_mean: -0.8722 return_mean: -14.1051 return_std: 26.3204 target_mean: -0.8896
td_error_abs: 0.2769 test_ep_length_mean: 124.5600 test_return_mean: -17.2800 test_return_std: 24.2796

Killed

Do you have any suggestions on how to debug this issue?

This is my run.json file:

{
"artifacts": [],
"command": "my_main",
"experiment": {
"base_dir": "/home/tsaisplus/Codes/MARBLER/epymarl/src",
"dependencies": [
"numpy==1.19.5",
"PyYAML==5.3.1",
"sacred==0.8.4",
"torch==2.0.0+cu118"
],
"mainfile": "main.py",
"name": "pymarl",
"repositories": [
{
"commit": "d82fc82823081346995081c41d370c931440b283",
"dirty": true,
"url": "https://github.com/uoe-agents/epymarl.git"
},
{
"commit": "d82fc82823081346995081c41d370c931440b283",
"dirty": true,
"url": "https://github.com/uoe-agents/epymarl.git"
},
{
"commit": "d82fc82823081346995081c41d370c931440b283",
"dirty": true,
"url": "https://github.com/uoe-agents/epymarl.git"
}
],
"sources": [
[
"main.py",
"_sources/main_f135a1dbadecdb77ab6ea3ee221e22a9.py"
],
[
"run.py",
"_sources/run_79ab728956cdd749855c86ef08770541.py"
],
[
"utils/logging.py",
"_sources/logging_744201ef94cc2445aea6ad1cdb783e46.py"
]
]
},
"heartbeat": "2023-09-12T04:18:02.383215",
"host": {
"ENV": {},
"cpu": "12th Gen Intel(R) Core(TM) i9-12900K",
"gpus": {
"driver_version": "525.125.06",
"gpus": [
{
"model": "NVIDIA GeForce RTX 3070",
"persistence_mode": false,
"total_memory": 8192
}
]
},
"hostname": "XPS-8950",
"os": [
"Linux",
"Linux-6.2.0-32-generic-x86_64-with-glibc2.17"
],
"python_version": "3.8.17"
},
"meta": {
"command": "my_main",
"config_updates": {},
"named_configs": [],
"options": {
"--beat-interval": null,
"--capture": null,
"--comment": null,
"--debug": false,
"--enforce_clean": false,
"--file_storage": null,
"--force": false,
"--help": false,
"--id": null,
"--loglevel": null,
"--mongo_db": null,
"--name": null,
"--pdb": false,
"--print-config": false,
"--priority": null,
"--queue": false,
"--s3": null,
"--sql": null,
"--tiny_db": null,
"--unobserved": false,
"COMMAND": null,
"UPDATE": [],
"help": false,
"with": false
}
},
"resources": [],
"result": null,
"start_time": "2023-09-12T02:34:44.038201",
"status": "RUNNING"
}

Thank you for your help!

[Reza Torbati]

The most recent push to the Robotarium's simulator essentially causes a memory leak when training. If you revert it to commit 6bb184e (the second more recent commit) it should work. I have updated the installation instructions in the README to reflect this.