zichuan-liu / NA2Q

[ICML'23] Official PyTorch Implementation of NA2Q, and a comprehensive benchmark based on pymarl

NA²Q: Neural Attention Additive Model for Interpretable Multi-Agent Q-Learning

Code for NA²Q: Neural Attention Additive Model for Interpretable Multi-Agent Q-Learning accepted by ICML 2023. NA²Q is implemented in PyTorch and tested on challenging tasks LBF and SMAC as benchmarks, which is based on PyMARL. [paper] [code]

Python MARL framework

This PyMARL includes baselines of the following algorithms:

• QMIX: QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• COMA: Counterfactual Multi-Agent Policy Gradients
• VDN: Value-Decomposition Networks For Cooperative Multi-Agent Learning
• IQL: Independent Q-Learning
• QTRAN: QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• WQMIX: Weighted Qmix: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Qatten: Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning
• QPLEX: Qplex: Duplex Dueling Multi-Agent Q-Learning
• MASAC: Actor-Attention-Critic for Multi-Agent Reinforcement Learning
• SHAQ: SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning
• SQDDPG: QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• UPDET: Updet: Universal Multi-Agent Reinforcement Learning via Policy Decoupling with Transformers
• CDS: Celebrating Diversity in Shared Multi-Agent Reinforcement Learning

Thanks to all the original authors!

Installation instructions

Build the Dockerfile using

cd docker
bash build.sh

Set up StarCraft II and SMAC:

bash install_sc2.sh

This will download SC2 into the 3rdparty folder and copy the maps necessary to run over.

Requirements

• Python 3.6+
• pip packages listed in requirements.txt

The requirements.txt file can be used to install the necessary packages into a virtual environment (not recommended).

Run an experiment

# demo sc2
python src/main.py --config=qnam --env-config=sc2 with env_args.map_name=8m_vs_9m gpu_id=0 t_max=2010000 epsilon_anneal_time=50000 seed=1
# demo foraging
nohup python src/main.py --config=qnam --env-config=foraging with env_args.map_name=lbf-4-2 use_cuda=False seed=1 &

The config files act as defaults for an algorithm or environment.

They are all located in src/config. --config refers to the config files in src/config/algs --env-config refers to the config files in src/config/envs

All results will be stored in the Results folder.

Citing NA²Q

If you find this repository useful for your research, please cite it in BibTeX format:

@inproceedings{liu2023na2q,
  title = 	 {{NA$^2$Q}: Neural Attention Additive Model for Interpretable Multi-Agent Q-Learning},
  author =       {Liu, Zichuan and Zhu, Yuanyang and Chen, Chunlin},
  booktitle = 	 {Proceedings of the 40th International Conference on Machine Learning},
  pages = 	 {22539--22558},
  year = 	 {2023},
  volume = 	 {202},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {23--29 Jul},
  publisher =    {PMLR},
  url = 	 {https://proceedings.mlr.press/v202/liu23be.html},
}

In case of any questions, bugs, suggestions or improvements, please feel free to drop me or open an issue.

If you need the experimental results data from our paper, please contact me at [e-mail] and I'll be happy to share them!