blefaudeux / salina

a Lightweight library for sequential learning agents, including reinforcement learning

SaLinA: SaLinA - A Flexible and Simple Library for Learning Sequential Agents (including Reinforcement Learning)

TL;DR

salina is a lightweight library extending PyTorch modules for developping sequential decision models. It can be used for Reinforcement Learning (including model-based with differentiable environments, multi-agent RL, etc...), but also in a supervised/unsupervised learning settings (for instance for NLP, Computer Vision, etc...).

• It allows to write very complex sequential models (or policies) in few lines
• It works on multiple CPUs and GPUs

News

• October 2021:
• • Week 18th of october
• • • Release of the library
• • • Various bug fixes
• • • Add train and eval mode in the RemoteAgent and NRemoteAgent
• • • Cleaning of the BRAX PPO Implementation to make it similar to other implementations
• • • Cleaning of the Behavioral Cloning implementation
• • Week 25th of october
• • • Updated documentation
• • • Transformer-based Agents
• • • Behavioral Cloning over complete episodes
• • • • With and without transformers
• • • PPO on Brax with Transformers

Quick Start

• Just clone the repo
• pip install -e .

Documentation

• Principles of the library
• Examples and Algorithms
• Tutorial through multiple A2C implementations
• Reinforcement Learning Benchmark
• Video Tutorials:
• • Tutorial 1: Agent and Workspace
• • Tutorial 2: Multi-CPUs
• • Tutorial 3: Implementing A2C
• • Tutorial 4: A2C on multi-CPUs
• Arxiv Paper

For development, set up pre-commit hooks:

• Run pip install pre-commit
  • or conda install -c conda-forge pre-commit
  • or brew install pre-commit
• In the top directory of the repo, run pre-commit install to set up the git hook scripts
• Now pre-commit will run automatically on git commit!
• Currently isort, black and blacken-docs are used, in that order

Organization of the repo

• salina is the core library
  • salina.agents is the catalog of agents (the same as torch.nn but for agents)
• salina_examples provide many examples (in different domains)

Dependencies

salina utilizes pytorch, hydra for configuring experiments, and gym for reinforcement learning algorithms.

Note on the logger

We provide a simple Logger that logs in both tensorboard format, but also as pickle files that can be re-read to make tables and figures. See logger. This logger can be easily replaced by any other logger.

Description

Sequential Decision Making is much more than Reinforcement Learning

• Sequential Decision Making is about interactions:
• Interaction with data (e.g attention-models, decision tree, cascade models, active sensing, active learning, recommendation, etc….)
• Interaction with an environment (e.g games, control)
• Interaction with humans (e.g recommender systems, dialog systems, health systems, …)
• Interaction with a model of the world (e.g simulation)
• Interaction between multiple entities (e.g multi-agent RL)

What salina is

• A sandbox for developping sequential models at scale.

• A small (300 hundred lines) 'core' code that defines everything you will use to implement agents involved in sequential decision learning systems.

  • It is easy to understand and use since it keeps the main principles of pytorch, just extending nn.Module to Agent in order to handle the temporal dimension

• A set of agents that can be combined (like pytorch modules) to obtain complex behaviors

• A set of references implementations and examples in different domains Reinforcement Learning, Imitation Learning, Computer Vision, with more to come...

What salina is not

• Yet another reinforcement learning framework: salina is focused on sequential decision making in general. It can be used for RL (which is our main current use-case), but also for supervised learning, attention models, multi-agent learning, planning, control, cascade models, recommender systems, among other use cases.
• A library: salina is just a small layer on top of pytorch that encourages good practices for implementing sequential models. Accordingly, it is very simple to understand and use, while very powerful.

Citing salina

Please use this bibtex if you want to cite this repository in your publications:

Link to the paper: SaLinA: Sequential Learning of Agents

    @misc{salina,
        author = {Ludovic Denoyer, Alfredo de la Fuente, Song Duong, Jean-Baptiste Gaya, Pierre-Alexandre Kamienny, Daniel H. Thompson},
        title = {SaLinA: Sequential Learning of Agents},
        year = {2021},
        publisher = {Arxiv},
        howpublished = {\url{https://gitHub.com/facebookresearch/salina}},
    }

Papers using SaLinA:

• Learning a subspace of policies for online adaptation in Reinforcement Learning. Jean-Baptiste Gaya, Laure Soulier, Ludovic Denoyer - Arxiv

License

salina is released under the MIT license. See LICENSE for additional details about it. See also our Terms of Use and Privacy Policy.