Difference and similarities between Adaptive Stress Testing packages
ll7 opened this issue · comments
I try to learn about Adaptive Stress Testing and the "Stanford Intelligent Systems Laboratory" provides three different Toolboxes:
- https://github.com/sisl/AdaptiveStressTestingToolbox
- https://github.com/sisl/AdaptiveStressTesting.jl
- https://github.com/sisl/POMDPStressTesting.jl
Could you explain the differences and similarities between these packages or guide me where I should start to learn about the implementation of Adaptive Stress Testing?
Hi @ll7
The ecosystem is indeed a little confusing at the moment. There are plans to clean it up.
AdaptiveStressTesting.jl is the original implementation of AST in julia used to support the ACAS Xa and some NASA projects. It is developed by folks at NASA and carries a NASA open source license. It only supports seed-based MCTS and DAST.
AdaptiveStressTestingToolbox is a python package that implements AST. It supports MCTS, DeepRL, Go-Explore algorithms. This package was used to support many of the autonomous vehicles papers.
POMDPStressTesting.jl is a new framework for AST in julia that has a much cleaner interface. It is based on and compatible with POMDPs.jl and RLInterface.jl. It supports MCTS and Deep RL (TRPO/PPO). It supports seed- and raw actions. It carries an MIT license.
NASA restrictions prevent us from mixing licenses, so NASA contributors cannot directly contribute to external open source projects; and external contributors cannot directly contribute to NASA open source projects. POMDPStressTesting.jl is currently more up-to-date and capable. If you're in julia, I would recommend going with POMDPStressTesting.jl. That said, there are planned updates for AdaptiveStressTesting.jl that will add more capability and make it compatible with POMDPStressTesting.jl. If you're in python, go with AdaptiveStressTestingToolbox.
Thank you very much for the clarification, and it is great that you share your implementations!