A comparison of solidity fuzzing tools Foundry, Echidna & Medusa. This challenge set is not intended to be an academically rigorous benchmark but rather to present the experiences of an auditor "in the trenches"; the goal is finding the best performance "out of the box" with as little guidance & tweaking as possible.
Ensure you are using a recent version of Foundry which includes PR6530.
Install a more recent build of Echidna which appears to work fine with Solidity 0.8.23.
Compile the latest Medusa source code as it is experimental and under active development.
Configure solc-select for Echidna & Medusa:
solc-select install 0.8.23
solc-select use 0.8.23
To compile this project:
forge build
Every exercise has a basic some optionally an advanced fuzz configuration for Foundry, Echidna & Medusa. The basic configuration does not guide the fuzzer at all; it simply sets up the scenario and allows the fuzzer to do whatever it wants. The advanced configuration guides the fuzzer to the functions it should call and helps to eliminate invalid inputs which result in useless fuzz runs.
echidna --config test/01-naive-receiver/NaiveReceiverBasicEchidna.yaml ./ --contract NaiveReceiverBasicEchidna
forge test --match-contract NaiveReceiverBasicFoundry
echidna --config test/01-naive-receiver/NaiveReceiverAdvancedEchidna.yaml ./ --contract NaiveReceiverAdvancedEchidna
forge test --match-contract NaiveReceiverAdvancedFoundry
In basic configuration both Foundry & Echidna are able to break the simpler invariant but not the more valuable and difficult one. In advanced configuration both Foundry & Echidna can break both invariants. Both Foundry & Echidna reduce the exploit chain to a very concise & optimized transaction set and present this to the user in an easy to understand output. As a result they are tied and there is no clear winner. Medusa is unable to be used for this challenge as it requires a feature currently under development.
medusa --config test/02-unstoppable/UnstoppableBasicMedusa.json fuzz
echidna --config test/02-unstoppable/UnstoppableBasicEchidna.yaml ./ --contract UnstoppableBasicEchidna
forge test --match-contract UnstoppableBasicFoundry
forge test --match-contract UnstoppableAdvancedFoundry
Echidna in basic configuration can frequently break both invariants while Foundry in basic configuration can sometimes break the easier invariant but never the harder one. Foundry using advanced configuration is able to break both invariants if given an extreme amount of targeting. Medusa in basic configuration can always break both invariants and achieves this much faster than Echidna, making Medusa the clear winner.
Both Foundry & Echidna in basic mode are able to easily break the invariant, resulting in a tie. Medusa is unable to be used for this challenge as it requires a feature currently under development. Please note that the fuzz solvers for this challenge are not able to be publicly released at this time.
In basic configuration Foundry, Echidna & Medusa are all able to break the easier invariant but not the more difficult one. All Fuzzers are able to provide the user with a minimal transaction set to generate the exploit. Hence they are tied, there is no clear winner. Please note that the fuzz solvers for this challenge are not able to be publicly released at this time.
In basic configuration Foundry & Echidna can only break the easier and more valuable invariant which leads to a Critical exploit but not the harder though less valuable invariant which leads to a High/Medium. However Medusa is able to almost immediately break both invariants in unguided basic mode, making Medusa the clear winner. Please note that the fuzz solvers for this challenge are not able to be publicly released at this time.
Both Echidna & Foundry are unable to break the assertion in this stateless fuzzing challenge, while Medusa is able to break it almost instantly.