Some questions about 'errors' in the result.csv
CheesyDetector opened this issue · comments
'CareerOnToken.sol' is a vulnerable smart contract with 'pragma solidity ^0.5.1' and it can be compiled successfully in my PC
But I used the command (I've tried it twice)
./smartbugs -t all -f dataset/tx_origin/CareerOnToken.sol --processes 4 --timeout 600
./reparse results
./results2csv -p results > results.csv
to generate the result.csv, I found many tools didn't produce 'findings'.
Confuzzius fails: exception (UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 1683: ordinal not in range(128))
Honeybadger Osiris Oyente infos: analysis incomplete
Honeybadger Osiris errors: Solidity compilation failed
It looks like static tools that don't need to compile source code will work just fine. But I still don't know what the problem is. Does anyone know the cause of the problem？ o(╥﹏╥)o
@CheesyDetector Thanks for reporting the issue(s). What you see are errors by and limitations of the tools, not of SmartBugs directly, with different causes. Since you did not provide the input file CareerOnToken.sol, I can only guess.
For confuzzius, the first problem seems to be that the source file contains Chinese characters within the comments, which leads to the error message you see. When removing the comments, the contract compiles, but confuzzius runs into another problem.
For honeybadger, osiris and oyente (the latter is the basis of the first two, so they often behave similarly), there is a slight change in the compiler output starting with version 0.5.something, so the regular expression for parsing the output fails. Compilation actually succeeds, but the tools interpret the failed parsing as a compilation problem. After fixing the regex, compilation is fine, but the tools stumble over an EVM instruction that was introduced after the tools were released.
Actually, there are efforts under way to "modernize" these four tools such that they can handle contracts compiled for newer EVM versions, but we are not ready yet. I'm afraid that for the moment, you can't use these four tools for contracts for Solidity 0.5 and beyond. You can try to circumvent compilation problems by compiling the contract yourself (using solc --bin-runtime), storing the line with the hex code in a file CareerOnToken.rt.hex, and run SmartBugs on this file. This does not, however, solve any subsequent problems like new instructions unknown to the tools.
@gsalzer Thank you for your help, please forgive me for taking a long time to reply.
In my survey, a very large number of the vulnerability detection tools and their papers were published between 2016 and 2020, which means that the tools may have been designed earlier. But soLidity only officially appeared in 2019. And that's why many tools do not adapt the versions(>0.4) well.
I tried to remove all Chinese, Russian and strange characters from the contract and repeated the experiment, the exception
Confuzzius fails: exception (UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 1683: ordinal not in range(128))
is gone. A new fail message has come up and I guess it has to do with version0.5+ and Confuzzius itself
EXIT_CODE_254,Problem while deploying contract CareerOnToken using account 0xcafebabecafebabecafebabecafebabecafebabe. Error message: b''
As for the Oyente-based tools, well, what you said is a very important shortcoming.
Lastly, I've tested Manticore with --timeout 1200 and --timeout 1800. However, regardless of whether the tool ran out of time its findings are all blank. I don't know how to get the vulnerability detection results.
@CheesyDetector There seems to be something wrong with the configuration of Manticore. An older version definitely worked with the given command line arguments, but you are right about the current version, which produces hardly anything useful. I have to check that.
@CheesyDetector If you are interested in the performance of the tools in SmartBugs over time (at least of some of them), see Evolution of Automated Weakness Detection in Ethereum Bytecode: a Comprehensive Study. The study shows that some tools gradually stop reporting weaknesses as more and more contracts use instructions they cannot handle.