Common tests for all clients to test against. See the documentation http://ethereum-tests.readthedocs.io/
Contents of this repository
Do not change test files in folders:
- StateTests
- BlockchainTests
- TransactionTests
- VMTests
It is being created by the testFillers which could be found at src folder. The filler specification and wiki are in development so please ask on gitter channel for more details.
If you want to modify a test filler or add a new test please contact @winsvega at https://gitter.im/ethereum/aleth
All files should be of the form:
{
"test1name":
{
"test1property1": ...,
"test1property2": ...,
...
},
"test2name":
{
"test2property1": ...,
"test2property2": ...,
...
}
}
Arrays are allowed, but don't use them for sets of properties - only use them for data that is clearly a continuous contiguous sequence of values.
Test Set Sanitation
Checkers
Several basic checks against the test-set are performed to enusre that they have been filled and are formatted correctly. Currently, there are three types of checks that we can perform:
make TEST_PREFIX.format
: check that the JSON is formatted correctly.make TEST_PREFIX.valid
: check that the JSON files are valid against the JSON schemas in./JSONSchema
.make TEST_PREFIX.filled
: check that the JSON tests are filled with the correct source hashes against the fillers.
The constant TEST_PREFIX
is a path prefix to the test-set you're interested in performing the checks on.
For instance:
make ./src/VMTestsFiller/vmArithmeticTest.format
will check that all JSON files in./src/VMTestsFiller/vmArithmeticTest
are formatted correctly.make ./src.valid
will check that all the JSON files in./src
are valid against the JSON schemas in./JSONSchema
.make ./BlockchainTests.filled
will check that the source hashes in the JSON tests in./BlockchainTests
are the same as the hashes of the fillers in./src/BlockchainTestsFiller
.
These checks are all performed by the file ./test.py
, which can be invoked on individual files as well.
Run ./test.py
with no arguments for help.
Sanitizers
The above checkers are packaged together into sanitizers for each test-suite, marking which testsuites are passing which testers.
See the TODO
s in the Makefile
to see which checkers are enabled for which test-suites.
-
make sani
: will run all passing sanitizers on all passing testsuites. -
make sani-TESTNAME
: will run just the passing sanitizers for the given testsuite.TESTNAME
can be one of:vm
: VMTests and VMTestsFillergs
: GeneralStateTests and GeneralStateTestsFillerbc
: BlockchainTests and BlockchainTestsFillertx
: TransactionTests and TransactionTestsFiller
Runners/Fillers
The tests can also be run/filled with the same Makefile
.
make run
: will usetesteth
to all the test-suites.make fill
: will usetesteth
to fill all the test-suites.make TEST_PREFIX.run
: runs a single testsuite.make TEST_PREFIX.fill
: fills and formats a single testsuite.
FAQ
I want to test my client, which version shall I use?
The develop branch in ethereum/tests is the version to use.