Missing documentation describing relationship between tests
davidben opened this issue · comments
There are a number of (partially?) redundant test sets, such as the different RSA, ECDH, and ECDSA files. Having some kind of README or documentation at the front of various files would be useful to know which to use.
I wrote a quick tool to compare the combined and split up files. The RSA and ECDSA split up ones do seem to be larger, as one would expect. I'm guessing, e.g., some edge cases were only tested at certain hashes or key sizes and now you do them everywhere. However there is a large discrepancy between the tests in both direction. Sampling a few, it looks like the RSA and ECDSA tests ended up using different random seeds between split up and combined. Is that correct?
ECDH is odd as ecdh_test.json tests brainpool*t1 curves, but there aren't separated files for them. After correcting for that, the test counts and most of the tests match. There are 10 "public key of order 3" that have slightly different value, but I gather those too are just a difference in random seed.
The general plan is to split large files into more files based on curves or key size and leave odd cases in the unparametrisized file. ecdsa_test.java, ecdh_test.java are just too large right now and will be reduced.
The current state is a bit unfortunate, since they were published in the middle of this transition.
Documentation in general is quite bad. It is still unclear what to use:
I.e. the minimum requirement would be to have support for equations and references and be simple to
use. Markup on github is unfortunately lacking almost every useful feature. It actually seems to be worse
than Microsoft word from 30 years ago.
A question that came up for us: is ecdsa_test.json a concatention of all of the other ecdsa_*_test.json files, or does it contain it's own things?