Tech-Audit #2: Code Tests
TomDonoghue opened this issue · comments
This issue is part of a 'technical audit' of neurodsp, with @elybrand.
Our current code tests are, broadly speaking, 'smoke tests', meaning they execute code in the module, and generally check that nothing breaks - but they generally do not test very much for accuracy of the outputs.
As part of the technical audit, it would be nice to update / extend the tests we have to do more accuracy checking. This might help to discover if there are any issues with our current implementations, as well as offering a better test suite to guide future development.
If you are using simulations to check expected output in #193, this this issue can broadly be addressed by collecting these examples and trying to add them to our code tests, in so far as we can do so effectively and efficiently.
All our tests currently work on simulated data, for which approximate results can be predicted. Testing analysis functions is quite difficult - so I think the goal here is to add more accuracy checking, if & where we can, but without "over-doing it" (as in, what can we add to be more confident in our implementations, and to be safer w.r.t. future development, without falling too deep into the rabbit hole of broadly difficult questions about how to precisely test code that computes estimations).