CMSIS-NN - Overwrite `output` Between Consecutive Unit Tests
fpedd opened this issue · comments
The CMSIS-NN unit tests often consecutively execute functions in the same unit test case. They make use of the same input values and parameters and are expected to produce the same output
. Taking the test_arm_convolve_s8.c
as an example. Here, we first test arm_convolve_s8()
and then arm_convolve_wrapper_s8
in every test case; using the same input and output
variables.
While tinkering around, I came across the (admittedly) rare case that a modified (broken) version of arm_convolve_wrapper_s8()
would not modify the output
array at all. It would simply do some calculations, but never write its result back to the output
array. However, due to the fact, that the output
array does not get reset between consecutive calls, the unit tests would still run through without indicating an error (they still had the correct values from the previous function invocation). This can of course potentially apply to all unit test cases where more than one function is called and expected to produce the same output
.
A simple and straightforward fix to this issue would be to overwrite the output
between function invocations. This can be done by adding a single line act[i] = 0;
in the validate()
function here (for validate_s16()
analogously).
To nitpick some more: In theory, the same can be said for the intermediate buffer ctx.buf
. It, or rather its underlying memory, should (ideally) also not be left untouched. While it does get free
ed and malloc
ed between consecutive calls, the chances of malloc
returning the same memory (buffer) are quite high. Thus, in the (admittedly very rare) case that only a single im2col
iteration would be needed to compute the output
tensor, a similarly implemented followup function could have a broken im2col
implementation and "cheat" its way through the unit tests by simply relying on the successful computation of the im2col
buffer from the previous function.
What both these cases come down to is that one should ideally reset all output (and ideally also intermediate) buffers between consecutive unit tests to either zero (or even better random) values. This ensures that functions do not "cheat" and that they have to compute all needed values themselves.
@fabianpedd Thanks for the issue. I can see what you are referring to. Clearing the output buffers between two consecutive calls, yes, it should be cleared for better robustness.
As for im2col a clarification in arm_nnfunctions.h that the caller is responsible for the integrity of it is something that is coming up. Making an example of that in the unit tests is a good place to show that. I'll split this issue into two for better visibility.