Issue in HEXL_CHECK_BOUNDS debug check in avx512 reduction when output_mod_factor is 2.
jcalafato1 opened this issue · comments
Hello,
I believe there is an issue in the check found at: hexl/eltwise/eltwise-reduce-mod-avx512.hpp:84. The code reads as follows:
if (input_mod_factor == modulus) { if (output_mod_factor == 2) { for (size_t i = 0; i < n_tmp; i += 8) { __m512i v_op = _mm512_loadu_si512(v_operand); v_op = _mm512_hexl_barrett_reduce64<BitShift, 2>( v_op, v_modulus, v_bf, v_bf_52, prod_right_shift, v_neg_mod); HEXL_CHECK_BOUNDS(ExtractValues(v_op).data(), 8, modulus, "v_op exceeds bound " << modulus); _mm512_storeu_si512(v_result, v_op); ++v_operand; ++v_result; } ...
This check will fail even though the operations succeeds. I believe you should be checking against modulus << 1u instead of against modulus, since the output is expected to be within [0, 2*modulus). I'm not sure if there are other problems across the codebase with these checks, but it is difficult to debug issues when these things are throwing.
Hello @jcalafato1, Sorry, I guess you already found a way around this issue. This does look like a bug as the output_mod_factor in this case is 2. However, do you have an input example that actually give you a result over the modulus value, [0, 2*modulus)?