is D in your fomation (2) calculated by this?
mittkongg opened this issue · comments
max = scale * (255 - zero_point);
min = scale * (0 - zero_point);
and this zero_point and scale both are real number , so is not calculated like what i do ?
Low-precision matrix multiplication
mittkongg opened this issue · comments
max = scale * (255 - zero_point);
min = scale * (0 - zero_point);
and this zero_point and scale both are real number , so is not calculated like what i do ?