Evaluated classifications onnx models almost with zero accuracy
Apisteftos opened this issue · comments
I evaluated all of the classifications models according to their preprocessing description with imagenet:
Models:
- squeezenet1.0-12.onnx
- bvlcalexnet-12.onnx
- caffenet-12.onnx
- rcnn-ilsvrc13-9.onnx
- inception-v1-12.onnx
- inception-v2-9.onnx
- zfnet512-12.onnx
Why should this models being in the model zoo if their behavior and accuracy is terrible?
Any updates?
Hi, there are many different versions for the same model and they are from more than one contributors.
We submitted part of them previously and verified the accuracy with our tools.
Since there are others contributions, we only add our instruction and keep others' instructions.
You can find the details of how to reproduce the result step by step in https://github.com/intel/neural-compressor/tree/master/examples/onnxrt/image_recognition/onnx_model_zoo, https://github.com/intel/neural-compressor/tree/master/examples/onnxrt/object_detection/onnx_model_zoo, https://github.com/intel/neural-compressor/tree/master/examples/onnxrt/body_analysis/onnx_model_zoo and https://github.com/intel/neural-compressor/tree/master/examples/onnxrt/nlp/onnx_model_zoo.
Hi @mengniwang95 thank you for your feedback. I tried following the pre-processing in the links you sent but that was either non-existent or didn't work. Could you please verify a model with issue such as inception-v1-12? According to the table here, the accuracy for that model should be 67.23% but I'm getting close to 0%. Thank you.