Unexpected performance issues with AOT compilation

[Matthias Brunel]

Hi there,

I benchmarked resnet50 in onnx format against onnxruntime. I found out on an Azuredsv3 server that the inference ran in 200ms for glow with AOT against 50ms for onnxruntime. Should I expect this?

I then tried to use the optimisations flags of the model-compiler (in release mode) but none of them seemed to do anything at all. Also passing flags like "-march=native" made the compiler segfault. How should the model-compiler be used ?

I also tried to use model-compiler to generate a library that doesn't use AVX512. I didn't find how to do it via flags, so I recompiled on a host without AVX512 support. I used that library on the Azure server (with AVX512 support), and actually got better performance (inference took in average 187ms). It looks like it is best not to use AVX512 by default. Is there a way to disable AVX 512 properly ?

Sorry if this looks like a rant, if you think I made some mistakes, I would be very glad to hear about it, because I'm interested in using your project.