Discrepancies between Conv layer outputs of the ONNX model and converted TensorFlow models
SuhwanSong opened this issue · comments
Issue Type
Feature Request
OS
Linux
onnx2tf version number
1.19.14
onnx version number
1.15.0
onnxruntime version number
1.17.0
onnxsim (onnx_simplifier) version number
0.4.33
tensorflow version number
2.15.0
Download URL for ONNX
https://github.com/PINTO0309/onnx2tf/files/14982911/poc.zip
Parameter Replacement JSON
{}
Description
When converting an ONNX model to TensorFlow Lite (TFLite) format and comparing the outputs of the ONNX model, TensorFlow model, and TFLite model, discrepancies are observed between the ONNX model and both TensorFlow models.
-
Expected Behavior
The outputs of the ONNX model, TensorFlow model, and TensorFlow Lite model should match. -
Actual Behavior
There are discrepancies between the outputs of the ONNX model and converted TensorFlow models.
Steps to Reproduce
- Download and unzip "poc.zip' to get "poc.onnx".
- Run the following code with poc file.
onnx2tf -i ./poc.onnx -cotof
![image](https://private-user-images.githubusercontent.com/41769825/322585955-82fb78d4-0f2f-40eb-8a5b-fc1619becd02.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjMyODM0MjcsIm5iZiI6MTcyMzI4MzEyNywicGF0aCI6Ii80MTc2OTgyNS8zMjI1ODU5NTUtODJmYjc4ZDQtMGYyZi00MGViLThhNWItZmMxNjE5YmVjZDAyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODEwVDA5NDUyN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQ0ZTdjYTkxYWIyMDQ1N2U2NzZmZGUzM2VlOGFiMzBlMmUyMjU0YWY0OTgyZGQzYjRhOGFiYzUzMGM3ZGYwNjAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.HV-6QDioDAMovRZAjJzbC00R_f88L7VJLntZv87M_ko)
Screenshots
Why
Solving discrepancies between the outputs of the ONNX model and both TensorFlow models will ensure consistent inference results across different runtime environments. This consistency is crucial for deploying machine learning models in production systems, where uniform behavior is expected regardless of the framework or format used.
Unfortunately, there is always an error of about 1e-4 due to differences in run-time calculation specifications.
If you can't tolerate that minor error, don't use tflite.
Unfortunately, there is always an error of about 1e-4 due to differences in run-time calculation specifications.
If you can't tolerate that minor error, don't use tflite.
Thanks :)
Sorry for bothering you but there is a discrepancy between the outputs of the TensorFlow model and the ONNX model.
Note that the outputs of TensorFlow Lite and TensorFlow models match.
Here's the new poc file
poc.zip
from os.path import join
import tensorflow
import onnxruntime
import numpy as np
import onnx2tf
from einops import rearrange
if __name__ == "__main__" :
onnx_model_path = 'poc.onnx'
tf_output_path = './tf_path'
# Convert ONNX model into TensorFlow
onnx2tf.convert(
input_onnx_file_path=onnx_model_path,
output_folder_path=tf_output_path,
copy_onnx_input_output_names_to_tflite=True,
non_verbose=True,
)
# input
input_np = np.random.randn(1, 3, 224, 224).astype('f')
# load and run onnx model
ort_session = onnxruntime.InferenceSession(onnx_model_path)
ort_output = ort_session.run(None, {'x' : input_np})
# Prepare input for TensorFlow models
input_for_tf = rearrange(input_np, 'b c h w -> b h w c')
interpreter = tensorflow.lite.Interpreter(join(tf_output_path, 'poc_float32.tflite'))
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
interpreter.allocate_tensors()
interpreter.set_tensor(input_details[0]['index'], input_for_tf)
interpreter.invoke()
output_data = interpreter.get_tensor(output_details[0]['index'])
#print (output_data)
# Load and run TensorFlow model
tf_model = tensorflow.saved_model.load(tf_output_path)
tf_output = tf_model(input_for_tf)
# Compare ONNX and TensorFlow outputs
if np.allclose(ort_output, tf_output, rtol=1e-03, atol=1e-04):
print("Test Passed: ONNX and TensorFlow outputs match\n")
else:
print("Test Failed: ONNX and TensorFlow outputs differ\n")
# Compare TFlite and TensorFlow outputs
if np.allclose(output_data, tf_output, rtol=1e-03, atol=1e-04):
print("Test Passed: TFlite and TensorFlow outputs match\n")
else:
print("Test Failed: TFlite and TensorFlow outputs differ\n")
Sorry, but I don't understand the point you are trying to make. The final outputs of ONNX and TFLite match.
pip show onnx2tf
Name: onnx2tf
Version: 1.19.15
onnx2tf -i poc.onnx -cotof