Giters

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

Home Page:https://onnxruntime.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Web] Significant difference between Python and Web outputs

josephrocca opened this issue · comments

commented

Describe the issue

This model has significantly different outputs depending on whether I use ONNX Python or ONNX Runtime Web.

Python:	[2.107932	3.161898	7.904746	1.844440	0.000000	-15.019016	6.850779	9.222202	5.269830	-7.377762	-3.688881	4.479355	-3.952373	5.796813	-1.580949	2.371424	-6.060305	-2.107932	0.526983	4.479355	3.688881	1.844440	-2.634915	2.634915	5.533322	-1.580949	-4.479356	0.000000	-2.371424	0.790474	-3.425390	2.898407	-1.317458	2.634915	0.000000	10.539660	-3.688881	-2.634915	-2.634915	-3.425390	-5.006339	-1.844440	0.526983	-1.844440	-2.107932	18.444405	2.107932	0.000000	3.688881	-5.269830	2.634915	1.844440	1.317458	2.107932	0.790474	-1.580949	-4.479356	-2.107932	4.215865	10.803152	-1.580949	-7.377762	14.492033	3.161898]
Web:	[2.898407	2.371424	3.952373	0.526983	-0.263492	-6.587288	3.952373	5.533322	0.526983	-3.688881	0.263492	2.634915	-3.161898	-2.107932	-1.053966	3.161898	-0.263492	0.000000	0.000000	0.000000	0.000000	0.790475	-1.317458	-1.053966	0.263492	0.526983	-1.053966	-0.263492	-1.317458	-0.526983	-0.263492	0.000000	-0.790475	1.053966	-0.263492	4.215864	-2.634915	-2.107932	-3.161898	-1.053966	-4.215864	0.526983	-0.526983	0.526983	0.000000	6.587288	1.317458	2.107932	0.526983	-2.371424	0.000000	1.317458	0.263492	-1.844441	0.000000	-1.580949	-0.263492	1.844441	-0.790475	0.000000	-0.790475	-2.371424	2.898407	1.053966]

To reproduce

Urgency

No hard deadlines. This blocks conversion of Google's new Lyra V2 model to work in the browser.

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.13.1

Execution Provider

WASM

commented

Hi @josephrocca,

Thanks for the well documented issue. I was able to reproduce the problem on my local machine as well.
I am debugging the problem and I will keep you updated with my findings.

BR,
Shalva

commented

Hi @josephrocca,

I have found a way to unblock you for now - I was able to reproduce the python results on WASM using a converted model.
All you need to do is to convert the ONNX model to an ORT model using this command - doc link

python -m onnxruntime.tools.convert_onnx_models_to_ort <onnx model file or dir>

after conversion all you need to do is to change the model file name from "soundstream_encoder.onnx" to "soundstream_encoder.with_runtime_opt.ort" in the session creation API.

I am still looking into the differences in output. But this find gives us a good indication that the issue is related to the graph optimization.

Please let me know if you have issue reproducing the python output using WASM on your end.

BR,
Shalva

commented

Thanks @shalvamist! That has fixed 3 out of the 4 models that I'm trying to get working - the outputs are exactly the same. However, the fourth model's outputs are still different:

Python:	-0.010311	-0.010905	-0.009789	-0.008309	-0.006717	-0.004137	-0.003409	-0.004980	-0.005251	-0.004721	-0.005261	-0.005853	-0.005381	-0.005115	-0.005928	-0.006925	-0.007212	-0.006463	-0.005685	-0.005797	-0.005958	-0.006103	-0.006542	-0.007077	-0.007207	-0.007065	-0.006971	-0.007442	-0.008179	-0.007883	-0.007552	-0.007236	-0.006697	-0.006460	-0.006765	-0.007334	-0.007280	-0.006637	-0.005996	-0.006260	-0.007017	-0.007709	-0.008267	-0.008697	-0.008366	-0.008220	-0.008015	-0.007978	-0.007872	-0.007843	-0.008012	-0.008417	-0.009045	-0.009409	-0.009244	-0.009423	-0.009672	-0.009493	-0.009145	-0.008602	-0.008152	-0.007736	-0.008264	-0.008783	-0.008747	-0.008961	-0.009408	-0.009703	-0.009473	-0.009041	-0.008715	-0.008475	-0.007979	-0.007757	-0.007796	-0.007646	-0.007783	-0.008491	-0.009041	-0.009588	-0.009899	-0.010044	-0.010022	-0.009699	-0.009106	-0.008733	-0.008107	-0.007758	-0.007865	-0.008195	-0.008993	-0.009687	-0.010184	-0.010315	-0.010075	-0.009981	-0.009448	-0.008546	-0.007576	-0.006843	-0.006724	-0.006901	-0.007362	-0.007594	-0.007234	-0.006818	-0.006294	-0.005860	-0.005695	-0.005511	-0.005165	-0.004944	-0.004755	-0.004884	-0.005296	-0.005626	-0.006009	-0.006287	-0.006847	-0.007051	-0.006904	-0.006761	-0.006570	-0.006155	-0.006050	-0.006026	-0.006543	-0.007119	-0.007489	-0.007827	-0.007788	-0.007541	-0.006916	-0.005807	-0.004797	-0.003785	-0.003431	-0.003451	-0.003890	-0.004367	-0.004890	-0.005455	-0.005575	-0.005498	-0.005066	-0.004628	-0.004104	-0.003680	-0.003675	-0.003712	-0.003724	-0.003823	-0.003987	-0.004335	-0.004607	-0.004825	-0.004973	-0.004941	-0.004740	-0.004204	-0.003310	-0.002924	-0.002564	-0.002157	-0.002059	-0.001838	-0.001745	-0.001762	-0.001904	-0.001670	-0.001343	-0.000908	-0.000629	-0.000569	-0.000387	-0.000138	-0.000028	-0.000132	-0.000409	-0.000782	-0.000989	-0.001427	-0.001653	-0.001449	-0.001124	-0.000607	0.000135	0.000642	0.000640	0.000485	0.000333	0.000010	-0.000438	-0.000675	-0.000905	-0.001050	-0.001037	-0.000420	0.000041	0.000418	0.000748	0.000974	0.001049	0.001053	0.000752	0.000505	0.000342	0.000289	0.000411	0.000477	0.000644	0.000851	0.001220	0.001179	0.001222	0.001437	0.001121	0.000714	0.000527	0.000454	0.000451	0.000470	0.000650	0.000716	0.000980	0.001058	0.001283	0.001201	0.001201	0.001550	0.001549	0.001878	0.001820	0.001701	0.001951	0.002036	0.002271	0.002607	0.002694	0.002814	0.003071	0.003081	0.002845	0.002864	0.002552	0.002118	0.001953	0.001698	0.001419	0.001203	0.001312	0.001475	0.001317	0.001470	0.001372	0.001060	0.001032	0.001253	0.001216	0.000913	0.000816	0.000765	0.000453	0.000226	0.000475	0.000805	0.000761	0.000756	0.000935	0.001314	0.001625	0.001555	0.001336	0.000999	0.000665	0.000544	0.000494	0.000138	0.000214	0.000498	0.000604	0.000545	0.000436	0.000354	0.000190	-0.000146	-0.000443	-0.000836	-0.001083	-0.001134	-0.000759	-0.000363	-0.000319	0.000101	0.000252	0.000368	0.000513	0.000322	0.000184	0.000138	0.000133	0.000407	0.000556	0.000678	0.000630	0.000810	0.000702	0.000641	0.000553	0.000607	0.000662	0.000593	0.000576	0.000444	0.000471	0.000474	0.000120	-0.000239	-0.000191	-0.000138
Web:	-0.009051	-0.009424	-0.008333	-0.006913	-0.005499	-0.003212	-0.002629	-0.004071	-0.004286	-0.003925	-0.004252	-0.004803	-0.004376	-0.004212	-0.005135	-0.006136	-0.006586	-0.005849	-0.005128	-0.005323	-0.005597	-0.005949	-0.006424	-0.007034	-0.007078	-0.006810	-0.006753	-0.007114	-0.007780	-0.007466	-0.006916	-0.006228	-0.005434	-0.005135	-0.005333	-0.005991	-0.006178	-0.005746	-0.005187	-0.005417	-0.006069	-0.006684	-0.007166	-0.007514	-0.007085	-0.007005	-0.006888	-0.006968	-0.007107	-0.007329	-0.007842	-0.008421	-0.008840	-0.008997	-0.008746	-0.008791	-0.009067	-0.009045	-0.008947	-0.008513	-0.008051	-0.007599	-0.007988	-0.008350	-0.008261	-0.008354	-0.008741	-0.009166	-0.009161	-0.008951	-0.008757	-0.008585	-0.008029	-0.007604	-0.007390	-0.007062	-0.007098	-0.007569	-0.008030	-0.008718	-0.009333	-0.009755	-0.009997	-0.009730	-0.009066	-0.008652	-0.007913	-0.007389	-0.007112	-0.007109	-0.007801	-0.008552	-0.009288	-0.009826	-0.010043	-0.010272	-0.009915	-0.009018	-0.007802	-0.006720	-0.005994	-0.005697	-0.005896	-0.005958	-0.005792	-0.005911	-0.005941	-0.005779	-0.005753	-0.005613	-0.005121	-0.004643	-0.004189	-0.004082	-0.004364	-0.004831	-0.005343	-0.005757	-0.006547	-0.006953	-0.006891	-0.006551	-0.006420	-0.006007	-0.005659	-0.005536	-0.005866	-0.006416	-0.006848	-0.007216	-0.007198	-0.006885	-0.006439	-0.005396	-0.004092	-0.002952	-0.002545	-0.002684	-0.003122	-0.003680	-0.004450	-0.005221	-0.005643	-0.005679	-0.005057	-0.004188	-0.003270	-0.002702	-0.002480	-0.002449	-0.002587	-0.003131	-0.003713	-0.004237	-0.004537	-0.004666	-0.004657	-0.004422	-0.003920	-0.003150	-0.002255	-0.002031	-0.001850	-0.001744	-0.001851	-0.001704	-0.001598	-0.001640	-0.001612	-0.001275	-0.001032	-0.000700	-0.000426	-0.000324	-0.000025	0.000301	0.000582	0.000559	0.000204	-0.000350	-0.000782	-0.001275	-0.001460	-0.001348	-0.000870	-0.000158	0.000608	0.001079	0.001094	0.000976	0.000663	0.000227	-0.000250	-0.000455	-0.000642	-0.000731	-0.000614	-0.000010	0.000364	0.000797	0.001153	0.001329	0.001446	0.001500	0.000998	0.000487	0.000217	0.000342	0.000455	0.000508	0.000686	0.000630	0.000968	0.000876	0.000958	0.001216	0.000868	0.000504	0.000285	0.000295	0.000402	0.000410	0.000501	0.000524	0.000779	0.000975	0.001275	0.001154	0.001063	0.001125	0.001010	0.001448	0.001313	0.001270	0.001600	0.001839	0.002410	0.003032	0.003155	0.003168	0.003304	0.003171	0.002596	0.002393	0.002113	0.001758	0.001651	0.001608	0.001501	0.001298	0.001409	0.001561	0.001307	0.001133	0.000778	0.000329	0.000017	0.000202	0.000278	0.000080	0.000367	0.000600	0.000511	0.000374	0.000652	0.001003	0.000886	0.000685	0.000670	0.000976	0.001286	0.001158	0.000843	0.000537	0.000395	0.000508	0.000592	0.000348	0.000331	0.000461	0.000467	0.000278	0.000006	0.000004	-0.000007	-0.000333	-0.000615	-0.000907	-0.001095	-0.001073	-0.000637	-0.000313	-0.000368	-0.000094	-0.000095	-0.000060	0.000058	-0.000117	-0.000155	0.000015	0.000144	0.000449	0.000690	0.000946	0.000766	0.000879	0.000611	0.000320	0.000258	0.000322	0.000356	0.000404	0.000509	0.000423	0.000509	0.000354	-0.000307	-0.000517	-0.000401	-0.000470
commented

Hey @josephrocca,

Thanks for the update, happy to hear most of the issues are bypassed.
I identified 2 separate issues here -

  • Model graph optimization isn't consistent between ORT python & WASM
  • The last model you mentioned needs additional investigation (might be optimization/quantization/op order not sure yet)

Right now, I am focusing on the first issue since I already have a good lead on what might be wrong.
once that is resolved I will move on to the next item.

I will update as soon as I have more details to share.

commented

Hey @josephrocca,

As small update - I confirmed that the first issue is connected to the optimization level used in the ORT session.
You can confirm that on your end as well just updated this line in the HTML.
Incorrect:

let encoderModel = await ort.InferenceSession.create("./soundstream_encoder.onnx", { executionProviders: ["wasm"], graphOptimizationLevel:"all"});

Correct:

let encoderModel = await ort.InferenceSession.create("./soundstream_encoder.onnx", { executionProviders: ["wasm"], graphOptimizationLevel:"extended"});

BTW, the issue is with WASM only (python is ok with "all" optimization level - if you feel like testing that as well - link
I am isolating the optimization transform right now - I think I should have it identified next week.
I will keep updating you.

BR & Happy holidays

commented

Hi All,

Hope you had a good new year's holiday!

I have an update about the original issue reported - difference in output between python and WASM inferencing.
We pinpointed the problem with the convolution implementation - it was fixed with this PR -
#14008

This should be part of the next official release. If you want to use the fix right now, you can pull the latest onnxruntime repo and build WASM using this script - #12643
I tested on my end, and it resolved the problem on WASM.

commented

this should be fixed in 1.14