The web-platform-tests tracking issue
orottier opened this issue · comments
The wpt suite was introduced to this lib in #42. It runs without crashing now and our aim is to improve the test score over time.
This tracking issue lists the high level TODOs. Single test failures are probably best reported and fixed with separate issues (either in the node bindings or the rust library).
- get rid of the wpt fork (how to deal with
webaudio/resources/buffer-loader.js, reinstate 8 deleted files) - how to handle the
crashtesttests? - look into idlharness suite #86
- collect results of the browser implementations to determine baseline performance
- implement major missing functionality (#31 #28 ...)
- improve reported errors (see #8 #54 )
- check reporter, it seems to count some log lines as errors e.g.
× Executing "test" - focus on foundational test failures (
the-audionode-interface,the-audioparam-interface,the-audiocontext-interface,the-offlineaudiocontext-interface,processing-model,the-audiobuffer-interface) - focus on other test failures
- re-enable 'the-audiocontext-interface/suspend-with-navigation.html'
- re-enable 'the-audionode-interface/audionode-disconnect-audioparam.html'
- re-enable 'the-audionode-interface/audionode-disconnect.html'
Thanks for the follow up, two notes:
get rid of the wpt fork (how to deal with webaudio/resources/buffer-loader.js, reinstate 8 deleted files)
I think this can be done by
- Implementing the
context.decodeAudioData(response, successCallback, errorCallback)signature, which should be rather simple to do on the JS side - Providing a mock to
XMLHttpRequestto the runner using rawfs.readFileSyncinternally
collect results of the browser implementations to determine baseline performance
Maybe a bit old but there is this report: https://webaudio.github.io/web-audio-api/implementation-report.html
I have submitted a PR over at wpt for the buffer-loader.js: web-platform-tests/wpt#43824
Let's see where that gets us
results from #42
RESULTS:
- # pass: 3413 (+27)
- # fail: 1353 (-27)
- # type error issues: 235 (=)
files ignored: 8 (=)
results with #61
RESULTS:
- # pass: 3605 (+192)
- # fail: 1284 (-69)
- # type error issues: 234 (-1)
results with #66
RESULTS:
- # pass: 3778 (+173)
- # fail: 1290 (+6)
- # type error issues: 59 (-175)
(the SNR is improving)
Results with v0.41.1 (no PR needed here due to patch release, just cargo update)
RESULTS:
- # pass: 3858 (+80)
- # fail: 1216 (-74)
- # type error issues: 62 (+3)
Results from #70
RESULTS:
- # pass: 3856 (-2)
- # fail: 1226 (+10)
- # type error issues: 63 (+1)
- # files ignored: 3 (-5)
Results from #76 (webaudio-rs v0.42 mainly):
RESULTS:
- # pass: 4304 (+448)
- # fail: 782 (-444)
- # type error issues: 59 (-4)
- # files ignored: 3 (=)
Results from 4b2dc95
RESULTS:
- # pass: 4394 (+90)
- # fail: 712 (-80)
- # type error issues: 57 (-2)
Results from other browsers.
Interesting to see they hit so many more assertions. Probably makes sense if there are worklet-node assertions for example. For more detail I will collect results per subdirectory (the-blabla-interface) and make another comparison.
Chrome 122.0.6261
./wpt run --log-mach=- --log-mach-level=info --headless --binary /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome chrome webaudio/
web-platform-test
~~~~~~~~~~~~~~~~~
Ran 8278 checks (8021 subtests, 257 tests)
Expected results: 8208
Unexpected results: 70
test: 3 (2 error, 1 timeout)
subtest: 67 (64 fail, 1 notrun, 1 precondition_failed, 1 timeout)
Firefox 123.0
./wpt run --log-mach=- --log-mach-level=info --headless --binary /Applications/Firefox\ Nightly.app/Contents/MacOS/firefox firefox webaudio/
web-platform-test
~~~~~~~~~~~~~~~~~
Ran 7662 checks (7405 subtests, 257 tests)
Expected results: 7097
Unexpected results: 565
test: 10 (9 error, 1 timeout)
subtest: 555 (526 fail, 24 notrun, 5 timeout)
Safari 17.3.1
./wpt run --log-mach=- --log-mach-level=info --headless --channel stable safari webaudio/
web-platform-test
~~~~~~~~~~~~~~~~~
Ran 8268 checks (8011 subtests, 257 tests)
Expected results: 8158
Unexpected results: 110
test: 14 (5 error, 9 timeout)
subtest: 96 (66 fail, 24 notrun, 6 timeout)
Results from #82 #85 and orottier/web-audio-api-rs#455 orottier/web-audio-api-rs#454 orottier/web-audio-api-rs#452 orottier/web-audio-api-rs#456
RESULTS:
- # pass: 4467 (+73)
- # fail: 736 (+24)
- # type error issues: 58 (+1)
- # files ignored: 2 (-1)
I have made the initial quantitative comparison now per directory (per node)
Methodology:
- our score = # pass + # type error
- total = the reported number of subtests via the official wpt runner
I have filed an issue over at wpt to remove the empty dirs: web-platform-tests/wpt#44749
Cool, quite interesting
With rust AudioParam PRs merged orottier/web-audio-api-rs#459 orottier/web-audio-api-rs#460
RESULTS:
- # pass: 5009 (+237)
- # fail: 659 (-393)
- # type error issues: 54 (=)
- # files ignored: 2 (=)
w/ #90
RESULTS:
- # pass: 5278
- # fail: 591
- # type error issues: 46
w/ update rc.3 and #103
RESULTS:
- # pass: 6240
- # fail: 1072
- # type error issues: 24
w/ #104
RESULTS:
- # pass: 6739
- # fail: 800
- # type error issues: 24
w/ #112
RESULTS:
- # pass: 6927
- # fail: 677
- # type error issues: 4
> wpt duration: 1:50.123 (m:ss.mmm)
v0.20.0
RESULTS:
- # pass: 7005
- # fail: 618
- # type error issues: 4
> wpt duration: 2:15.084 (m:ss.mmm)
The results for the AudioWorkletNode on my machine (with the latest fix for zero-output nodes) are
npm run wpt -- --filter the-audioworklet-interface
RESULTS:
- # pass: 259
- # fail: 41
- # type error issues: 0
> wpt duration: 2:57.474 (m:ss.mmm)
But they are plagued with messages Cannot read properties of undefined (reading 'getElementsByTagName'). Should we port any browser method to NodeJs again?
But they are plagued with messages Cannot read properties of undefined (reading 'getElementsByTagName'). Should we port any browser method to NodeJs again?
Yup I guess something like but I could find where this comes from yet...
/w #132
RESULTS:
- # pass: 7316
- # fail: 528
- # type error issues: 3
> wpt duration: 3:00.313 (m:ss.mmm)