Fail to append buffer for multiplexed content with discontinuity
michellezhuogg opened this issue · comments
Shaka Player uses mux.js to convert TS to MP4. For multiplexed content with discontinuity, we noticed that SourceBuffer cannot append the buffers after the discontinuity tags successfully. This might be caused by audio streams getting trimmed if there's also video, based on the original timestamps.
https://github.com/videojs/mux.js/blob/master/lib/mp4/transmuxer.js#L1016
I've attached two pieces of contents clipped from a live stream playlist, one right before EXT-X-DISCONTINUITY tag, and the other one right after the tag for your reference.
We found that because the first segment after the discontinuity has a smaller timestamp than the first segment we appended, the post-discontinuity segment has its audio completely removed by mux.js.
We use the keepOriginalTimestamps flag, and it seems to me that this behavior is not in line with that flag conceptually. You're throwing away content based on the timestamps, whereas we want to be in complete control of the timestamps.
So we came up with this potential fix, to skip this behavior if the keepOriginalTimestamps flag is set. If this is acceptable, I'd be happy to send it as a PR. If not, we'd be happy to have an alternate fix from y'all.
For now, we'll use our fork to unblock our own launch goals, but we would strongly prefer to ship with an official release long-term.
Thanks for posting, and for sharing a potential fix! We're going over some of our logic and trying to determine the best approach. I think the PR may be the route to go, but we want to do a couple tests and reviews first.
Sounds great. Thanks for letting us know!
We looked at the code again and you're right, it is definitely a subtle form of timestamp manipulation. The video start is sent as the earliestAllowedDts to the audio track, which trims frames according to that value. While the frames don’t change their times, we do recalculate the baseMediaDecodeTime based on the new earliest frame value.
So the values are all technically the same, but the baseMediaDecodeTime is different (and there are fewer frames). I think that’s enough to be considered timestamp manipulation, which you're right, we shouldn't do when keepOriginalTimestamps is true.
We'd definitely appreciate the PR. And thank you!
Our pleasure! Thank you for taking time to look into this with us.
I created a PR. Please let me know if I overlooked any processes or rules I need to follow.
Also, please let me know if there's anything I can do to make it easier for y'all to push a new release after this. We're very eager to ship with standard mux.js releases instead of a fork. Thanks!
Released in v5.5.3. Thanks again!
Excellent! Thanks so much!