ExoPlayer signals `Player.STATE_ENDED` with MPD transition from dynamic to static
stevemayhew opened this issue · comments
Version
Media3 main branch
More version details
Main@1ffeafecc374ed82d94ce16fb57c23f99cb78765
Devices that reproduce the issue
Any
Devices that do not reproduce the issue
None
Reproducible in the demo app?
Yes
Use Case Background
Start Over / Catch Up (SoCu) is a use case where live playback is stored by the origin in a rolling buffer that the clients can address within a time window in order to perform a "Start Over", that is re-start live playback of an in-progress linear at the start of the program bounded to the end. This is typically done with a "start-over URL" is a SoCu URL where start_time is in the past and end_time is in the future. The origin will play this as live (dynamic MPD) until the end_time is reached then transition to VOD (static MPD). The VOD is accessible (the "Catch Up" case) for some time after the end_time, always starting as a simple static MPD.
Issue Steps
The steps to recreate the use-case are:
- Create a "start-over URL", start playing it. Player starts live as expected
- Seek back in the window to build some buffer back from live.
- Wait until program ends (end_time is in the past)
It is expected that once end_time is in the past the MPD becomes static (see background notes below on "Transitioning Live to Static"). Playback should continue and play all the way to the end of the show, with ExoPlayer it transitons to Player.STATE_ENDED
as soon as the current buffered media is exausted.
Transitioning Live to Static (On-Demand)
The DASH-IF IOP Working Group on updating the description of how to do this properly in a working group that is updating the IOP "Section 4 Liver Services". This is in the DASH-IF Interoperability WG, I am a reviewer for this and am looking at how ExoPlayer handles it.
The origin that reproduces this bug is following these guidelines (as currently written). The basic transiton to static does the following
Period@duration
to the on-demand content durationPeriod@startTime
to 0.- removes all dynamic attributes from the MPD
- adjusts
SegmentTemplate@presentationTimeOffset
to map the existing segments time to the new period start at 0.
Analysis
Note that the working group has put a "fix" for this in dash.js
, see issue Dash.js issue #3311, this "fix" is far from perfect as the fix does not appear to update the HTMLMediaElement's view of time correctly so seeks are not possible. For ExoPlayer a much more robust fix is possible as the mapping from Timeline to Period position is completely in control of ExoPlayer code.
My analysis of the ExoPlayer code shows it is handling the update of the period to a fixed duration correctly (setting the duration) but not reseting the mapping to render time correctly (so the current Period Position does not update)
Background
This background material is for readers that are not familure with how ExoPlayer handles position internally.
Positions In ExoPlayer
ExoPlayer maintains position in three coordinate spaces:
- Timeline — The Timeline abstracts one or more media items, the current playing Timeline is available to the client to determine a User Facing Position
- Window Position — The position relative to the current playing Window, this is the position returned by getCurrentPosition(), as long as an ad is not playing.
- Period Position — Stored internally as
PlaybackInfo.positionUs
, this is position relative to the current playingPeriod@startTime
for DASH (note for HLS this is 0 as HLS is "single period". Most of ExoPlayer internally maintains position in the Period Time coordinate space. Window Position maps to Period Position via thePeriod.positionInWindowUs
.
As ExoPlayer loads segments and extracts samples from them the actual queued timestamps are adjusted to Period Position as they are, for DASH the position on the sample timeline, actual EPT of the samples is adjusted to Period Position using the presentationTimeOffset
when the MediaChunk
is queued for loading. For HLS there is a single Period
that starts at time 0, and sample PTS/EPT time is normalized to this time by the timestamp master (selected by the TimestampAdjuster
)
User Facing Position
ExoPlayer exposes getCurrentPosition() to the client (UI). For non-ad playback this is the position relative to the start of the current playing Window. If the window is "live" (Window.isLive()), then the Window.windowStartTimeMs will be the time since the epoch of the start of the window. For HLS windowStartTimeMs
comes from the EXT-X-PROGRAM-DATE-TIME
, and for DASH it is the availability window
Period Position
ExoPlayer maintains multiple Periods in a set of MediaPeriodHolder
objects, these match up with DASH Periods. The methods toPeriodTime()
and fromPeriodTime()
map between render position (the Sample Timeline) and Period Position
Suggested Fix
Looking at the code in ExoPlayerImplInternal.handleMediaSourceListInfoRefreshed()
IMO the solution is to recoginze the Period Position has changed and update PlaybackInfo.positionUs
to match this, note the renderPositionUs
would not change and it would be best not to stop and/or flush the renderers as only the mapping has changed not the actual samples and segments.
Logging shows the update is seem (the change to positionInWindow
after the new DashTimeline
is copied) but this is after the position discontinuity check, here is the change to dynamic is false:
06-10 14:31:01.818 20849 21140 D ExoPlayerImplInternal: window became static
06-10 14:31:01.822 20849 21140 D ExoPlayerImplInternal: call handlePositionDiscontinuity() - periodPositionChanged: false requestedPositionChanged: true
06-10 14:31:01.825 20849 21140 D ExoPlayerImplInternal: before playbackInfo: PlaybackInfo - bufferedPosition: 537398862 position: 537363466 period: [ id: id_PT1717517582S duration: -9223372036854775807 positionInWindow: -537074566] window: [ dynamic: true placehold: false duration: 324296 positionInFirstPeriod: 537074566 windowStartTime: 1718054657500]
06-10 14:31:01.827 20849 21140 D ExoPlayerImplInternal: after playbackInfo: PlaybackInfo - bufferedPosition: 537398862 position: 537363466 period: [ id: id_PT1717517582S duration: -9223372036854775807 positionInWindow: -537074566] window: [ dynamic: true placehold: false duration: 324296 positionInFirstPeriod: 537074566 windowStartTime: 1718054657500]
06-10 14:31:01.827 20849 21140 D ExoPlayerImplInternal: renderPosition: 537363466 renderPosition.toPeriodTime: 537363466
06-10 14:31:01.828 20849 21140 D ExoPlayerImplInternal: before copyWithTimeline: PlaybackInfo - bufferedPosition: 537398862 position: 537363466 period: [ id: id_PT1717517582S duration: -9223372036854775807 positionInWindow: -537074566] window: [ dynamic: true placehold: false duration: 324296 positionInFirstPeriod: 537074566 windowStartTime: 1718054657500]
06-10 14:31:01.828 20849 21140 D ExoPlayerImplInternal: after copyWithTimeline: PlaybackInfo - bufferedPosition: 537398862 position: 537363466 period: [ id: id_PT1717517582S duration: 361000 positionInWindow: -28] window: [ dynamic: false placehold: false duration: 354326 positionInFirstPeriod: 28 windowStartTime: -9223372036854775807]
06-10 14:31:01.829 20849 21140 D ExoPlayerImplInternal: renderPosition: 537363466 renderPosition.toPeriodTime: 537363466
06-10 14:31:02.599 20849 20849 D EventLogger: timeline [eventTime=105.82, mediaPos=537363.44, buffered=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
06-10 14:31:02.600 20849 20849 D EventLogger: period [361.00]
06-10 14:31:02.600 20849 20849 D EventLogger: window [354.33, seekable=true, dynamic=false]
06-10 14:31:02.600 20849 20849 D EventLogger: ]
Note this is proved out pretty simply by issuing a seek right after the source update,
06-10 14:52:10.242 20849 20849 D EventLogger: positionDiscontinuity [eventTime=121.31, mediaPos=254.00, buffered=0.00, window=0, period=0, reason=SEEK, PositionInfo:old [window=0, period=0, pos=538644269], PositionInfo:new [window=0, period=0, pos=254000]]
06-10 14:52:10.245 20849 20849 I EventLogger: state [eventTime=121.31, mediaPos=254.00, buffered=0.00, window=0, period=0, BUFFERING]
06-10 14:52:10.274 20849 20849 D EventLogger: isPlaying [eventTime=121.34, mediaPos=254.00, buffered=0.00, window=0, period=0, false]
The pull request pull #1451 basically does this.
Expected result
Media plays to the end.
Actual result
It is expected that once end_time is in the past the MPD becomes static (see background notes below on "Transitioning Live to Static"). Playback should continue and play all the way to the end of the show, with ExoPlayer it transitons to Player.STATE_ENDED
as soon as the current buffered media is exausted.
Media
I can send a sample URL with instructions how to set start/end time.
Bug Report
- You will email the zip file produced by
adb bugreport
to android-media-github@google.com after filing this issue.
Thanks for the detailed explanation :) I'm still not fully sure I understand everything correctly. Let me rephrase what I understand and maybe you can point out where I got this wrong if needed:
- The live MPD keeps updating as usual. It has a period starting at a fixed point in time, but the live window only allows to play the last N seconds or keeps growing to allow 'catch-up'. Notably, the available live window has an offset in the period (which I think is often corresponding to UTC timestamps, but can be arbitrary).
- The switch to a static MPD keeps the same period(s), but changes the availability window to cover all the content. Now for the interesting part:
- Generally, the assumption is that timestamps in the periods always match to the same content. This I think is the reason the IOP lists "
SegmentTemplate@presentationTimeOffset
does not change" as a requirement for MPD updates, which is also mentioned in the main ISO 23009-1 (although there is just says "any Representation shall provide functionally identical Segments with the same indices in the corresponding Representation", no mention ofpresentationTimeOffset
). - Contrary to that, you described above that
SegmentTemplate@presentationTimeOffset
does change and this is also what I can see in your logs (changingpositionInWindow
despite the samewindow.durationUs
). So I think this also means the same period timestamp no longer corresponds to the same content? - The IOP rules also say that "In a static MPD, the first period starts at the time zero of the MPD timeline.". This is interesting because as soon as you change the type to 'static' it becomes impossible to both start the period at time zero and keep the
SegmentTemplate@presentationTimeOffset
unchanged. - The new Workgroup proposal tries to fix this conundrum by allowing an update to
SegmentTemplate@presentationTimeOffset
to let the period start at zero (I think, don't have access to the linked WG document).
- Generally, the assumption is that timestamps in the periods always match to the same content. This I think is the reason the IOP lists "
- ExoPlayer's playback is likely broken by the fact that it keeps its assumption of "same timestamp in period" == "same content". And the proposed fix PR tries to change that by updating the playback position at the point where this specific transition is detected.
If all of that makes sense, I think the fix needs to be in the DashManifestParser
or maybe rather in the logic that handles the update in DashMediaSource
.
- We should definitely not violate the assumption of "same timestamp in period" == "same content". Doing that opens a whole can of worms of potential pitfalls and special cases we might need to add in the future. Your PR for example may cause problems for other non-DASH streams that see a similar dynamic->static update.
- Instead, we should be able to keep the old timing model intact and consistent with what ExoPlayer already published. That is, keep the
Period.positionInWindow
as it was before and only update the duration of the period/window as needed. This still needs some thought to cover cases like re-preparing the player after an error (which likely needs to keep this offset that is no longer in the MPD at all).
Could you confirm this matches your understanding? And if you agree on the proposed fix, we are happy to look into another PR to implement that.