Allow play/pause/seek control when streaming recorded data through web socket
vincentyang-plus opened this issue · comments
When streaming recorded data with web socket, we also like to seek back and forth in the recorded timeline or pause the streaming to explore the data.
Recorded bag data do have start and end time, enable play bar will give users the same experience as play a bag file.
#38 is a related feature request.
This would be helpful for the new mcap-play example!
#299 is one incremental step closer to this. The server can now tell the client what time it is independently of sending messages on subscribed topics, allowing time to be paused, stepped, seeking, etc. This makes play/pause/seek on the server side possible. The next step would be a way for the client to instruct whatever is doing playback to play/pause/seek. This likely wouldn't be part of the ws-protocol specification itself, but an agreed-upon topic and message format for playback control Studio could publish to and "rosbag play" (or an equivalent node) would subscribe to.
There are a couple more methods required than just play/pause/seek.
Speaking from experience with implementing the server ends of playback, knowing the desired playback direction and target speed on seek is essential for correctly opening the file. Minimizing the number of necessary state changes on the server side is key.
An example for the following interface design can be found e.g. in the ONVIF extensions of RTSP - so it's well proven. Deviations from the reference are based on experience with difficulties to get the server side implemented correctly, there are a couple of traps where the "correct" behavior requires way to much bookkeeping.
When there is a play method, it should always have a "speed" (setting it to 0 is pause - so pause is redundant!) parameter as well as an "position" parameter (which also renders a dedicated seek method irrelevant).
Omitting either parameter proceeds with current speed / direction.
When opening a file, a compliant server is expected to start at 1.0x playback speed at the start of the track for finished recordings, respectively at the "live" end for ongoing recordings.
A "speed" parameter of > 1.0x or < -1.0x denotes a request for faster-than-realtime playback.
It's important to understand that there are 2 ways to handle requests for high speed playback. Either stable clock rates or reliable transmission can be achieved (both applies to the server as well as the client by back-pressure). Generally, a sophisticated "best effort" strategy omitting redundant data will yield the best user experience. However, for some use cases (such as briding data between systems) even at off-rate playback a "reliable" transmission may need to be enforced.
When in doubt, define the "reliable" strategy as default and consider an optional "best effort" strategy which may be requested by the client explicitly.
The "speed" parameter requires special values for "+Inf" and "-inf". Even though the latter two are pretty much useless for something like the 3D panels, such settings are great for populating any form of plot.
+Inf is semantically no the same as "go very fast". E.g. video Payback at requested 100x speed with "best effort" may efficiently skip all frames but I-frames, but +inf with "reliable" strategy is an explicit request for all data frames with no skips, throttled by the client only!
The "position" parameter requires special values for "end of track" and "begin of track" - do not fall for the wrong assumption that these could be queried. Some server implementations may provide a transparent switch between playback and live with continuous recording, so both track boundaries are shifting continuously.
The server must provide a notification when playback has paused due to reaching a boundary. This is essential in order for the client to schedule any form of playlist.
The server also needs to issue a notification when performing a play command, with the actual position, limit and speed it accepted.
Do not attempt to implement something like "loop mode" on the server - it will result in an extremely poor user experience long term. General rule is that the client must be ready to handle a seek before receiving looped data (e.g. historic state needs to be discarded first), so the server must never loop or seek in reverse to the current playback direction without an explicit request.
Unlike the RTSP PLAY method we are using as a reference, do not attempt to implement "queued commands" on the server side - it's way too difficult to implement correctly and most implementations will be unable to conform. That makes all play method requests "immediately".
Issuing a new navigation request while the previous one has not been aknowled may be left as explicitly undefined behavior - inconsistent server behavior would otherwise to be expected. When in doubt, the client shall therefore make fully qualified play requests when in an unclear state.
In addition to the "position" parameter, the play method may require a "limit" parameter which denotes an upper bound for playback.
This is essential for all types of displays which need to accumulate data from a time period in order to show as much as a snapshot. E.g. jumping to time X in a complex 3D scene may require message from time [X - 10s, X]. The server should support unrestricted seek, limit and +Inf playback speed for such an application.
It's also necessary for the client to query some parameters from the server. Specifically, the server needs to describe:
- track start (inclusive) - may become invalid, just for reference in time lines
- track end (inclusive) - may become invalid, may require special value to denote "always up to now" for ongoing recordings
- supported playback speeds (list or value range). Some servers will be able to support off-rate or even reverse playback, others can't! Some will not even be able to provide pause.
- seek support - some servers can seek arbitrarily, for some it's extremely expensive (discouraged!) and others can't seek at all.
- supported off-rate playback strategies
A useful extension is remote file picking.
At minimum, the server needs to inform about the currently loaded file(s). A file consists not only of a URI, but also of an optional human readable name, track boundaries as well as unstructured key/value meta data.
In order to provide file picking, the server needs the capability to describe files it could offer.
If the server has the capability to open multiple files at once for synchronous playback, it needs to announce that.
The method for opening files by URI must cover that "open multiple files at once" case.
Opening files should optionally include the same parameters as a play request, so that an potentially expensive state change on the server can be avoided. Alternatively, the file list may be passed as an optional argument to the play method.
We would be really interested in that feature.