Need a way to determine whether rendering is falling below target frame rate
cvan opened this issue · comments
Monday Apr 04, 2016 at 00:09 GMT
Originally opened as MozillaReality/webvr-spec#29
I've been thinking about the possibility of adapting render quality to achieve the optimal frame rate, as described in this article. The article mentions things like adjusting the pre-distortion render buffer scale, adjusting anti-aliasing detail and some more complicated ideas. I was also thinking about adjusting the far end of z-clipping (with some fog) as it's appropriate for content, levels of detail, etc.
In order to do this, we need to know how well we're doing at meeting the target frame rate. The simplest way I can think of doing this is to report the frame rate of the VRDisplay device. Presumably, this would be fixed for now, but I suppose not necessarily forever. I suppose there are some arguments to be made against providing this information, but since there are annoying but feasible ways to figure it out, we may as well provide it easily and correctly.
One way that this falls short, however, is that if you are rendering at the optimal frame rate, there isn't really a good way to know if you've got a little extra time so you can bump the quality up or run some other tasks. You'd have to try raising the quality and then waiting to see if that drops the frame rate, which is certainly not ideal. If anybody has any ideas about that, I'd love to hear them.
cc: @toji, per our chat in webvr slack
Wednesday Apr 06, 2016 at 03:00 GMT
API could provide but timing of two rAF will tell you the frame rate pretty simply without any rendering required, etc. Say currently running at consistent frame rate. (frametime - rendertime) / frametime ratio <1 which can have a goal with a buffer of leaving x% of frametime.
You shouldn't have to do anything that would cause missing framerate assuming can move in suitably small increments to test is still under budget after increase. This seems like a more generic problem of performance scaling like would have targeting mobile + desktop.
I'd like to give this a bump, since I've been working on an effort to implement adaptive rendering. The case has been made that this should be left out for privacy reasons, but I'd argue against it. Having this information is critical to maintaining the frame rate necessary to keep people from getting sick. Developers will try to infer the frame rate through timing trickery, so they'll have the information anyway, but it will be error prone and not so future-proof. Those techniques may also compromise user experience since it would require rendering a few empty frames to determine the maximum possible frame rate. It's better to just provide the information accurately.
As of now, for most devices, the maximum frame rate is the target. But in the future as frame rates go much higher (120hz?), it may make sense to provide a minimum frame rate for preventing discomfort that is distinct from the maximum frame rate. I don't think we need to provide all that information right now, but it's worth keeping in mind when coming up with the API.
At the W3C workshop, @toji mentioned the idea of having requestAnimationFrame pass the amount of time left before the next draw, similar to requestIdleCallback. I think this is an excellent idea. I think it should solve this problem.
At the W3C workshop, @toji mentioned the idea of having
requestAnimationFramepass the amount of time left before the next draw, similar torequestIdleCallback. I think this is an excellent idea. I think it should solve this problem.
window.requestIdleCallback's callback function takes two parameters:
timeRemaining(a method that returns aDOMHighResTimeStamp)didTimeout(aboolean)
window.requestAnimationFrame's callback function takes only one:
time(aDOMHighResTimeStamp, which is the same as value returned bywindow.performance.now())
IMO, it would be odd to have window.requestAnimationFrame and VRDisplay.prototype.requestAnimationFrame not have the same callback signatures.
If it's not going to behave in the same manner, perhaps VRDisplay#requestAnimationFrame should be named differently. Thoughts? I'm sure @igrigorik and others have thoughts here.
@cvan That's a great point, and you're right that they should definitely have matching callback signatures. But it shouldn't match rIC, since that has a different behavior and purpose.
Maybe it makes sense to add timeRemaining as a second parameter to the rAF callback, since those callbacks are supposed to be synced to a refresh rate as well. I imagine it could be useful, though I also imagine it might require a lot more approval outside of the webvr community.
Otherwise, yeah, I'd consider picking a different name.
BTW, would it make sense for the browser to decrease the time remaining in response to overheating?
I think we need at least two different pieces of timing data since JS processing and GL rendering happen in parallel:
rAF ... (JS processing) ... submitFrame ... (GPU rendering) ... Frame done
rAF ... (JS processing) ...
Either one can be a bottleneck, and an application that wants to do dynamic quality scaling would need information about both.
The proposed "timeRemaining" argument sounds as if this would just address the JS processing side of the issue, basically telling the JS app how much time it has left before it has to call submitFrame. I think it would also be necessary to tell the application how long the post-submit rendering took for recent frames so that it can adjust resolution or other dynamic features as needed.
What would you think about adding some timing fields to getFrameData with some basic statistics about recent frame timing? I think this would potentially fit alongside the existing "timestamp" field.
The Media Playback Quality spec seems interesting to possibly extend for canvas (or strictly VR contexts):
let video = document.querySelector('video');
video.getVideoPlaybackQuality.creationTime; // DOMHighResTimeStamp (from `performance.now`)
video.getVideoPlaybackQuality.corruptedVideoFrames; // long
video.getVideoPlaybackQuality.droppedVideoFrames; // long
video.getVideoPlaybackQuality.totalVideoFrames; // long
So it doesn't get forgotten:
This was brought up at the Seattle F2F in the context of "What timing information can we reliably deliver cross-platform that is actually actionable for developers?" The most immediate, practical answer was a indication of how many dropped frames there had been since the last iteration of the animation loop. (Different platforms call this different things, it's sometimes reported as a "reprojection count".) Given the conversation we've previously had about timestamps (#347) and where additional XR-centric timing data should go, it seems like the most logical place for this is a timing interface that's given as an XRFrame attribute:
interface XRFrameTiming {
readonly attribute unsigned long droppedFrames;
}
partial interface XRFrame {
readonly attribute XRFrameTiming timing;
}Which could conceivably be used like so:
function onXRFrame(timestamp, frame) {
if (frame.timing.droppedFrames > 0) {
scene.reduceComplexity();
}
// Render as usual
}I'm not sure how high priority I find this to be, but it's a relatively simple mechanism so if we have strong indications that the development community can make good use of it I'm not opposed to adding something along these lines to the spec.
@toji I'm glad to hear this issue hasn't been forgotten.
While it certainly doesn't hurt to have info about dropped frames, by that point it's a little too late. The original Valve technique (and my own implementation) takes a more conservative approach of checking how much time you had left over on the last frame. So if you need to get a frame out in 11ms, and it takes 9ms to render, you can reduce complexity on the next frame to hopefully avoid dropping frames at all.
There's a bigger problem here though, which is that the number of dropped frames doesn't tell you when you can bump the complexity back up again. Per Valve, if you've rendered in, let's say, 4ms for a few frames in a row without a problem, you can turn it back up a notch. This comes up all the time in practice for me, where rendering temporarily slows down - maybe because some background application used up the CPU a bunch or because you looked at a really complicated part of the scene for a moment - and then it speeds up again when that other process eases off or when you turn around and look at a simpler part of the scene.
I agree with @brianchirls, especially
the number of dropped frames doesn't tell you when you can bump the complexity back up again.
Hence the solution proposed initially:
@toji mentioned the idea of having requestAnimationFrame pass the amount of time left before the next draw
sounds more useful. It could possibly even be used to reduce latency/timewarp effect by delaying polling of the device pose (where frame.getDevicePose(...) isn't just implemented as a simple getter) in cases where you know that your scene isn't going to require all the time until the next draw is due.
This solution otoh would leave render timing to the user if I'm correct?
That's probably difficult, though, since the GPU may render stuff even after the frame callback ended as @klausw already stated (Hope I understood that correctly), except if you gl.flush() and gl.finish() correctly before the end of the frame, but that's bad since the js processing after the callback that he mentioned cannot be done in parallel anymore this way. Having the last frame duration or surplus as @brianchirls suggested would allow API implementors to do this properly.
Also: the render time budget could be a property of frame instead of passed as an extra argument maybe? 🤔 Doesn't feel entirely clean, though.
Note: I'm new here and no browser or JS expert, take everything with a grain of salt.