value of optimizeForLatency in emitted decoder configuration

[Bernard Aboba]

It seems odd that the value of optimizeForLatency in the decoder configuration spit out by the encoder does not reflect the value of latencyMode set in the encoder configuration.

In the encode-decode sample, I had expected (but not checked) that setting latencyMode: "realtime" in the encoder config would result in optimizeForLatency: true in the decoder config. Because the decoder config didn't automatically set optimizeForLatency: true, the measured decoder latencies for VP8 and AV1 were considerably higher (~ 100 ms) than expected (and higher than the encoder latencies).

[Dan Sanders]

This is somewhat subjective; an encoder must set necessary values and probably should set hints that it is aware of, but does realtime encoding also always imply that you want low-latency decode?

Annoyingly "realtime" is a measure of throughput rather than latency, so the mapping is not direct, but both flags can be interpreted to mean "RTC-like usage", and under that interpretation it does make sense to propagate the setting.

There are likely to be combinations of devices and streams where optimizing for latency prevents reaching realtime throughput. Since we don't have throughput knob for decoding, this isn't quite a quality-of-implementation problem. Maybe both knobs should actually be "realtime" vs "performance"?

Ergonomically it makes more sense to add new hints to an emitted config than to delete unwanted hints.