suggestions for animated QOI?
psy0rz opened this issue · comments
I'm streaming low res images (75x8) to an RGB led marquee via udp @60fps. I dont need an ALPHA channel. ( The reason i need compression is that we use many of these displays at the same time to display different stuff, so bandwidth is an issue)
Things like marquees or fire animations: An old prototype demo where you can see some examples is here: https://ledder.datux.nl/#(page:#ledder-category-page)
A lot of times only a few pixels change compared to the previous frame. Many times a pixel is gradually changed compared to the previous frame.
Idea 1
I was thinking of modifiying QOI_OP_DIFF and QOI_DIFF_LUMA so that it looks at the same pixel in the previous frame.
And perhaps modify QOI_OP_RUN, and use bit 5 to indicate a RUN vs a SKIP.
A SKIP would mean: skip this many pixels, since they are unchanged since the previous frame.
Idea 2
An alternative idea: Use a special "flip" operator ( 0b11111111 for example, since i dont use alpha anyway). This switches the operators to look at the previous frame instead of the current one. Using the operator again flips it back.
This could use a slower encoder that looks ahead in chunks or 64 bytes or something to see whether it needs to flip again to get better compression for the next block.
These are just some wild ideas, anyone has any suggestions or tips?
keeping the index for the next frame would also be more efficent
Interesting use-case! Without thinking about it too much, I would probably alter the format to encode pixels in tiles (e.g. 8x8 pixels) and have a header for each tile that either:
- indicates the tile didn't change from the previous one
- the tile did change and the values are encoded as the difference from the previous tile
- the tile did change and the values are encoded in the default QOI format
If you want to get fancy, you could also implement some motion compensation for each tile and make this into a full blown video codec :)
There's been some work in this direction already: https://github.com/wide-video/qov - this doesn't use a tiled format and instead encodes differences for a whole frame.
cool ill have a look at it!
i dont want to make it too complex of a project but indeed its an interesting usecase. :)
standard qoi achieves 50%-90% reduction already, however flames are a problem. :) (actually uses more data in some cases.)
however i think that can be solved by looking at the previous frame, since flamepixels usually gradually change.
qov seems to do pretty much what i need, and its already typescript. excellent! ill try it out and/or modify it.
On a some what related note I have been poking around at created something similar to mjpeg using qoi for video steaming from embeded devices. (Or like now sound streams of live webcameras new stations use are mjpeg.)
Here's a showertought I had: Use a normal 3-channel RGB image as an i-frame, then 4 channel RGBA as p-frames. Overlay the transparent p-frames on top of the previous frames. Use the transparency in the p-frame to dictate how much to change the existing pixels.
Example:
- Send a normal image as the first frame
- Send the next frame as a diff represented as transparent image. Everywhere it's fully transparent you use the previous frame, everything opaque is the new pixels. Should compress very well still.
- You can even do effects like slow fades by using partial alpha.
Advantage: No format change required, just some higher level algorithms on top of the standard qoi encoders/decoders.
Video Decoder is simple: just remember the previous frame and composit the new probably transparent frame on top.
Video Encoder is a little trickier:
- Either send a whole frame as is, especially as the first frame
- Compute a diff frame that will result in the new frame if they're composited
- Figure out when to inject whole frames again, perhaps do both i-frames and p-frames in parallel and pick the one that's shorter (you can even do an early abort)?
Here's a showertought I had: Use a normal 3-channel RGB image as an i-frame, then 4 channel RGBA as p-frames. Overlay the transparent p-frames on top of the previous frames. Use the transparency in the p-frame to dictate how much to change the existing pixels.
Oh hey, looks like APNG uses the same strategy, here's an article about it (with pictures!): http://littlesvr.ca/apng/inter-frame.html