Avoid scaling the DC component too much at lower quality levels

[Jeff Muizelaar]

https://uploadcare.com/blog/fighting-jpeg-color-banding/ has some examples of the improvement that this can give.

[звездочёт]

Hi @jrmuizel .

See also Image specific quantization tables #182 comment 2.

[Jon Craton]

This seems like it could be beneficial. I saw this post as well and ran a test of this basic method against the full Kodak image set that a couple of the quantization tables have been optimized for. This does appear to generally reduce banding at lower qualities. Here's a test showing CAMBI score compared to relative file size for the images:

This change does seem to come at a cost of MSE, PSNR, and SSIM:

However, VMAF looks to slightly improve on average just below quality=50:

Netflix does point out in the linked CAMBI article that VMAF appears to not account for banding well, so it may be that this change actually is a net improvement in psychovisual quality.

Here's an example image from my testing. This is image 3 from the Kodak set at quality=14 with a size of 7.9k:

And here it is at quality 10 with the DC clamped at its q50 level:

This change certainly helps with banding, but I'm not sure whether it is generally preferable or not. In images that aren't impacted by banding, this change likely wastes bits on DC values that would be better spent elsewhere.

To be clear, the specific change I tested was to simply not decrease the DC value from the value found in the base quantization table even as quality is lowered. My thinking is that this still allows for the tuning of the DC value in the selected quantization table to have meaningful impact. The default qtable (3) has a default of 16, so this is the value that is shown in the above graphs. Using a different qtable would clamp the DC value to a different number.

I've uploaded my encoding results. This includes a quality sweep from 10 to 50 for the modified encoder and the current encoder from the main branch with quality metrics included for every image in the Kodak image set. I include separate results for qtables 2, 3, 4, and 6.

I'm not sure if this is something that we'd want to override the current default behavior, but it does seem like there would a benefit in putting a feature along these lines behind a --reduce-banding or similar flag. If there's interest in that, I can work on putting together a PR for it.

[звездочёт]

Hi @jncraton .

See Fighting JPEG color banding

louvre.q50.jpg, Q50, 23kb	louvre.q18.jpg, Q18, 11.1kb

Use jpegquant:

$ jpegquant -q 4 louvre.q50.jpg louvre.q50.jq4.jpg 
Read DCT coefficients successfully written to louvre.q50.jpg
Quant = 4.000000
QuantErr = 0.088432
Caution: quantization tables are too coarse for baseline JPEG
New DCT coefficients successfully written to louvre.q50.jq4.jpg

louvre.q50.jpg, Q50, 23kb	louvre.q50.jq4.jpg, Q50/4, 11.1kb \

$ ls -l louvre*
-rw-r--r-- 1 zvezdochiot zvezdochiot 11111 jun 28 21:58 louvre.q18.jpg
-rw-r--r-- 1 zvezdochiot zvezdochiot 23084 jun 28 21:56 louvre.q50.jpg
-rw-r--r-- 1 zvezdochiot zvezdochiot 11103 jun 28 21:57 louvre.q50.jq4.jpg

[Kornel]

Ideally this should have a couple of improvements:

1. Take into account how large is the area with flat gradients. Higher-level frequencies mask posterization/banding (not how you see banding in the sky, but on on the bulding), so DC should be coarse for noisy (or entirely flat!) images, and smooth for gradientful images.

2. When using high-quality DC, its quality should be artificially lowered in noisy areas by leveraging trellis quantization.

[Jon Craton]

@kornelski That makes sense. That turns this into a significantly more complex addition rather than a quick hack to reduce banding. If that's the best way forward, it may make sense to tackle this along with the other coefficients as part of something like RD-OPT discessed in #182.