BC5 / ATI2N codec missing red channel with CCodecBuffer_RG8

[carlschissler]

When trying to compress some 2-channel RG-8 images to BC5 (aka ATI2N) format, the output texture only contains valid data in the green channel when displayed in a graphics API (red channel is totally black/all zero).

This is caused by a bug in how the swizzling is handled, where the ATI2N codec reads from the B channel instead of the R channel when the m_bSwizzle member of the input buffer is true (around line 75 in codec_ati2n.cpp). The CCodecBuffer_RG8::ReadBlockB() function always sets the output data to zeros. Therefore, the output image always has a red channel that is all zero.

There are a few ways to fix this. The way I fixed it was to make the following changes:

1. In CCodec_ATI2N::Compress(), change it so that it always calls ReadBlockR(), rather than ReadBlockB(), regardless of the value of bufferIn.m_bSwizzle.
2. In codecbuffer_rg8.cpp change the RG8_CHANNEL_R define to 0 and RG8_CHANNEL_G to 1.

This produces the correct output when the texture is displayed in a graphics API (same as uncompressed texture).

Once again, I'm surprised to find such a bug in basic usage of a 15 year old library. No one ever bothered to see if BC5/ATI2N worked for a very common use case of compressing XY normal maps? I noticed this bug immediately after trying this codec, this isn't some hidden bug that only shows under special conditions. How did you even verify that the codec works? I guess if there was any testing at all it only tested the cmp_core low-level block compression functions. The codec buffer code is totally broken.