Deflate/Inflate does not work with zlib/gzip c++

[Josh Menzel]

Zlib and Gzip deflate results do not match zlib/gzip c/c++ implementation thus giving Z_DATA_ERROR when trying to inflate in c/c++ OR on pako's own module.

[101arrowz]

Could you send some code I can use to reproduce this? As far as I know the streams work as expected.

[Josh Menzel]

Sure - I'll also include the c++ code for reference:

I'm also not using streams but rather strings.

C++ Zlib Compress/Decompress (Deflate/Inflate)

//Z_COMPRESSION_BUFFER_BLOCK_SIZE is 16384
void Zlib::Compress(Compression::Binary_Data_T& data)
{
    z_stream zs;                        // z_stream is zlib's control structure
    memset(&zs, 0, sizeof(zs));

    if (deflateInit(&zs, Z_BEST_COMPRESSION) != Z_OK)
        throw(std::runtime_error("deflateInit failed while compressing."));

    zs.next_in = (Bytef*)data.Data();
    zs.avail_in = data.Size();           // set the z_stream's input

    int ret;
    char outbuffer[Z_COMPRESSION_BUFFER_BLOCK_SIZE];
    Compression::Binary_Data_T Output = Compression::Binary_Data_T();

    // retrieve the compressed bytes blockwise
    do {
        zs.next_out = reinterpret_cast<Bytef*>(outbuffer);
        zs.avail_out = sizeof(outbuffer);

        ret = deflate(&zs, Z_FINISH);

        if (Output.Size() < zs.total_out) {
            // append the block to the output string
            Output.Append(outbuffer,
                zs.total_out - Output.Size());
        }
    } while (ret == Z_OK);

    deflateEnd(&zs);

    if (ret != Z_STREAM_END) {          // an error occurred that was not EOF
        std::ostringstream oss;
        oss << "Exception during zlib compression: (" << ret << ") " << zs.msg;
        throw(std::runtime_error(oss.str()));
    }
    data.Clear();
    data.Append(Output.Data(), Output.Size());
}


void Zlib::Decompress(Compression::Binary_Data_T& data)
{
    z_stream zs;                        // z_stream is zlib's control structure
    memset(&zs, 0, sizeof(zs));

    if (inflateInit(&zs) != Z_OK)
        throw(std::runtime_error("inflateInit failed while decompressing."));

    zs.next_in = (Bytef*)data.Data();
    zs.avail_in = data.Size();

    int ret;
    char outbuffer[Z_COMPRESSION_BUFFER_BLOCK_SIZE];
    Compression::Binary_Data_T Output = Compression::Binary_Data_T();

    // get the decompressed bytes blockwise using repeated calls to inflate
    do {
        zs.next_out = reinterpret_cast<Bytef*>(outbuffer);
        zs.avail_out = sizeof(outbuffer);

        ret = inflate(&zs, 0);

        if (Output.Size() < zs.total_out) {
            Output.Append(outbuffer,
                zs.total_out - Output.Size());
        }

    } while (ret == Z_OK);

    inflateEnd(&zs);

    if (ret != Z_STREAM_END) {          // an error occurred that was not EOF
        std::ostringstream oss;
        oss << "Exception during zlib decompression: (" << ret << ") "
            << zs.msg;
        throw(std::runtime_error(oss.str()));
    }

    data.Clear();
    data.Append(Output.Data(), Output.Size());
}

Javascript (ReactJS/npm/runs in browser/client side) - this is in an OnMessage websocket event:

        //event.data is base64 encoded string containing the compressed bytes
        const buff = fflate.strToU8(atob(event.data));
        //ERROR - invalid data - but pako deflates it just fine
        const decompressed = fflate.inflateSync(buff);
        const rawJson = fflate.strFromU8(decompressed);

Javascript (ReactJS/npm/runs in browser/client side) - this is in the SendMessage function

const buff = fflate.strToU8(message);
//when inflated in the c++ side - code -3 is returned (Z_DATA_ERROR)
const compressed = fflate.deflateSync(buff);
const b64 = btoa(compressed);

[101arrowz]

fflate's deflateSync and inflateSync actually correspond to Pako's deflateRaw and inflateRaw. I probably should document that better. (To be clear, Pako and Zlib are using the wrong naming here, not fflate - for some reason the Zlib C library calls the Zlib format "deflate".)

Can you try replacing your inflateSync/deflateSync calls with unzlibSync and zlibSync?

[Josh Menzel]

Yeah I tried that too and I got "invalid zlib data" on the javascript side when trying to inflate, and Z_DATA_ERROR on the c++ side

[Josh Menzel]

Nevermind - it seems that did the trick! Was missing a couple of steps to get the data from the javascript into the right format.

[101arrowz]

Glad you got it working! For anyone else who finds this issue, I think I saw the encoding issue you were talking about - your Base64 strings weren't being encoded/decoded properly. atob returns a binary string, so you need to run fflate.strToU8(atob(data), true) to decode Base64 to a Uint8Array. Similarly, to encode the fflate output to Base64 you should do btoa(fflate.strFromU8(compressed, true)).

[Josh Menzel]

To be clear - yeah that's exactly what the issue was.

What I ended up with on the javascript side:

To Send A Compressed Message:

            const buff = fflate.strToU8(message);
            const compressed = fflate.zlibSync(buff);
            const data = btoa(fflate.strFromU8(compressed,true));
            this.#ws.send(data);

To Receive A Compressed Message:

        const buff = fflate.strToU8(atob(event.data),true);
        const decompressed = fflate.unzlibSync(buff);
        const data = fflate.strFromU8(decompressed);

On the C++ Side:
To Send A Compressed Message:

                Compression::Zlib::Compress(*message);
		message->ToBase64();

To Receive A Compressed Message:

	Compression::Binary_Data_T msg = Compression::Binary_Data_T();
	msg.Append(data);
	msg.FromBase64();
	Compression::Zlib::Decompress(msg);
	data.clear();
	data = msg.to_string();