Support for content-encoding: deflate raw
smdjeff opened this issue · comments
How does one ask Express to process a POST with a header for 'content-encoding: deflate' where the data is raw (without the data headers and footers)?
I'm noticing there's code in express node-fetch that checks for magic bytes in the first block to decide between createInflate() and createInflateRaw() but it's not in body-parser.
body-parser read.js
switch (encoding) {
case 'deflate':
stream = zlib.createInflate()
req.pipe(stream)
break
case 'gzip':
...
node-fetch index.js
const raw = res.pipe(new PassThrough$1());
raw.once('data', function (chunk) {
// see http://stackoverflow.com/questions/37519828
if ((chunk[0] & 0x0F) === 0x08) {
body = body.pipe(zlib.createInflate());
} else {
body = body.pipe(zlib.createInflateRaw());
}
So body-parser will use the Node.js zlib.createInflate() to inflate the data from deflate. Does Node.js support inflating that particular data? If not, then it would not be aupported here, either. If you're not sure, if you can provide the raw bytes you are trying to inflate I can help investigate. Are you also getting any error and if so what is the error?
Oh, I see what you are asking. This is not supported because that is not allowed for content-encoding deflate token. The zlib header is required for that token. You can find more information in the token registry: https://www.iana.org/assignments/http-parameters/http-parameters.xhtml
I am happy to reopen if you can point to where in the specifcication / iana registry that content-encoding is allowed to be set to "deflate" without the zlib header.
How does one ask Express to process a POST with a header for 'content-encoding: deflate' where the data is raw (without the data headers and footers)?
Sorry, I missed this direct question at the top 😅. There isn't really a good way to do this with body-parser currently. You can always use the raw() parser and delete req.headers['content-encoding'] . But perhaps a better method would be someone who needs this feature can contribute an API to define custom decoders based on the content-encoding (and override existing ones with their custom impls).
I'm after cutting some fat out of a large internet of things deployment. Gzip wastes 18 bytes per transaction with redundant information like crc and data size.
deflate with zlib header wastes 2 and 4 for the footer.
deflate raw drops all 18 bytes.
I'm not sure how to peek that first byte in body-parser the way node-fetch does to switch off for both... but just changing that one line does work.
case 'deflate':
stream = zlib.createInflateRaw()
It's interesting that expressjs goes outside the standard to support deflate raw in node-fetch, but remains strict in body-parser.
It's also interesting to realize that after all this time, the standard is a bit off and could have been improved.
What it says now...
'deflate'
Using the zlib structure (defined in RFC 1950) with the deflate compression algorithm (defined in RFC 1951).
What it should say...
'deflate'
Using the deflate compression algorithm (defined in RFC 1951).
'zlib'
Using the zlib structure (defined in RFC 1950) with the deflate compression algorithm (defined in RFC 1951).
It's interesting that expressjs goes outside the standard to support deflate raw in node-fetch, but remains strict in body-parser.
I'm not sure what leads you to believe that the expressjs project manages node-fetch. We only manage the projets in our expresssjs github organization. I'm not even sure who manages node-fetch, but it is certainly unrelated to expressjs.
It's also interesting to realize that after all this time, the standard is a bit off and could have been improved.
We don't manage the standards, just implement them. If you think the standards should be changed, you should reach out to the standard organization that manages that particular one and have them update it.