Request User-Agent seems to change the compression behavior

[Guillaume Acard]

Hello,

I have noticed a strange behavior when activating the compression in express. Depending on the request value for User-Agent, gzip compression will not activate.

Here is the code I use to reproduce:

const express = require("express");
const compression = require("compression");

let val = ""; for(let i=0; i<10000;i++) val += "abcd123";

const app = express();
app.use(compression());

app.get("/test", (req, res) => res.send(val));
app.listen(3030);

When making the call with a random agent, it works:
curl -i -H "User-Agent: random" -H "Accept-Encoding: gzip, deflate" http://localhost:3030/test
Response:

HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
ETag: W/"11170-VMZQjzsMfaCheitlrBuLoAvjE4k"
Vary: Accept-Encoding
Content-Encoding: gzip    <=== As expected
Date: Fri, 29 Dec 2017 14:48:25 GMT
Connection: keep-alive
Transfer-Encoding: chunked

Warning: Binary output can mess up your terminal. Use "--output -" to tell

But when using the user-agent AppleWebKit or Geeko/, it does not:
curl -i -H "User-Agent: AppleWebKit" -H "Accept-Encoding: gzip, deflate" http://localhost:3030/test
Response:

HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
ETag: W/"11170-VMZQjzsMfaCheitlrBuLoAvjE4k"
Vary: Accept-Encoding
Date: Fri, 29 Dec 2017 14:49:07 GMT
Connection: keep-alive
Transfer-Encoding: chunked

abcd123abcd123abcd123abcd123abcd123abcd123abcd123....(removed for simplicity)

I can reproduce this also in Chrome when using a custom device (and overriding the User-Agent value).

Is it normal that the User-Agent request header impacts how compression works? Any idea how to prevent this?

Thanks!

[Douglas Wilson]

Hi @guillaume-acard Im sorry you're having trouble using this module. There is no code in this module that even looks at the User-Agent, so I really have no idea who it could change behavior based on that (you're welcome to take a look -- it's source is open for viewing). I tried to reproduce it as you described above so I could use a debugger to see what is happening, but it seems both curls you provided above result in gzip when I run the code you provided. Here is a copy of my terminal session:

$ npm i compression express
+ compression@1.7.1
+ express@4.16.2
added 51 packages in 2.645s

$ cat << EOF > app.js
> const express = require("express");
> const compression = require("compression");
> 
> let val = ""; for(let i=0; i<10000;i++) val += "abcd123";
> 
> const app = express();
> app.use(compression());
> 
> app.get("/test", (req, res) => res.send(val));
> app.listen(3030);
> EOF

$ node app &
[1] 55439

$ curl -i -H "User-Agent: random" -H "Accept-Encoding: gzip, deflate" http://localhost:3030/test
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
ETag: W/"11170-VMZQjzsMfaCheitlrBuLoAvjE4k"
Vary: Accept-Encoding
Content-Encoding: gzip
Date: Fri, 29 Dec 2017 15:45:49 GMT
Connection: keep-alive
Transfer-Encoding: chunked

??70M???5`?3y????$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?k??+p

$ curl -i -H "User-Agent: AppleWebKit" -H "Accept-Encoding: gzip, deflate" http://localhost:3030/test
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
ETag: W/"11170-VMZQjzsMfaCheitlrBuLoAvjE4k"
Vary: Accept-Encoding
Content-Encoding: gzip
Date: Fri, 29 Dec 2017 15:46:14 GMT
Connection: keep-alive
Transfer-Encoding: chunked

??70M???5`?3y????$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?$I?k??+p

I would love to take a look, can you provide steps to reproduce the issue?

[Guillaume Acard]

Hi @dougwilson, Thanks for the prompt response.

I tried on a different machine and could not reproduce either but can every time on my main computer...
There must be some specific settings that somehow influences the generated response. (I already checked the os, curl, node version, npm configs... nope all same).

My feeling is something is messing around with the headers depending on the User-Agent, then impacting the way the compression lib reacts.

I will continue to investigate and will share any findings of interest here but in the mean time will close the issue as it does not seems related to a code issue from the compression lib in the end.