Content-length is wrong if GZIP is used, leading to unexpected EOF in prometheus

[Mathias Fußenegger]

I was trying to put the metrics endpoint exposing by jmx_exporter behind nginx to add basic auth, something like this:

location /metrics_cratedb {
  auth_basic           "Prometheus";
  auth_basic_user_file /etc/nginx/conf.d/wacklig.htpasswd;

  proxy_pass http://127.0.0.1:9071/;
}

But within Prometheus I got a unexpected EOF error.
I already used a similar proxy directive for another metrics endpoint and that worked, so after some debugging I concluded that the error must be related to jmx_exporter and related to the Content-Length header.

I then found this fix: prometheus/client_java@e58b9df

I think the prometheus client used here should be updated. There are quite a few changes there:

https://github.com/prometheus/client_java/releases

0.6.0 would contain the fix that I think would solve my problem.

[Mathias Fußenegger]

Got it working by setting the http version to 1.1:

  location /metrics_cratedb {
    auth_basic            "Prometheus";
    auth_basic_user_file  /etc/nginx/conf.d/wacklig.htpasswd;

    proxy_http_version    1.1;
    proxy_pass            http://127.0.0.1:9071/;
  }