Plugin broken with rabbitmq >= 3.6.7
jperville opened this issue · comments
I packaged this plugin as a part of my custom RabbitMQ docker image, which is based on the official rabbitmq:3.6 image.
Until today, the plugin worked (RabbitMQ version was 3.6.6). I rebuilt the image today, RabbitMQ was upgraded to version 3.6.8, and this plugin does not work anymore. When I curl the /api/metrics route on the management port I get 404 error. As a workaround, I rebuilt my image using rabbitmq:3.6.6 as the base, but I still want to report the issue to get it fixed.
Here are the list of the plugins installed in my image:
=INFO REPORT==== 24-Mar-2017::17:15:37 ===
Server startup complete; 11 plugins started.
* prometheus_rabbitmq_exporter
* rabbitmq_management
* rabbitmq_web_dispatch
* autocluster
* rabbitmq_delayed_message_exchange
* rabbitmq_management_agent
* prometheus_process_collector
* amqp_client
* cowboy
* cowlib
* prometheus
=INFO REPORT==== 24-Mar-2017::17:17:18 ===
Yes, looks like they finally moved to cowboy, will release update soon. Thanks for the report.
Btw there is prometheus_httpd - zero deps /metric endpoint. Maybe you can consider it in the meantime.
Hello @deadtrickster . Thank you very much for the new release, it seems to mostly work.
Just a small detail to report, when I curl the /api/metrics route, I get a nasty stack trace in the RabbitMQ log and the content is served to stdout gzipped (but curl does not uncompress it; after all, I did not request Accept-Encoding: gzip).
If I query with curl --compressed or with wget (without specifying any Accept-Encoding) the page is decompressed.
wget --no-proxy -qO- http://rabbitmq-myproject.192.168.0.175.xip.io/api/metrics # works
curl --noproxy \* http://rabbitmq-myproject.192.168.0.175.xip.io/api/metrics # gzipped output sent, not decompressed by curl, stacktrace in the log
curl --compressed --noproxy \* http://rabbitmq-myproject.192.168.0.175.xip.io/api/metrics # worksI suspect that the metric contents is sent gzipped to the http client, regardless of the presence of the Accept-Encoding header in the request.
Hey, could you please give me your stack trace?
Hello, actually, there is stack trace in both wget and curl case. In both cases a gzipped output is produced, but RabbitMQ outputs ugly stack traces.
I packaged RabbitMQ using Docker.
Assuming a directory with the following content in Dockerfile:
FROM rabbitmq:3.6.8
# rabbitmq-management
RUN rabbitmq-plugins enable --offline rabbitmq_management
EXPOSE 15671 15672
# prometheus exporter plugin
ADD ["https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/rabbitmq-3.6.8.0/accept-0.3.0.ez", \
"https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/rabbitmq-3.6.8.0/prometheus-3.2.2.ez", \
"https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/rabbitmq-3.6.8.0/prometheus_httpd-1.1.0.ez", \
"https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/rabbitmq-3.6.8.0/prometheus_process_collector-1.0.2.ez", \
"https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/rabbitmq-3.6.8.0/prometheus_rabbitmq_exporter-v3.6.8.0.ez", \
"/usr/lib/rabbitmq/lib/rabbitmq_server-${RABBITMQ_VERSION}/plugins/"]
RUN chmod a+r /usr/lib/rabbitmq/lib/rabbitmq_server-${RABBITMQ_VERSION}/plugins/prometheus*.ez /usr/lib/rabbitmq/lib/rabbitmq_server-${RABBITMQ_VERSION}/plugins/accept*.ez \
&& rabbitmq-plugins enable --offline prometheus_rabbitmq_exporter prometheus_process_collector prometheus_httpd \
&& chmod -R 777 /etc/rabbitmqI build the docker image and then ran it like this:
docker build -t dev-rabbitmq .
docker run --rm -ti -p 15672:15672 dev-rabbitmqStacktrace for wget (produced by running wget -qO- http://localhost:15672/api/metrics):
=ERROR REPORT==== 27-Mar-2017::11:41:35 ===
Error in process <0.589.0> on node rabbit@a696e7b01682 with exit value:
{[{reason,undef},
{mfa,{prometheus_rabbitmq_exporter_handler,terminate,3}},
{stacktrace,
[{prometheus_rabbitmq_exporter_handler,terminate,
[{normal,shutdown},
{http_req,#Port<0.26381>,ranch_tcp,keepalive,<0.589.0>,<<"GET">>,
'HTTP/1.1',
{{172,17,42,1},55076},
<<"localhost">>,undefined,15672,<<"/api/metrics">>,undefined,
<<>>,[],[],
[{<<"user-agent">>,<<"Wget/1.17.1 (linux-gnu)">>},
{<<"accept">>,<<"*/*">>},
{<<"accept-encoding">>,<<"identity">>},
{<<"host">>,<<"localhost:15672">>},
{<<"connection">>,<<"Keep-Alive">>}],
[{<<"connection">>,[<<"keep-alive">>]}],
undefined,[],waiting,<<>>,undefined,true,locked,[],<<>>,
already_called},
undefined],
[]},
{cowboy_handler,handler_terminate,4,
[{file,"src/cowboy_handler.erl"},{line,285}]},
{cowboy_handler,terminate_request,5,
[{file,"src/cowboy_handler.erl"},{line,273}]},
{cowboy_protocol,execute,4,
[{file,"src/cowboy_protocol.erl"},{line,442}]}]},
{req,
[{socket,#Port<0.26381>},
{transport,ranch_tcp},
{connection,keepalive},
{pid,<0.589.0>},
{method,<<"GET">>},
{version,'HTTP/1.1'},
{peer,{{172,17,42,1},55076}},
{host,<<"localhost">>},
{host_info,undefined},
{port,15672},
{path,<<"/api/metrics">>},
{path_info,undefined},
{qs,<<>>},
{qs_vals,[]},
{bindings,[]},
{headers,
[{<<"user-agent">>,<<"Wget/1.17.1 (linux-gnu)">>},
{<<"accept">>,<<"*/*">>},
{<<"accept-encoding">>,<<"identity">>},
{<<"host">>,<<"localhost:15672">>},
{<<"connection">>,<<"Keep-Alive">>}]},
{p_headers,[{<<"connection">>,[<<"keep-alive">>]}]},
{cookies,undefined},
{meta,[]},
{body_state,waiting},
{buffer,<<>>},
{multipart,undefined},
{resp_compress,true},
{resp_state,done},
{resp_headers,[]},
{resp_body,<<>>},
{onresponse,already_called}]},
{state,undefined},
{terminate_reason,{normal,shutdown}}],
[{cowboy_handler,handler_terminate,4,
[{file,"src/cowboy_handler.erl"},{line,287}]},
{cowboy_handler,terminate_request,5,
[{file,"src/cowboy_handler.erl"},{line,273}]},
{cowboy_protocol,execute,4,[{file,"src/cowboy_protocol.erl"},{line,442}]}]}
=ERROR REPORT==== 27-Mar-2017::11:41:35 ===
Ranch listener rabbit_web_dispatch_sup_15672 had connection process started with cowboy_protocol:start_link/4 at <0.589.0> exit with reason: {[{reason,undef},{mfa,{prometheus_rabbitmq_exporter_handler,terminate,3}},{stacktrace,[{prometheus_rabbitmq_exporter_handler,terminate,[{normal,shutdown},{http_req,#Port<0.26381>,ranch_tcp,keepalive,<0.589.0>,<<"GET">>,'HTTP/1.1',{{172,17,42,1},55076},<<"localhost">>,undefined,15672,<<"/api/metrics">>,undefined,<<>>,[],[],[{<<"user-agent">>,<<"Wget/1.17.1 (linux-gnu)">>},{<<"accept">>,<<"*/*">>},{<<"accept-encoding">>,<<"identity">>},{<<"host">>,<<"localhost:15672">>},{<<"connection">>,<<"Keep-Alive">>}],[{<<"connection">>,[<<"keep-alive">>]}],undefined,[],waiting,<<>>,undefined,true,locked,[],<<>>,already_called},undefined],[]},{cowboy_handler,handler_terminate,4,[{file,"src/cowboy_handler.erl"},{line,285}]},{cowboy_handler,terminate_request,5,[{file,"src/cowboy_handler.erl"},{line,273}]},{cowboy_protocol,execute,4,[{file,"src/cowboy_protocol.erl"},{line,442}]}]},{req,[{socket,#Port<0.26381>},{transport,ranch_tcp},{connection,keepalive},{pid,<0.589.0>},{method,<<"GET">>},{version,'HTTP/1.1'},{peer,{{172,17,42,1},55076}},{host,<<"localhost">>},{host_info,undefined},{port,15672},{path,<<"/api/metrics">>},{path_info,undefined},{qs,<<>>},{qs_vals,[]},{bindings,[]},{headers,[{<<"user-agent">>,<<"Wget/1.17.1 (linux-gnu)">>},{<<"accept">>,<<"*/*">>},{<<"accept-encoding">>,<<"identity">>},{<<"host">>,<<"localhost:15672">>},{<<"connection">>,<<"Keep-Alive">>}]},{p_headers,[{<<"connection">>,[<<"keep-alive">>]}]},{cookies,undefined},{meta,[]},{body_state,waiting},{buffer,<<>>},{multipart,undefined},{resp_compress,true},{resp_state,done},{resp_headers,[]},{resp_body,<<>>},{onresponse,already_called}]},{state,undefined},{terminate_reason,{normal,shutdown}}],[{cowboy_handler,handler_terminate,4,[{file,"src/cowboy_handler.erl"},{line,287}]},{cowboy_handler,terminate_request,5,[{file,"src/cowboy_handler.erl"},{line,273}]},{cowboy_protocol,execute,4,[{file,"src/cowboy_protocol.erl"},{line,442}]}]}
Stacktrace for curl (produced by running curl http://localhost:15672/api/metrics; no --compress option so output on stdout is gzipped despite of no "Accept-Encoding: gzip" being specified):
=ERROR REPORT==== 27-Mar-2017::11:41:54 ===
Error in process <0.600.0> on node rabbit@a696e7b01682 with exit value:
{[{reason,undef},
{mfa,{prometheus_rabbitmq_exporter_handler,terminate,3}},
{stacktrace,
[{prometheus_rabbitmq_exporter_handler,terminate,
[{normal,shutdown},
{http_req,#Port<0.26396>,ranch_tcp,keepalive,<0.600.0>,<<"GET">>,
'HTTP/1.1',
{{172,17,42,1},55128},
<<"localhost">>,undefined,15672,<<"/api/metrics">>,undefined,
<<>>,[],[],
[{<<"host">>,<<"localhost:15672">>},
{<<"user-agent">>,<<"curl/7.47.0">>},
{<<"accept">>,<<"*/*">>}],
[],undefined,[],waiting,<<>>,undefined,true,locked,[],<<>>,
already_called},
undefined],
[]},
{cowboy_handler,handler_terminate,4,
[{file,"src/cowboy_handler.erl"},{line,285}]},
{cowboy_handler,terminate_request,5,
[{file,"src/cowboy_handler.erl"},{line,273}]},
{cowboy_protocol,execute,4,
[{file,"src/cowboy_protocol.erl"},{line,442}]}]},
{req,
[{socket,#Port<0.26396>},
{transport,ranch_tcp},
{connection,keepalive},
{pid,<0.600.0>},
{method,<<"GET">>},
{version,'HTTP/1.1'},
{peer,{{172,17,42,1},55128}},
{host,<<"localhost">>},
{host_info,undefined},
{port,15672},
{path,<<"/api/metrics">>},
{path_info,undefined},
{qs,<<>>},
{qs_vals,[]},
{bindings,[]},
{headers,
[{<<"host">>,<<"localhost:15672">>},
{<<"user-agent">>,<<"curl/7.47.0">>},
{<<"accept">>,<<"*/*">>}]},
{p_headers,[]},
{cookies,undefined},
{meta,[]},
{body_state,waiting},
{buffer,<<>>},
{multipart,undefined},
{resp_compress,true},
{resp_state,done},
{resp_headers,[]},
{resp_body,<<>>},
{onresponse,already_called}]},
{state,undefined},
{terminate_reason,{normal,shutdown}}],
[{cowboy_handler,handler_terminate,4,
[{file,"src/cowboy_handler.erl"},{line,287}]},
{cowboy_handler,terminate_request,5,
[{file,"src/cowboy_handler.erl"},{line,273}]},
{cowboy_protocol,execute,4,[{file,"src/cowboy_protocol.erl"},{line,442}]}]}
=ERROR REPORT==== 27-Mar-2017::11:41:54 ===
Ranch listener rabbit_web_dispatch_sup_15672 had connection process started with cowboy_protocol:start_link/4 at <0.600.0> exit with reason: {[{reason,undef},{mfa,{prometheus_rabbitmq_exporter_handler,terminate,3}},{stacktrace,[{prometheus_rabbitmq_exporter_handler,terminate,[{normal,shutdown},{http_req,#Port<0.26396>,ranch_tcp,keepalive,<0.600.0>,<<"GET">>,'HTTP/1.1',{{172,17,42,1},55128},<<"localhost">>,undefined,15672,<<"/api/metrics">>,undefined,<<>>,[],[],[{<<"host">>,<<"localhost:15672">>},{<<"user-agent">>,<<"curl/7.47.0">>},{<<"accept">>,<<"*/*">>}],[],undefined,[],waiting,<<>>,undefined,true,locked,[],<<>>,already_called},undefined],[]},{cowboy_handler,handler_terminate,4,[{file,"src/cowboy_handler.erl"},{line,285}]},{cowboy_handler,terminate_request,5,[{file,"src/cowboy_handler.erl"},{line,273}]},{cowboy_protocol,execute,4,[{file,"src/cowboy_protocol.erl"},{line,442}]}]},{req,[{socket,#Port<0.26396>},{transport,ranch_tcp},{connection,keepalive},{pid,<0.600.0>},{method,<<"GET">>},{version,'HTTP/1.1'},{peer,{{172,17,42,1},55128}},{host,<<"localhost">>},{host_info,undefined},{port,15672},{path,<<"/api/metrics">>},{path_info,undefined},{qs,<<>>},{qs_vals,[]},{bindings,[]},{headers,[{<<"host">>,<<"localhost:15672">>},{<<"user-agent">>,<<"curl/7.47.0">>},{<<"accept">>,<<"*/*">>}]},{p_headers,[]},{cookies,undefined},{meta,[]},{body_state,waiting},{buffer,<<>>},{multipart,undefined},{resp_compress,true},{resp_state,done},{resp_headers,[]},{resp_body,<<>>},{onresponse,already_called}]},{state,undefined},{terminate_reason,{normal,shutdown}}],[{cowboy_handler,handler_terminate,4,[{file,"src/cowboy_handler.erl"},{line,287}]},{cowboy_handler,terminate_request,5,[{file,"src/cowboy_handler.erl"},{line,273}]},{cowboy_protocol,execute,4,[{file,"src/cowboy_protocol.erl"},{line,442}]}]}
Hi, thank you for a very detailed reply.
This issue uncovered two things:
Default content encoding
Here is excerpt from RFC:
A request without an Accept-Encoding header field implies that the
user agent has no preferences regarding content-codings. Although
this allows the server to use any content-coding in a response, it
does not imply that the user agent will be able to correctly process
all encodings.
Curl, etc doesn't send Accept-Encoding at all!
So although default gzip is conforming, I changed to identity.
Missing Cowboy handler callback
This is a bug that caused lengthy stack-traces. Fixed.
Release: https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/tag/rabbitmq-3.6.8.1
Also, regarding docker, do you mind if I'll use your docker file to publish "official" rabbitmq+prometheus containers?
You are welcome to use my Docker file as example for this project. Thank you for the fixes, I will try them tomorrow and will report here.
Thank you very much, I tested the new release and there is no more ugly stacktraces in the rabbitmq log.