Giters

stefanprodan / swarmprom

Docker Swarm instrumentation with Prometheus, Grafana, cAdvisor, Node Exporter and Alert Manager

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

store prometheus metrics in postgresql

shambakey1 opened this issue · comments

commented

Hi

I'm trying to store prometheus metrics in postgresql based on prometheus-postgresql-adapter. I modified the docker-compose.yml to the docker-compose-pg-old.yml.pdf (which includes 2 additional services corresponding to the first 2 containers in prometheus-postgresql-adapter, and comments out the local storage for prometheus). The prometheus.yml is modified as shown in the prometheus.yml.pdf to direct "read" and "write" to postgresql. I had to build the prometheus docker image to include the modified prometheus.yml.

The stack is deployed under the name "mon". The mon_prometheus should connect to the "mon_prometheus_postgresql_adapter", which in turn connects to mon_pg_prometheus (the postgresql database). The problem is that "mon_prometheus" service is unable to connect the "mon_prometheus_postgresql_adapter". The logs from "mon_prometheus" says:

level=error ts=2018-02-20T04:13:33.284782524Z caller=engine.go:544 component="query engine" msg="error selecting series set" err="error sending request: Post http://mon_prometheus_postgresql_adapter:9201/read: dial tcp: lookup mon_prometheus_postgresql_adapter on 127.0.0.11:53: no such host"

Regards

commented

mon is the stack name, Swarm DNS does not contain the stack name, so you should use prometheus_postgresql_adapter instead of mon_prometheus_postgresql_adapter.

commented

Thanks Stefan. The previous error is gone, but the postgresql metrics tables are still empty.

I removed all configurations related to "--storage.tsdb" from prometheus Dockerfile and the modified docker-compose.yml file. "remote_write" and "remote_read" point to "prometheus_postgresql_adapter" instead of "mon_prometheus_postgresql_adapter", but I don't know why metrics are not inserted into postgresql.

The logs of the "mon_prometheus" say:
level=warn ts=2018-02-20T16:21:33.776289124Z caller=queue_manager.go:225 component=remote msg="Remote storage queue full, discarding sample. Multiple subsequent messages of this kind may be suppressed.". If postgresql can't take more data, I expected, at least, to record some initial data, then stops receiving any more data.

Regards

commented

You should ask the Prometheus team, I've never used PostgreSQL as a backend.