Giters

bosh-prometheus / shield_exporter

Shield Prometheus Exporter

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

exporter leaking file descriptors

frodenas opened this issue · comments

commented

After some time running, the shield_exporter is unable to scrape data from shield. The logs shows the following error messages:

2017/05/07 10:12:43 http: Accept error: accept tcp 0.0.0.0:9179: accept4: too many open files; retrying in 5ms
time="2017-05-07T10:12:43Z" level=error msg="Error while listing jobs: Get http://127.0.0.1:80/v1/jobs: dial tcp 127.0.0.1:80: socket: too many open files" source="jobs_collector.go:140"
time="2017-05-07T10:12:43Z" level=error msg="Error while listing schedules: Get http://127.0.0.1:80/v1/schedules: dial tcp 127.0.0.1:80: socket: too many open files" source="schedules_collector.go:136"
time="2017-05-07T10:12:43Z" level=error msg="Error while listing targets: Get http://127.0.0.1:80/v1/targets: dial tcp 127.0.0.1:80: socket: too many open files" source="targets_collector.go:139"
time="2017-05-07T10:12:43Z" level=error msg="Error while listing archives: Get http://127.0.0.1:80/v1/archives: dial tcp 127.0.0.1:80: socket: too many open files" source="archives_collector.go:139"
time="2017-05-07T10:12:43Z" level=error msg="Error while listing stores: Get http://127.0.0.1:80/v1/stores: dial tcp 127.0.0.1:80: socket: too many open files" source="stores_collector.go:139"
time="2017-05-07T10:12:43Z" level=error msg="Error while listing retention policies: Get http://127.0.0.1:80/v1/retention: dial tcp 127.0.0.1:80: socket: too many open files" source="retention_policies_collector.go:136"
time="2017-05-07T10:12:43Z" level=error msg="Error while getting internal status: Get http://127.0.0.1:80/v1/status/internal: dial tcp 127.0.0.1:80: socket: too many open files" source="status_collector.go:189"
time="2017-05-07T10:12:43Z" level=error msg="Error while listing tasks: Get http://127.0.0.1:80/v1/tasks: dial tcp 127.0.0.1:80: socket: too many open files" source="tasks_collector.go:154"

Running a netstat -anp | grep shield at the vm shows:

tcp        0      0 0.0.0.0:9179            0.0.0.0:*               LISTEN      18551/shield_export
tcp        0      0 0.0.0.0:5444            0.0.0.0:*               LISTEN      15805/shield-agent
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      15760/shieldd
tcp        0      0 127.0.0.1:33230         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33148         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33262         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33156         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33258         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33260         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33226         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33182         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33256         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33232         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33176         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33264         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 10.213.10.75:34574      10.213.10.74:5432       ESTABLISHED 15760/shieldd
tcp        0      0 127.0.0.1:33236         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33180         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33220         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33146         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33184         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33188         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33224         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33152         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33228         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33222         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 10.213.10.75:34644      10.213.10.74:5432       ESTABLISHED 15760/shieldd
tcp        0      0 127.0.0.1:33144         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33254         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33218         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33158         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33186         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33150         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33154         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        1      0 127.0.0.1:33118         127.0.0.1:80            CLOSE_WAIT  18551/shield_export
tcp        0      0 127.0.0.1:33190         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33252         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 127.0.0.1:33178         127.0.0.1:80            ESTABLISHED 18551/shield_export
tcp        0      0 10.213.10.75:42440      10.213.10.74:5432       ESTABLISHED 15760/shieldd

Seems then that the exporter is leaking file descriptors by not closing (or time out) the connection.

commented

Fixed at v0.2.0.