sql_exporter does not recover from killed connection
0xErnie opened this issue · comments
Even if the MySQL database is available again, the exporter still indicates that the database would be offline.
Restarting the exporter solves the problem.
I0609 02:55:40.728795 25695 promhttp.go:38] Error gathering metrics: [from Gatherer #1] Error 1927: Connection was killed
[mysql] 2021/06/09 02:55:55 packets.go:36: unexpected EOF
I0609 02:55:55.728435 25695 promhttp.go:38] Error gathering metrics: [from Gatherer #1] invalid connection
[mysql] 2021/06/09 02:56:10 connection_go18.go:22: invalid connection
I0617 02:55:25.729805 25695 promhttp.go:38] Error gathering metrics: [from Gatherer #1] Error 1927: Connection was killed
I0617 02:55:40.729637 25695 promhttp.go:38] Error gathering metrics: [from Gatherer #1] bad connection
I0617 02:55:55.729257 25695 promhttp.go:38] Error gathering metrics: [from Gatherer #1] bad connection
...
I0617 09:38:10.729547 25695 promhttp.go:38] Error gathering metrics: [from Gatherer #1] bad connection
I0617 09:38:25.729122 25695 promhttp.go:38] Error gathering metrics: [from Gatherer #1] bad connection
I0617 09:38:40.730307 25695 promhttp.go:38] Error gathering metrics: [from Gatherer #1] bad connection
Our configuration:
---
# Global defaults.
global:
scrape_timeout_offset: 500ms
min_interval: 0s
max_connections: 3
max_idle_connections: 3
target:
data_source_name: {{ item.url }}
collectors: [managed_mysql]
collector_files:
- managed_mysql_collector.yml
---
collector_name: managed_mysql
metrics:
- metric_name: mysql_stats
type: gauge
help: 'Get lots of stats'
key_labels:
- Variable_name
values: [Value]
query: |
SHOW status
WHERE variable_name
IN ('threads_connected',
'Aborted_clients',
'Aborted_connects',
'Access_denied_errors',
'Acl_users',
'Bytes_received',
'Bytes_sent',
'Connections',
'Max_used_connections',
'Memory_used',
'Uptime',
'Innodb_data_read',
'Innodb_data_reads',
'Innodb_data_writes',
'Innodb_data_written'
)
I decided to switch to the fork burningalchemist/sql_exporter which has updated multiple dependencies.
I am experiencing the same problem with the sql_exporter, where the exporter needs to be restarted in case of network connection problems to the database. Anybody has a working solution for this problem?
Still suffering from the same issue. The exporter is not recovering from DB connectivity problems. Anybody has an idea on how to resolve?
I0513 23:50:20.063691 1 promhttp.go:38] Error gathering metrics: [from Gatherer #1] bad connection
I0513 23:51:20.061513 1 promhttp.go:38] Error gathering metrics: [from Gatherer #1] bad connection
I0513 23:52:20.062154 1 promhttp.go:38] Error gathering metrics: [from Gatherer #1] bad connection
I0513 23:53:20.064057 1 promhttp.go:38] Error gathering metrics: [from Gatherer #1] bad connection
I0513 23:54:20.063563 1 promhttp.go:38] Error gathering metrics: [from Gatherer #1] bad connection
@rgardelein Have you tried my solution?
Hello 0xErnie,
Thank you for your reply. I have not tried your solution by switching to the burningalchemist/sql_exporter. Is this solution fully upward compatible? What is your experience with it?
rgardelein