ElasticHQ connection failing after running few days

[prasadrashika]

General information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux CentOS 7
• ElasticHQ Version: 3.5.12
• Elasticsearch Version: 7.6.1
• Python version (ignore is using docker image):
• Browser Vendor and Version (if applicable):

Issue Description

I am using docker elastic HQ, could connect to elasticsearch clusters( connected with 2 clusters) , I can see indcies ,metrics etc , every thing works fine . but after few days( 2 to 3 days ) , not working, not showing any data. As per the logs elasticHQ not able to connect to elasticsearch cluster. Elastic clusters working fine but not connecting.
docker restart also not fixed this issue.

Source Code / Logs

application.log
2020-11-06 04:50:33,383 ERROR elastichq exceptions._request_wrapper:33 There is no connection with alias 'loganalytics1_es'.
Traceback (most recent call last):
File "/src/elastichq/vendor/elasticsearch/connections.py", line 88, in get_connection
return self.create_connection(alias, **self._kwargs[alias])
KeyError: 'loganalytics1_es'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/src/elastichq/service/ConnectionService.py", line 208, in get_connection
return CONNECTIONS.get_connection(cluster_name)
File "/src/elastichq/vendor/elasticsearch/connections.py", line 91, in get_connection
raise ConnectionNotFoundException('There is no connection with alias %r.' % alias)

Large logs and files should be attached. Try to provide a reproducible test case or steps that are the bare minimum necessary to generate the problem.

Be Patient

I have a day job. ;-)