Kibana cannot connect to elasticsearch: Unable to revive connection

[Valentin Levitov]

Describe the bug
Clean install of the Cluster Logging Operator with default configuration of the ClusterLogging instance leads to all the services run successfully except Kibana, it shows the error on its web page:
Kibana server is not ready yet

And in the logs every several seconds repeating message:

{"type":"log","@timestamp":"2021-07-19T13:30:13Z","tags":["warning","elasticsearch","admin"],"pid":117,"message":"Unable to revive connection: https://elasticsearch.openshift-logging.svc.cluster.local:9200/"}

If I restart the kibana pod, I see the error in the logs:

{"type":"log","@timestamp":"2021-07-19T13:30:13Z","tags":["error","elasticsearch","admin"],"pid":117,"message":"Request error, retrying\nHEAD https://elasticsearch.openshift-logging.svc.cluster.local:9200/ => Hostname/IP does not match certificate's altnames: Host: elasticsearch.openshift-logging.svc.cluster.local. is not in the cert's altnames: IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1, DNS:localhost, DNS:elasticsearch, DNS:elasticsearch.openshift-logging.svc"}

So, kibana is trying to reach elasticsearch using hostname elasticsearch.openshift-logging.svc.cluster.local, but the certificate elastic works from, does not contain such AltName.

Environment

• OKD 4.7.0-0.okd-2021-07-03-190901
• ClusterLogging: Red Hat OpenShift Logging 5.0.5-11 provided by Red Hat

ClusterLogging instance spec:

spec:
  collection:
    logs:
      fluentd: {}
      type: fluentd
  curation:
    curator:
      schedule: 30 12 * * *
    type: curator
  logStore:
    elasticsearch:
      nodeCount: 3
      redundancyPolicy: SingleRedundancy
      resources:
        limits:
          memory: 8Gi
      storage:
        size: 200G
    retentionPolicy:
      application:
        maxAge: 5d
      audit:
        maxAge: 7d
      infra:
        maxAge: 7d
    type: elasticsearch
  managementState: Managed
  visualization:
    kibana:
      replicas: 1
    type: kibana

[Jeff Cantrill]

Please open an issue at https://issues.redhat.com as:

• Bugs for Official RedHat deployments are tracked there
• This error would likely be related to the Elasticsearch-operator that controls ES and kibana deployments