Clusterlogforwarders not support external elasticsearch v8.7.0

[dlaczeg]

Describe the bug
I am trying setup forwarding logs to external third-party logging systems Elasticsearch managed Azure service v8.7.0 in my openshift cluster 4.12.9 with Red Hat OpenShift Logging 5.6.5 and vector setup or fluentd

I am using procedure from https://docs.openshift.com/container-platform/4.12/logging/cluster-logging-external.html

Environment

• Versions of OpenShift, Cluster Logging and any other relevant components

Elasticsearch managed Azure service v8.7.0
openshift cluster 4.12.9
Red Hat OpenShift Logging 5.6.5 and vector or fluentd setup

• ClusterLogging instance

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  annotations:
    logging.openshift.io/preview-vector-collector: enabled
  name: instance
  namespace: openshift-logging
spec:
  collection:
    logs:
      type: vector
  logStore:
    elasticsearch:
      nodeCount: 3
      proxy:
        resources:
          limits:
            memory: 256Mi
          requests:
            memory: 256Mi
      redundancyPolicy: SingleRedundancy
      resources:
        limits:
          memory: 16Gi
        requests:
          memory: 16Gi
      storage:
        size: 200G
    retentionPolicy:
      application:
        maxAge: 1d
      audit:
        maxAge: 1d
      infra:
        maxAge: 1d
    type: elasticsearch
  managementState: Managed
  visualization:
    kibana:
      replicas: 1
    type: kibana

Logs
Capture relevant logs, post them to http://gist.github.com/ and post the links in the issue.

https://gist.github.com/dlaczeg/e3b57dc6bf2a1c8a8fd5624dfff90eae

Expected behavior
A clear and concise description of what you expected to happen.

Forwarder should recognize Elastic version 8 and skip send to api type of bulk _doc

Actual behavior
A clear and concise description of what actually happened.

To Reproduce
Steps to reproduce the behavior:

1. Try forward logs to external elasticsearch with fluentd or vector collector

Additional context
Add any other context about the problem here.

[Jeff Cantrill]

This is an example of clusterlogging CR and not a clusterlogforwarder CR. Please try utilizing that to defining log forwarding targets which ES 8 is a supported feature

[dlaczeg]

I forgot add Clusterforwader instance yaml

apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
  - name: elasticsearch
    type: "elasticsearch"
    url: https://url
    secret:
      name: openshift-test-secret
  pipelines:
  - name: application-logs
    inputRefs:
    - application
    - audit
    outputRefs:
    - elasticsearch

[dlaczeg]

Please reopen issue.

[Jeff Cantrill]

Please read ALL of my previous comment and the related links

[Andy Repton]

Hi @jcantrill , I am a Managed OpenShift Black Belt helping @dlaczeg with this issue. I suggested to him this might be a good place to open an issue but if there is a better location please do let me know.

ElasticSearch version 8 is very clearly stated as being supported with the Vector log collector, which you can see in the log file is being used. However, this issue remains.

As far as I can tell from the documentation this appears to be a bug in OpenShift Cluster Logging Operator, and it would be appreciated if this could be investigated before closing this issue. Either way, the ClusterLogForwarder CRD is identical if you use FluentD (not supported) or Vector (Supported).

Thanks,

Andy

[Andy Repton]

For anyone else who finds this moving forwards, the solution is to add version: 8 like so in the ClusterLogForwarder CR:

From this:

apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
  - name: elasticsearch
    type: "elasticsearch"
    url: https://url
    secret:
      name: openshift-test-secret
  pipelines:
  - name: application-logs
    inputRefs:
    - application
    - audit
    outputRefs:
    - elasticsearch

To:

apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
  - name: elasticsearch
    type: "elasticsearch"
    url: https://url
    secret:
      name: openshift-test-secret
    elasticsearch:
      version: 8
  pipelines:
  - name: application-logs
    inputRefs:
    - application
    - audit
    outputRefs:
    - elasticsearch