fluent / fluentd

Fluentd: Unified Logging Layer (project under CNCF)

Home Page:https://www.fluentd.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Match directive not working

ankit-groverDevops opened this issue · comments

Describe the bug

I have writted one fluentd configuration output as elasticsearch plugin but it's not working as it should work please check the below configurationn
scenerio is i have multiple containers running i want to index each and every containers logs index individualy

@type tail @id tail_mylogs tag xxxx.logs path "/var/log/containers/xxx-yyyy-*.log" pos_file "/var/log/containers/fluentd-xxxxx.log.pos" exclude_path ["/var/log/containers/fluentd-*.log"] refresh_interval 30 read_from_head true @type "json" time_format "%Y-%m-%dT%H:%M:%S.%NZ" time_type string unmatched_lines @type elasticsearch @log_level "debug" host "elastic-cluster-es-http.elastic-system.svc.cluster.local" port 9200 scheme https ssl_verify false user "elastic" password xxxxxx reload_connections true logstash_format true flush_thread_count 8 flush_interval 5s chunk_limit_size 2M queue_limit_length 32 retry_max_interval 30 retry_forever true log_level debug

image :-

FROM fluentd:v1.16-debian-1
USER root
RUN fluent-gem install fluent-plugin-elasticsearch

To Reproduce

Tried the latest version builted fluentd elasticsearch plugin changes configuration used file plugin but match directive is not working with tag

Expected behavior

The match pattern should work for the source tag and output the logs to elasticsearch,file or anywhere

Your Environment

- Fluentd version: 1.16.2 
- TD Agent version: 1.16.2
- Operating system: dockerimage
- Kernel version: 1.16.2  dockerversion
- fluent-plugin-elasticsearch-version '5.4.3'

Your Configuration

<source>
    @type tail
    @id tail_mylogs
    tag xxxx.logs
    path "/var/log/containers/xxx-yyyy-*.log"
    pos_file "/var/log/containers/fluentd-xxxxx.log.pos"
    exclude_path ["/var/log/containers/fluentd-*.log"]
    refresh_interval 30
    read_from_head true
    <parse>
      @type "json"
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      time_type string
      unmatched_lines 
    </parse>
  </source>
  <match xxxx.logs>
    @type elasticsearch
    @log_level "debug"
    host "elastic-cluster-es-http.elastic-system.svc.cluster.local"
    port 9200
    scheme https
    ssl_verify false
    user "elastic"
    password xxxxxx
    reload_connections true
    logstash_format true
    <buffer>
      flush_thread_count 8
      flush_interval 5s
      chunk_limit_size 2M
      queue_limit_length 32
      retry_max_interval 30
      retry_forever true
    </buffer>
  </match>
  <system>
    log_level debug
  </system>
</ROOT>

Your Error Log

Match directive not working

Additional context

No response

anyone here??