fluent-bit aws es plugin is not working with AWS_ROLE_ARN for on-prem cluster

[Dolis Sharma]

I am trying to send logs from the on-prem cluster to AWS opensearch using the fluent bit.
I see the below error in the pod log when trying to configure aws_role_arn for fluent bit

[2022/02/03 19:08:50] [ warn] [aws_credentials] No cached credentials are available and a credential refresh is already in progress. The currentco-routine will retry. [2022/02/03 19:08:50] [error] [signv4] Provider returned no credentials, service=es [2022/02/03 19:08:50] [error] [output:es:es.0] could not sign request with sigv4 [2022/02/03 19:08:50] [ warn] [engine] failed to flush chunk '1-1643909218.776499800.flb', retry in 20 seconds: task_id=0, input=tail.0 > output=es.0 [2022/02/03 19:08:50] [error] [aws_credentials] Could not read shared credentials file /root/.aws/credentials [2022/02/03 19:08:50] [error] [aws_credentials] Failed to retrieve credentials for AWS Profile default

Below are the configuration for fluentbit:

[OUTPUT]
Name es
Match *
Host search-nirmata-iglp62yproy.us-west-1.es.amazonaws.com
Port 443
Index my_index
Type my_type
AWS_Auth On
AWS_Region us-west-1
tls On
AWS_ROLE_ARN arn:aws:iam::8*536:role/role-name
AWS_STS_Endpoint https://sts.us-west-1.amazonaws.com
Logstash_Format On
Replace_Dots On
Retry_Limit False

DS env variables:

   env:
    - name: "FLUENT_ELASTICSEARCH_HOST"
      value: "elasticsearch"
    - name: "FLUENT_ELASTICSEARCH_PORT"
      value: "9200"
    - name: "AWS_ROLE_ARN"
      value: "arn:aws:iam::8**36:role/role-name"

I tried a couple of variations with my output cm and env variables but that doesn't work with aws_role_arn.
Expected behavior
should send logs to OpenSearch

[José Lecaros]

Hi @dolisss, this repo is for fluent-bit documentation. You can get help in Slack https://launchpass.com/fluent-all