bug: Configuration not being live-reloaded
bruce-y opened this issue · comments
Describe the issue
I'm not sure if this is the right repository to open this issue, but the problem seems to be in the live-reloading feature in the kubesphere/fluent-bit
image. It seems the configuration is not updated until I forcibly restart the pod.
To Reproduce
We updated the host value in the cluster output:
Before:
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
labels:
fluentbit.fluent.io/component: logging
fluentbit.fluent.io/enabled: "true"
name: my-output
spec:
alias: my_pattern
customPlugin:
config: |
Name http
Match my_pattern
Host ingress.coralogix.com
Port 443
URI /logs/v1/singles
Format json_lines
TLS On
Header Authorization Bearer <redacted>
compress gzip
Retry_Limit 10
match: my_pattern
After:
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
labels:
fluentbit.fluent.io/component: logging
fluentbit.fluent.io/enabled: "true"
name: myoutput
spec:
alias: mypattern
customPlugin:
config: |
Name http
Match mypattern
Host ingress.cx498-aws-us-west-2.coralogix.com
Port 443
URI /logs/v1/singles
Format json_lines
TLS On
Header Authorization Bearer <redacted>
compress gzip
Retry_Limit 10
match: mypattern
However, we were still observing that in the fluent-bit logs, it was trying to send the logs to:
fluent-bit-45lzx fluent-bit [2024/02/05 23:15:00] [error] [output:http:mypattern] ingress.coralogix.com:443, HTTP status=403
The config generated by the fluent-operator was correct as well:
[Output]
Match mypattern
Alias mypattern
Name http
Match mypattern
Host ingress.cx498-aws-us-west-2.coralogix.com
Port 443
URI /logs/v1/singles
Format json_lines
TLS On
Header Authorization Bearer <redacted>
compress gzip
Retry_Limit 10
This only resolved after I did a kubectl rollout restart daemonset fluent-bit
and then it began sending via the correct configuration:
fluent-bit-45lzx fluent-bit [2024/02/06 00:00:52] [ info] [output:http:mypattern] ingress.cx498-aws-us-west-2.coralogix.com:443, HTTP status=200
Expected behavior
The new configuration should be loaded by the fluent-bit pods without a forced restart.
Your Environment
- Fluent Operator version: docker.io/kubesphere/fluent-operator:v2.7.0
- Fluent Bit version: docker.io/kubesphere/fluent-bit:v2.2.2
- Container Runtime: containerd
- Operating system: Amazon Linux 2023
- Kernel version: 5.10.205-195.804.amzn2.x86_64
How did you install fluent operator?
I'm using the helm chart. This is deployed to AWS EKS.
Additional context
No response
I see something similar with changes in a Lua script / ClusterFilter
, but not even a DaemonSet restart helps there. The rendered fluentbit.conf
and other ConfigMaps / Secrets look fine. I have to re-create the whole Fluentbit
resource to get fluent-bit picking up the changes.