[Loki] Issues with Implementing Retention Policy and Excluding Specific Namespaces/Pods from Logging

[Shadab Akram]

Kindly help! I am encountering issues with configuring the retention policy in Loki, resulting in pods entering a CrashLoopBackOff state after the implementation. Additionally, I am seeking guidance on how to exclude certain namespaces and pods from having their logs collected and stored in an S3 bucket.
Loki Version: 2.9.4

failed parsing config: /etc/loki/config/config.yaml: yaml: unmarshal errors:
line 17: field retention_deletes_enabled not found in type compactor.Config
line 19: field retention_period not found in type compactor.Config. Use -config.expand-env=true flag if you want to expand environment variables in your config file.

Please find my configuration.

storage:
bucketNames:
chunks: xxxxx-loki-chunks
ruler: xxxxx-loki-ruler
admin: xxxxx-admin
type: s3
s3:
region: af-south-1
s3ForcePathStyle: false
insecure: false

compactor:
apply_retention_interval: 1h
working_directory: /data/loki/compactor <-- after commenting retention_delete_enabled=true I face issues creating this dir structure and I'm not sure where exactly to create this dir.
shared_store: s3
compaction_interval: 1h
retention_enabled: true
retention_delete_delay: 2h
retention_delete_worker_count: 150
#retention_deletes_enabled: true
#retention_period: 72h # 3 days

schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: s3
schema: v11
index:
prefix: index_
period: 24h

Other configurations...

I am looking to optimize our logging by excluding certain namespaces and pods from being logged to reduce storage and processing overhead. I am unsure whether this configuration should be applied in Loki or Promtail, and how to properly configure it.

Any guidance or examples on resolving these issues would be highly appreciated. Thank you in advance for your help!

[Leandro Saldivar]

Hey! Have you checked out this page? https://grafana.com/docs/loki/latest/operations/storage/retention/

For the retention period, make sure to configure it within limits_config:

limits_config:
  retention_period: 744h
  retention_stream:
  - selector: '{namespace="dev"}'
    priority: 1
    period: 24h
per_tenant_override_config: /etc/overrides.yaml

Regarding the namespace scrape config, it seems to be a Promtail issue. I've experimented with various configs, but none seem to work. No matter what, all the namespaces' logs are still being scraped. If you come across a solution, I would appreciate it if you could share it.

This is the last config i tried:

  scrape_configs: | 
    - job_name: kubernetes-pods
      kubernetes_sd_configs:
        - role: pod
      pipeline_stages:
        - cri: {}
      relabel_configs:
        - source_labels:
            - __meta_kubernetes_namespace
          regex: ^(bps|redis)$
          action: keep
        - source_labels:
            - __meta_kubernetes_namespace
          regex: .+
          action: drop

[Leandro Saldivar]

Finally got the solution to the namespace filtering problem. You need to look for the extraRelabelConfigs inside config.snippets in the promtail yaml

extraRelabelConfigs: 
      - action: keep
        source_labels: 
          - namespace 
        regex: ^(namespace1|namespace2)$

And you should be good with that