coffeepac / chart-logging

Logging for Kubernetes Cluster

This is a light-weight logging solution for a production grade Kubernetes cluster. This system pulls logs from pods, enriches them with Kubernetes metadata, saves them in a data store and makes them available for visualizing and querying - without ever leaving the kubernetes cluster. This system also handles Kubernetes event stream logs with eventrouter, but does not yet collect etcd logs.

How to install on running Kubernetes cluster with helm

Get Helm here. Make sure your cluster has at least 6 clusterNodes and update the providerConfig.type to m4.xlarge Install Helm and the Helm registry plugin with these instructions.

helm registry install quay.io/samsung_cnct/logging

To install at cluster creation with kraken-lib , edit your configuration template helmConfigs:

helmConfigs:
  - &defaultHelm
    name: defaultHelm
    kind: helm
    repos:
      - name: stable
        url: https://kubernetes-charts.storage.googleapis.com
    charts:
      - name: logging
        registry: quay.io
        chart: samsung_cnct/logging
        channel: stable
        # or you may use version instead of channel (ex: version: 0.0.1-42)
        # visit https://quay.io/application/samsung_cnct/logging to see available versions

Once this system is set up, you can see your logs by running kubectl get svc kibana-logging -owide, then view your logs at <EXTERNAL_IP>:5601

Get kraken to help you deploy a Kubernetes cluster.

Assets

Assets for each component in the centralized logging system including github repo, quay repo for the container and quay app registry for the chart.

On Node Collector: A Fluent-bit daemonset is responsible for collecting logs from applications on pods.

• Github Container Repo: https://github.com/samsung-cnct/container-fluent-bit
• Github Chart Repo: https://github.com/samsung-cnct/chart-fluent-bit
• Image: https://quay.io/repository/samsung_cnct/fluent-bit-container
• Chart: https://quay.io/application/samsung_cnct/fluent-bit

Eventstream Log Collector: The eventrouter is used to watch for kubernetes apiserver events. Eventrouter is configured to write the events to stdout. Fluentbit forwards the stdout events to Elasticsearch.

• Github Container Repo: https://github.com/samsung-cnct/container-eventrouter
• Github Chart Repo: https://github.com/samsung-cnct/chart-eventrouter
• Image: https://quay.io/repository/samsung_cnct/eventrouter-container
• Chart: https://quay.io/application/samsung_cnct/eventrouter

Queryable Datastore: ElasticSearch

• Github Container Repo: https://github.com/samsung-cnct/container-elasticsearch
• Github Chart Repo: https://github.com/samsung-cnct/chart-elasticsearch
• Image: https://quay.io/repository/samsung_cnct/elasticsearch-container
• Chart: https://quay.io/application/samsung_cnct/elasticsearch-chart

Index Manager for Elasticsearch Curator

• Github Container Repo: https://github.com/samsung-cnct/container-curator
• Github Chart Repo: https://github.com/samsung-cnct/chart-curator
• Image: https://quay.io/repository/samsung_cnct/curator-container
• Chart: https://quay.io/application/samsung_cnct/curator

Data Visualization: Kibana

• Github Container Repo: https://github.com/samsung-cnct/container-kibana
• Github Chart Repo: https://github.com/samsung-cnct/chart-kibana
• Image: https://quay.io/repository/samsung_cnct/kibana-container
• Chart: https://quay.io/application/samsung_cnct/kibana-chart

Vendoring strategy

This project is built with the helm application registry and pushed to quay. Unfortunately the helm registry plugin doesn't play nicely with standard helm dependencies. The vendored files for a given release (stable or alpha) are built in the CI process and saved off to quay as a built package.

Please see the CI definition and the dependency management script for details.

Contributing

1. Fork it!
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -am 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Submit a pull request :D

Credits

Created and maintained by the Samsung Cloud Native Computing Team.