This is a light-weight logging solution for a production grade Kubernetes cluster. This system pulls logs from pods, enriches them with Kubernetes metadata, saves them in a data store and makes them available for visualizing and querying - without ever leaving the kubernetes cluster. This system also handles Kubernetes event stream logs with eventrouter, but does not yet collect etcd logs.
Get Helm here. Make sure your cluster has at least 6 clusterNodes and update the providerConfig.type to m4.xlarge Install Helm and the Helm registry plugin with these instructions.
helm registry install quay.io/samsung_cnct/logging
To install at cluster creation with kraken-lib , edit your configuration template helmConfigs
:
helmConfigs:
- &defaultHelm
name: defaultHelm
kind: helm
repos:
- name: stable
url: https://kubernetes-charts.storage.googleapis.com
charts:
- name: logging
registry: quay.io
chart: samsung_cnct/logging
channel: stable
# or you may use version instead of channel (ex: version: 0.0.1-42)
# visit https://quay.io/application/samsung_cnct/logging to see available versions
Once this system is set up, you can see your logs by running kubectl get svc kibana-logging -owide
, then view your logs at <EXTERNAL_IP>:5601
Get kraken to help you deploy a Kubernetes cluster.
Assets for each component in the centralized logging system including github repo, quay repo for the container and quay app registry for the chart.
On Node Collector: A Fluent-bit daemonset is responsible for collecting logs from applications on pods.
- Github Container Repo: https://github.com/samsung-cnct/container-fluent-bit
- Github Chart Repo: https://github.com/samsung-cnct/chart-fluent-bit
- Image: https://quay.io/repository/samsung_cnct/fluent-bit-container
- Chart: https://quay.io/application/samsung_cnct/fluent-bit
Eventstream Log Collector: The eventrouter is used to watch for kubernetes apiserver events. Eventrouter is configured to write the events to stdout. Fluentbit forwards the stdout events to Elasticsearch.
- Github Container Repo: https://github.com/samsung-cnct/container-eventrouter
- Github Chart Repo: https://github.com/samsung-cnct/chart-eventrouter
- Image: https://quay.io/repository/samsung_cnct/eventrouter-container
- Chart: https://quay.io/application/samsung_cnct/eventrouter
Queryable Datastore: ElasticSearch
- Github Container Repo: https://github.com/samsung-cnct/container-elasticsearch
- Github Chart Repo: https://github.com/samsung-cnct/chart-elasticsearch
- Image: https://quay.io/repository/samsung_cnct/elasticsearch-container
- Chart: https://quay.io/application/samsung_cnct/elasticsearch-chart
Index Manager for Elasticsearch Curator
- Github Container Repo: https://github.com/samsung-cnct/container-curator
- Github Chart Repo: https://github.com/samsung-cnct/chart-curator
- Image: https://quay.io/repository/samsung_cnct/curator-container
- Chart: https://quay.io/application/samsung_cnct/curator
Data Visualization: Kibana
- Github Container Repo: https://github.com/samsung-cnct/container-kibana
- Github Chart Repo: https://github.com/samsung-cnct/chart-kibana
- Image: https://quay.io/repository/samsung_cnct/kibana-container
- Chart: https://quay.io/application/samsung_cnct/kibana-chart
This project is built with the helm application registry and pushed to quay. Unfortunately the helm registry plugin doesn't play nicely with standard helm dependencies. The vendored files for a given release (stable or alpha) are built in the CI process and saved off to quay as a built package.
Please see the CI definition and the dependency management script for details.
- Fork it!
- Create your feature branch:
git checkout -b my-new-feature
- Commit your changes:
git commit -am 'Add some feature'
- Push to the branch:
git push origin my-new-feature
- Submit a pull request :D
Created and maintained by the Samsung Cloud Native Computing Team.