Loggregator is a BOSH release deployed as a part of cf-release. Loggregator provides a highly-available (HA) and secure stream of logs and metrics for all applications and components on Cloud Foundry. It does so while not disrupting the behavior of the the applications and components on the platform (i.e. "backpressure").
The Loggregator Design Notes presents an overview of Loggregator components and architecture.
- Generating TLS Certificates
- Streaming Application Logs
- Consuming the Firehose
- Emitting Logs and Metrics into Loggregator
- Tools for Testing and Monitoring Loggregator
- Troubleshooting Reliability
- More Resources and Documentation
In order to secure transport of logs throughout the system, Loggregator needs
several certificates to be generated for each of the components. You start
this process by running the scripts/generate-certs, and then configure the
corresponding certificates for each of the components you are deploying. For
more details see our Certificate Configuration README
Any user of Cloud Foundry can experience Loggregator by using two simple interfaces for streaming application specific logs. These do not require any special User Account and Authentication(UAA) Scope.
The fastest way to see your logs is by running the cf logs command using the
CF CLI. Check the Cloud Foundry CLI docs for more details.
If you’d like to save all logs for an application in a third party or custom tool that expects the syslog format, a log drain allows you to do so. Check the Cloud Foundry docs for more details.
Loggregator does not provide any guaruntees around the order of delivery of logs in streams. That said, there is enough precision in the timestamp provided by diego that streaming clients can batch and order streams as they receive them. This is done by the cf cli and most other streaming clients.
For syslog ingestion it also possible to ensure log order. See the details in cf-syslog-drain-release
The firehose is an aggregated stream of all application logs and component metrics on the platform. This allows operators to ensure they capture all logs within a microservice architecture as well as monitor the health of their platform. See the Firehose README.
In order to consume the firehose you’ll need the doppler.firehose scope from
UAA. For more details see the Firehose README.
Once you have configured appropriate authentication scope you are ready to start developing a nozzle for the firehose. See our Nozzle community page for more details about existing nozzles and how to get started.
Loggregator and other Cloud Foundry components emit regular messages through the Firehose that monitor the health, throughput, and details of a component's operations. For more detials about Loggregator’s metrics see our Loggregator Metrics README.
For components of Cloud Foundry or standalone BOSH deployments, Loggregator provides a set of tools for emitting Logs and Metrics.
The RLP is the v2 implementation of the Loggregator API. This component is intended to be a replacement for traffic controller.
By default, the RLP communicates with clients via gRPC over mutual TLS. To enable HTTP access to the Reverse Log Proxy, deploy the RLP Gateway. The RLP Gateway API is documented in the Loggregatgor repository
The RLP Gateway is deployed via this ops-file:
bosh -d cf deploy cf-deployment.yml -o manifests/operations/add-rlp-gateway-to-cf.yml
Standalone Loggregator provides logging support for services that do not require a CloudFoundry.
TrafficController has several CloudFoundry concerns. Therefore, there is an operations file to deploy standalone Loggregator without it.
bosh -e lite deploy -d loggregator manifests/loggregator.yml \
--vars-store=/tmp/loggregator-vars.yml
The Loggregator API is a replacement of the Dropsonde Protocol. Loggregator API defines an envelope structure which packages logs and metrics in a common format for distribution throughout Loggregator. See the Loggregator API README for more details.
The Metron Agent listens on both UDP and gRPC endpoints for multiple versions of Loggregator API which it forwards onto the Firehose. To include Metron in your component or deployment see the Setting up up Metron README.
The statsd-injector is a companion component to Metron and allows use of the statsd metric aggregator format. For more see the [statsd-injector README][statsd-injector-readme].
For some components (such as UAA) it makes sense to route logs separate from the Firehose. The syslog release using rsyslog to accomplish this. For more information see the syslog-release README (note this release is maintianed by the bosh team).
Loggregator provides a set of tools for testing the performance and reliability of your loggregator installation. See the loggregator tools repo for more details.
The Loggregator Operator Guidebook provides details for scaling and managing Loggregator along with detailed results of capacity planning tests.
In addition to the scaling recommendations above, it is important that the resources for Loggregator are dedicate VM’s with similar footprints to those used in our capacity tests. Even if you are within the bounds of the scaling recommendations it may be useful to scale Loggregator and Nozzle components aggressively to rule out scaling as a major cause log loss.
Another common reason for log loss is due to an application producing a large amount of logs that drown out the logs from other application on the cell it is running on. To identify and monitor for this behavior the Loggregator team has created a Noisy Neighbor Nozzle and CLI Tool. This tool will help operators quickly identify and take action on noise producing applications. Instruction for deploying and using this nozzle are in the repo.
We communicate our long term planning using a Product Road Map, and are always looking to gather feedback and input from Loggregator operators. Get in touch or file an issue if you have feature suggestions you'd like to see added.
Items marked as "In Flight" on the Roadmap are tracked as new Features in Pivotal Tracker.