The LogDNA agent is a resource-efficient log collection client that forwards logs to LogDNA. This version of the agent is written in Rust to ensure maximum performance, and when coupled with LogDNA's web application, provides a powerful log management tool for distributed systems, including Kubernetes clusters.
The agent is supported for Kubernetes 1.9+ and Red Hat OpenShift 4.5+ environments.
You can use the manifest YAML files in this repository or use Helm package manager to deploy the agent in your Kubernetes cluster.
NOTE: The Kubernetes manifest YAML files in this repository (and referenced in this documentation) describe the version of the LogDNA Agent in the current commit tree and no other version of the LogDNA agent. If you apply the Kubernetes manifest found in the current tree then your cluster will be running the version described by the current commit which may not be ready for general use.
Follow the full instructions in the documentation to deploy on Kubernetes using resource files.
Visit the documentation to install the agent on Kubernetes using Helm.
Follow the instructions for deploying the agent on Red Hat®️ OpenShift®️.
By default the agent will run as root. Below are environment-specific instructions for running the agent as a non-root user.
If you configure the LogDNA Agent to run as non-root, review the documentation about enabling "statefulness" for the agent.
More information about managing your deployments is documented for Kubernetes or OpenShift. This includes topics such as
- Version specific upgrade paths
- Collecting system logs through Journald
Obtain the source code to build the agent from our GitHub repository. You can either download the source files as a .zip file, or clone the repository.
- Use the following commands to clone and then cd into this repository:
git clone https://github.com/logdna/logdna-agent-v2.git
cd logdna-agent-v2
Next, select how you want to build the agent: as a docker image or as a Linux binary.
To build a Docker image of the agent, ensure the Docker command is installed properly, verify that the Docker engine is running, and then run the following command:
make build-image
The resulting image can be found by listing the images:
foo@bar:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
logdna-agent-v2 dcd54a0 e471b3d8a409 22 seconds ago 135MBYou can also obtain the image and review our tagging scheme on DockerHub.
The agent requires v1.42+ of rustc cargo-make to build. If the proper versions of rustc and cargo are installed; then simply run the following command to build the agent:
cargo build --release
The compiled binary will be built to ./target/release/logdna-agent.
The agent accepts configuration from three different sources: environment variables, command line arguments and/or a
configuration YAML file. The default configuration yaml file is located at /etc/logdna/config.yaml. The following
options are available:
| Variable Name(s) | Description | Default |
|---|---|---|
LOGDNA_INGESTION_KEYDeprecated: LOGDNA_AGENT_KEY |
Required: The ingestion key associated with your LogDNA account | |
LOGDNA_CONFIG_FILEDeprecated: DEFAULT_CONF_FILE |
Path to the configuration yaml | /etc/logdna/config.yaml |
LOGDNA_HOSTDeprecated: LDLOGHOST |
The host to forward logs to | logs.logdna.com |
LOGDNA_ENDPOINTDeprecated: LDLOGPATH |
The endpoint to forward logs to | /logs/agent |
LOGDNA_USE_SSLDeprecated: LDLOGSSL |
Whether to use a SSL for sending logs | true |
LOGDNA_USE_COMPRESSIONDeprecated: COMPRESS |
Whether to compress logs before sending | true |
LOGDNA_GZIP_LEVELDeprecated: GZIP_COMPRESS_LEVEL |
If compression is enabled, this is the gzip compression level to use | 2 |
LOGDNA_HOSTNAME |
The hostname metadata to attach to lines forwarded from this agent | |
LOGDNA_IP |
The IP metadata to attach to lines forwarded from this agent | |
LOGDNA_TAGS |
Comma separated list of tags metadata to attach to lines forwarded from this agent | |
LOGDNA_MAC |
The MAC metadata to attach to lines forwarded from this agent | |
LOGDNA_LOG_DIRSDeprecated: LOG_DIRS |
Comma separated list of folders to recursively monitor for log events | /var/log/ |
LOGDNA_EXCLUSION_RULESDeprecated: LOGDNA_EXCLUDE |
Comma separated list of glob patterns to exclude files from monitoring 1 | /var/log/wtmp,/var/log/btmp,/var/log/utmp, /var/log/wtmpx,/var/log/btmpx,/var/log/utmpx, /var/log/asl/**,/var/log/sa/**,/var/log/sar*, /var/log/tallylog,/var/log/fluentd-buffers/**/*, /var/log/pods/**/* |
LOGDNA_EXCLUSION_REGEX_RULESDeprecated: LOGDNA_EXCLUDE_REGEX |
Comma separated list of regex patterns to exclude files from monitoring | |
LOGDNA_INCLUSION_RULESDeprecated: LOGDNA_INCLUDE |
Comma separated list of glob patterns to includes files for monitoring 1 | *.log,!(*.*) |
LOGDNA_INCLUSION_REGEX_RULESDeprecated: LOGDNA_INCLUDE_REGEX |
Comma separated list of regex patterns to include files from monitoring | |
LOGDNA_LINE_EXCLUSION_REGEX |
Comma separated list of regex patterns to exclude log lines. When set, the Agent will NOT send log lines that match any of these patterns. | |
LOGDNA_LINE_INCLUSION_REGEX |
Comma separated list of regex patterns to include log lines. When set, the Agent will ONLY send log lines that match any of these patterns. | |
LOGDNA_REDACT_REGEX |
Comma separated list of regex patterns used to mask matching sensitive information before sending it the log line. | |
LOGDNA_JOURNALD_PATHS |
Comma separated list of paths (directories or files) of journald paths to monitor | |
LOGDNA_LOOKBACK |
The lookback strategy on startup | smallfiles |
LOGDNA_USE_K8S_LOG_ENRICHMENT |
Determines whether the agent should query the K8s API to enrich log lines from other pods. | always |
LOGDNA_LOG_K8S_EVENTS |
Whether the agent should log Kubernetes resource events. This setting only affects tracking and logging Kubernetes resource changes via watches. When disabled, the agent may still query k8s metadata to enrich log lines from other pods depending on the value of LOGDNA_USE_K8S_LOG_ENRICHMENT setting value. |
never |
LOGDNA_DB_PATH |
The directory the agent will store it's state database. Note that the agent must have write access to the directory and be a persistent volume. | |
LOGDNA_METRICS_PORT |
The port number to expose a Prometheus endpoint target with the agent internal metrics. |
All regular expressions use Perl-style syntax with case sensitivity by default. If you don't
want to differentiate between capital and lower-case letters, use non-capturing groups with a flag: (?flags:exp),
for example:
(?i:my_case_insensitive_regex)
- We support this flavor of globber syntax.
To configure the DaemonSet, modify the envs section of the DaemonSet [spec.template.spec.containers.0.env]. For example, to change the hostname add the following environment variable to the env list:
env:
- name: LOGDNA_HOSTNAME
value: my-hostnameCheck out Kubernetes documentation for more information about injecting environment variables into applications!
The lookback strategy determines how the agent handles existing files on agent startup. This strategy is determined by the LOGDNA_LOOKBACK variable.
By default, the agent provides a "stateful", or persistent, collection of files that can be referenced whenever the agent is restarted, in order to return (or look back) to see any files that were ingested during the time that the agent was not running. The state directory location is defined using the LOGDNA_DB_PATH environment variable in the YAML file (the default path is /var/lib/logdna).
The valid values for this option are:
- When set to
none:- lookback is disabled, and LogDNA Agent will read new lines as those are added to the file, ignoring the lines that were written before the time the Agent restarted.
- When set to
smallfiles(default):- If there is information in the “state file”, use the last recorded state.
- If the file is not present in the “state file” and the file is less than 8KiB, start at the beginning. If the file is larger than 8KiB, start at the end.
- When set to
start:- If there is information in the “state file”, use the last recorded state.
- If the file is not present in the “state file”, start at the beginning.
Notes:
- If you configure the LogDNA Agent to run as non-root, review the documentation about enabling "statefulness" for the LogDNA Agent.
- When upgrading from LogDNA Agent version 3.0 to 3.1, the state file will initially be empty, so the lookback setting will be used for existing files. After that (i.e. on process restart), the state file will be present and will be used.
If the agent pods have access to journald log files or directories, monitoring can be enabled on them with the LOGDNA_JOURNALD_PATHS. Common values include /var/log/journal and /run/systemd/journal. To specify both, use a comma separated list: /var/log/journal,/run/systemd/journal.
Take a look at enabling journald monitoring for Kubernetes or OpenShift.
A Kubernetes event is exactly what it sounds like: a resource type that is automatically generated when state changes occur in other resources, or when errors or other messages manifest across the system. Monitoring events is useful for debugging your Kubernetes cluster.
By default, the LogDNA agent captures Kubernetes events (and OpenShift events, as well, since OpenShift is built on top of Kubernetes clusters).
To control whether the LogDNA agent collects Kubernetes events, configure the LOGDNA_LOG_K8s_EVENTS environment variable using on of these two values:
always- Always capture eventsnever- Never capture events Note: The default option isnever.
⚠️ Due to a "won't fix" bug in the Kubernetes API, the LogDNA agent collects events from the entire cluster, including multiple nodes. To prevent duplicate logs when running multiple pods, the LogDNA agent pods defer responsibilty of capturing events to the oldest pod in the cluster. If that pod is down, the next oldest LogDNA agent pod will take over responsibility and continue from where the previous pod left off.
You can define rules, using regex (regular expressions), to control what log data is ingested:
- include only specific log lines (with
LOGDNA_LINE_INCLUSION_REGEX) - exclude specific log lines (with
LOGDNA_LINE_EXCLUSION_REGEX) - redact parts of a log line (with
LOGDNA_REDACT_REGEX)
For example, you can identify certain types of logs that are noisy and not needed at all, and then write a regex pattern to match those files and preclude them from ingestion. Conversely, you can use regex expressions to match the log lines that you DO want in to include, and ingest only those log lines. Additionally, you can use the environment variable LOGDNA_REDACT_REGEX to remove certain parts of a log line. Any redacted data is replaced with [REDACTED].
To access our library of common regex patterns, refer to our regex library documentation.
Notes:
- Exclusion rules overwrite inclusion rules. That is, for a line to be ingested, it should match all inclusion rules (if any) and not match any exclusion rule.
- To preclude entire log files from being monitored, use the
LOGDNA_EXCLUSION_RULESor theLOGDNA_EXCLUSION_REGEX_RULESenvironment variable. - Note that we use commas as separators for environment variable values, making it not possible to use the comma character (,) as a valid value. We are addressing this limitation in upcoming versions. If you need to use the comma character in a regular expression, use the unicode character reference:
\u002C, for example:hello\u002C worldmatcheshello, world. - All regular expressions are case sensitive by default. If you don't want to differentiate between upper and lower-case letters, use non-capturing groups with a flag: (?flags:exp), for example: (?i:my_case_insensitive_regex)
The agent is deployed as a Kubernetes DaemonSet, creating one pod per node selected. The agent collects logs of all the pods in the node. The resource requirements of the agent are in direct relation to the amount of pods per node, and the amount of logs producer per pod.
The agent requires at least 128Mib and no more than 512Mib of memory. It requires at least
twenty millicpu (20m).
Different features can also increase resource utilization. When line exclusion/inclusion or redaction rules are specified, you can expect to additional CPU consumption per line and per regex rule defined. When Kubernetes event logging is enabled (disabled by default), additional CPU usage will occur on the oldest agent pod.
We do not recommend placing traffic shaping or CPU limits on the agent to ensure data can be sent to our log ingestion service.
The LogDNA agent records internal metrics that can be relevant for monitoring and alerting, such as number log files currently tracked or number of bytes parsed, along with process status information. Check out the documentation for internal metrics for more information.