| permalink |
|---|
/guides/app-logging/ |
OpenShift provides a preconfigured EFK(Elasticsearch, FluentD, and Kibana) stack, which DevOps uses to aggregate all container logs. You install the EFK stack by using the ansible-playbook command that resides in the openshift-ansible repository. After the installation completes, the EFK deployments reside inside the openshift-logging namespace of the OpenShift cluster.
Set up two separate EFK stacks. One stack is for EFK deployments that are dedicated exclusively for OpenShift and Kubernetes logs. The other EFK stack is for user applications.
There are several advantages to having an ops EFK stack.
-
It is much easier to find applications logs in Kibana, as it is not polluted with all the Kubernetes logs happening all the time.
-
It provides more flexibility for memory allocations, because users can independently assign system memories for each of the EFK deployments.
Installing EFK on OpenShift requires an OpenShift host inventory file and the openshift-ansible playbook for logging the installation. After you create the inventory file, add the following Ansible variables to the [OSEv3:vars] section of the inventory file.
openshift_logging_use_ops=True
openshift_logging_es_ops_nodeselector={“node-role.kubernetes.io/infra”:“true”}
openshift_logging_es_nodeselector={"node-role.kubernetes.io/infra":"true"}
openshift_logging_es_ops_memory_limit=5G
openshift_logging_es_memory_limit=3GThere are many other Ansible variables provided by OpenShift for advanced fine-tuning the EFK stack. Detailed information about the installation and configuration can be found on OpenShift documentation Aggregating Container Logs.
Let’s examine each of the variables in detail.
-
Setting
openshift_logging_use_ops=Trueinstructs Ansible to install two EFKs with a dedicated ops deployment. -
openshift_logging_es_nodeselectorandopenshift_logging_es_ops_nodeselectorare two variables required by the openshift-logging ansible-playbook to install Elasticsearch, You usually set these variables to the infra nodes. -
openshift_logging_es_memory_limitandopenshift_logging_es_ops_memory_limitare self-explanatory and can be set according to your preference. For stable operation, allocate at least 2 GB of memory for each of the Elasticsearch deployments. If memory is not specified explicitly, OpenShift allocates 16 GB of memory for each of the Elasticsearch deployment by default. It is highly recommended to install openshift-logging on systems with at least 32 GB of RAM whenopenshift_logging_use_opsis set to true.
After all the variables are set in the inventory file, the user would need to run the ansible-playbook command to install the EFK stack onto its current OpenShift Cluster.
ansible-playbook -i <inventory_file> openshift-ansible/playbooks/openshift-logging/config.yml -e openshift_logging_install_logging=true
The installation process can take a few minutes to complete. After the installation is completed without any error, you can see the following pods that are running in the openshift-logging namespace.
[root@rhel-2EFK ~]# oc get pods -n openshift-logging
NAME READY STATUS RESTARTS AGE
logging-curator-1565163000-9fvpf 0/1 Completed 0 20h
logging-curator-ops-1565163000-5l5tx 0/1 Completed 0 20h
logging-es-data-master-iay9qoim-4-cbtjg 2/2 Running 0 3d
logging-es-ops-data-master-hsmsi5l8-3-vlrgs 2/2 Running 0 3d
logging-fluentd-vssj2 1/1 Running 1 3d
logging-kibana-2-tplkv 2/2 Running 6 4d
logging-kibana-ops-1-bgl8k 2/2 Running 2 3dYou can see that Elasticsearch, Kibana, and Curator all have two types of pods: the main one and a secondary set of pods with -ops postfix, except for Fluentd. This situation is expected as the split of application logs, and the OpenShift operations logs occurs inside the single Fluentd instance. All of the node system logs and the logs from projects, default, openshift, and openshift-infra are considered as operation (or ops) logs and are aggregated to the ops Elasticsearch server. The logs from any other namespaces are aggregated to the main Elasticsearch server.
The openshift-logging ansible-playbook also exposes two routes for external access to the Kibana and ops Kibana web console.
[root@rhel-2EFK ~]# oc get routes -n openshift-logging
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
logging-kibana kibana.apps.9.37.135.153.nip.io logging-kibana <all> reencrypt/Redirect None
logging-kibana-ops kibana-ops.apps.9.37.135.153.nip.io logging-kibana-ops <all> reencrypt/Redirect NoneIf you examine the logging-kibana-ops URL, all the operation logs generated by Openshift and Kubernetes are visible on Kibana’s Discover page.
Figure 1: Kibana ops page with operation log entries
Before you use the logging-kibana for application logs, make sure that the application is already deployed in a namespace NOT from one of the ops namespaces. The ops namespaces are default, openshift, and openshift-infra.
To fully take advantage of the Kibana’s dashboard functions, output the application in JSON format. Kibana is then able to process the data from each individual field of the JSON object to create customized visualization for that field.
See the Kibana dashboard page by using the routes URL https://kibana.apps.9.37.135.153.nip.io. Log in using your OpenShift user and password, then the page should redirect you to Kibana’s Discover page where the newest logs of the selected index are being streamed. Select the project.\* index to view the application logs generated by the deployed application.
Figure 2: Kibana page with the application log entries
The project.\* index contains only a set of default fields at the start, which does not include all of the fields from the deployed application’s JSON log object. Therefore, the index needs to be refreshed to have all the fields from the application’s log object available to Kibana.
To refresh the index, click on the Management option on the left pane.
Click Index Pattern, and find the project.\* index in Index Pattern. Then, click the refresh fields button, which is on the right. After the Kibana is updated with all the available fields in the project.\* index, import the preconfigured dashboards to view the application logs.
Figure 3: Index refresh button on Kibana
To import the dashboard and its associated objects, navigate back to the Management page and click Saved Objects. Click Import and select the dashboard file. When prompted, click the Yes, overwrite all option
Head back to the Dashboard page and enjoy the logging on the dashboard that you imported previously.
Figure 4: Kibana dashboard for Open Liberty application logs
If changes need to be made for the installed EFK stack, rerun the ansible-playbook installation command with updated ansible variables values in the inventory file. It’s always safe to reinstall openshift-logging by running this command. If the aggregated containers logging is no longer needed in the current cluster, you can use the same ansible-playbook command to uninstall the openshift-logging feature. Uninstall the feature by setting the openshift_logging_install_logging variable to False.