Observability with Elasticsearch

Introduction

Elastic Stack (ELK) Docker Composition, reporting APM client + server side metrics and traces to APM Server.

Suitable for Demoing and MVPs.

Stack Version: 8.5.3 🎉 - Based on Official Elastic Docker Images

You can change Elastic Stack version by setting ELK_VERSION in .env file and rebuild your images. Any version >= 8.0.0 is compatible with this template.

Based on elastdocker

Main Features 📜

Elasticsearch, configured as a Production Single Node Cluster.
Security Enabled on Elasticsearch and Kibana.
APM Server receives Non secure connections.
Use Docker-Compose and .env to configure your entire stack parameters.
Persist Elasticsearch's Keystore and SSL Certifications.

Requirements

Docker 20.05 or higher
Docker-Compose 1.29 or higher
4GB RAM (For Windows and MacOS make sure Docker's VM has more than 4GB+ memory.)

Setup

Clone the Repository
```
git clone $ADDR
```
copy from .env.sample and change ELASTIC_PASSWORD and ELASTIC_APM_SECRET_TOKEN .
```
$ cp .env.sample .env
```
Initialize Elasticsearch Keystore and TLS Self-Signed Certificates
```
$ make setup
```
For Linux's docker hosts only. By default virtual memory is not enough so run the next command as root sysctl -w vm.max_map_count=262144

Start Elastic Stack

$ make elk           <OR>         $ docker-compose up -d		<OR>		$ docker compose up -d

Visit Kibana at https://localhost:5601/app/apm to see the APM traces from our webapp

Default Username: elastic, Password: changeme

Configuration

Some Configuration are parameterized in the .env file.
- ELASTIC_PASSWORD, user elastic's password (default: changeme pls).
- ELK_VERSION Elastic Stack Version (default: 8.3.2)
- ELASTICSEARCH_HEAP, how much Elasticsearch allocate from memory (default: 1GB -good for development only-)
- Other configurations which their such as cluster name, and node name, etc.
Elasticsearch Configuration in elasticsearch.yml at ./elasticsearch/config.
Kibana Configuration in kibana.yml at ./kibana/config.

Setting Up Keystore

You can extend the Keystore generation script by adding keys to ./setup/keystore.sh script. (e.g Add S3 Snapshot Repository Credentials)

To Re-generate Keystore:

make keystore

Notes

⚠️ Elasticsearch HTTP layer is using SSL, thus mean you need to configure your elasticsearch clients with the CA in secrets/certs/ca/ca.crt, or configure client to ignore SSL Certificate Verification (e.g --insecure in curl).
Makefile is a wrapper around Docker-Compose commands, use make help to know every command.
Elasticsearch will save its data to a volume named elasticsearch-data
Elasticsearch Keystore (that contains passwords and credentials) and SSL Certificate are generated in the ./secrets directory by the setup command.
Make sure to run make setup if you changed ELASTIC_PASSWORD and to restart the stack afterwards.
Important: For Linux Users it's recommended to set the following configuration (run as root)
```
sysctl -w vm.max_map_count=262144
```
By default, Virtual Memory is not enough.

License

Contribution

PR(s) are Open and Welcomed.

mojtabaakbari221b / docker-elastic-observability-with-apm