digital-asset / daml-platform-observability-example

Project showcasing observability features of the Daml Platform.

Daml Enterprise - Observability Example

This project demonstrates a Daml Enterprise deployment's observability features, along with example Grafana dashboards.

The project is self-contained: providing scripts to generate requests for which metrics are collected for display in the Grafana dashboards. It requires Daml 2.6.5 or newer and has been tested on MacOS and Linux.

This project is provided for illustrative purposes only, the various configurations are tailored for local run and may not fit other use cases.

This repository does not accept contributions at the moment.

Use the Makefile — run make help for available commands!

TL;DR Quick Start

To quickly get up and running, make sure you have the prerequisites installed and then:

• Ensure you have enough resources allocated to running the example containers; if a memory resource limit is reached then a container will be randomly killed. In particular, the daml_observability_canton container can reach a steady state memory usage of 4GB.
• Deploy the containers locally, which include several Daml Enterprise components, Prometheus, Grafana, and related containers. This is done with the command: docker compose up.
• Start the request generation. There are three scripts that generate the requests and running them in multiple terminals is best. The scripts are:
  • scripts/generate-load.sh: this runs the ledger API conformance suite in a loop (10 times by default), it will generate gRPC traffic that has successful and unsuccessful return codes.
  • scripts/send-json-api-requests.sh: this generates HTTP traffic against the HTTP JSON API Service which has successful and unsuccessful requests.
  • scripts/send-trigger-requests.sh: generates traffic for the Trigger Service.
• Login to the Grafana dashboard at http://localhost:3000/dashboards. The default user and password: digitalasset. You can set the time range to 5 minutes and refresh to 10seconds to see results quickly.
• When you stop everything, it is recommended to cleanup volumes to start fresh next time.

You should see a dashboard like this:

Prerequisites

• Docker
• Docker Compose plugin 2.x
• Artifactory credentials to access our private Canton Enterprise container images

Docker Compose will automatically build the image for the HTTP JSON API service from the release JAR file.

The .env file has environment variables to select which Canton and SDK versions are being used. See this section below for more details. Please be aware that different Daml Enterprise versions may not generate all the metrics in the Grafana dashboards so they may not show up in the dashboard.

Components

Docker Compose will start the following services:

• PostgreSQL database server
• Daml Platform
  • All-in-one Canton node (domain topology manager, mediator, sequencer and participant)
  • HTTP JSON API service
  • Trigger service
• Monitoring
  • Prometheus 2.x
  • Grafana 9.x
  • Node Exporter
  • Loki + Promtail 2.x

Prometheus and Loki are preconfigured as datasources for Grafana. You can add other services/exporters in the Docker compose file and scrape them changing the Prometheus configuration.

Startup

Start everything (blocking command, show all logs):

docker compose up

Start everything (detached: background, not showing logs)

docker compose up -d

If you see the error message no configuration file provided: not found please check that you are placed at the root of this project.

Starting a Remote Canton Console

docker exec -it daml_observability_canton_console bin/canton -c /canton/config/console.conf

Stopping

• If you used a blocking docker compose up, just cancel via keyboard with [Ctrl]+[c]

• If you detached compose: docker compose down

Cleanup Everything

Stop everything, remove networks and all Canton, Prometheus & Grafana data stored in volumes:

docker compose down --volumes

Important Endpoints to Explore

• Prometheus: http://localhost:9090/
• Grafana: http://localhost:3000/ (default user and password: digitalasset)
• Loki: http://localhost:5000/ (default user and password: digitalasset)
• Participant's Ledger API endpoint: http://localhost:10011/

Check all exposed services/ports in the Docker compose YAML definition.

Canton Console

docker exec -it daml_observability_canton_console bin/canton -c /canton/config/console.conf

Logs

docker logs daml_observability_postgres
docker logs daml_observability_prometheus
docker logs daml_observability_grafana

You can open multiple terminals and follow logs (blocking command) of a specific container:

docker logs -f daml_observability_postgres
docker logs -f daml_observability_prometheus
docker logs -f daml_observability_grafana

Configuration

Canton and HTTP JSON API Log Level

For the Canton node and HTTP JSON API service only, you can change LOG_LEVEL in the .env file:

LOG_LEVEL=DEBUG

Prometheus

prometheus.yml [documentation]

Reload or restart on changes:

• Reload:
  • Signal: docker exec -it daml_observability_prometheus -- kill -HUP 1
  • HTTP: curl -X POST http://localhost:9090/-/reload
• Restart: docker compose restart prometheus

Grafana

• Grafana itself: grafana.ini [documentation]
• Data sources: datasources.yml [documentation]
• Dashboard providers: dashboards.yml [documentation]

Restart on changes: docker compose restart grafana

Dashboards

All dashboards (JSON files) are auto-loaded from directory grafana/dashboards/

• Automatic: place your JSON files in the folder (loaded at startup, reloaded every 30 seconds)
• Manual: create/edit via Grafana UI

Examples Source

• Prometheus and Grafana [source]
• Node exporter full [source]
• Loki and Promtail [source]

Accessing Daml Enterprise container images

• Get an access key from Digital Asset. This step needs to be performed once. If you haven't logged in before, you can get an access key by logging in to digitalasset.jfrog.io and generating an identity token on your Artifactory profile page. If your email is [e-mail], your Artifactory username is john.doe.

• Make sure the Docker daemon is logged in to digitalasset-docker.jfrog.io:

docker login digitalasset-docker.jfrog.io -u <your-username>

• Assign the .env file environment variables CANTON_IMAGE and CANTON_VERSION to the version you want. Several .env example configurations are shown below.

• This example retrieves the Daml Enterprise container images for Daml v2.6.5:

CANTON_IMAGE=digitalasset-docker.jfrog.io/digitalasset/canton-enterprise
CANTON_VERSION=2.6.5
SDK_VERSION=2.6.5
LOG_LEVEL=INFO

• This example is for a weekly snapshot release that can be used to verify a bug is fixed. This requires help from the support team.

CANTON_IMAGE=digitalasset-docker.jfrog.io/digitalasset/canton-enterprise
CANTON_VERSION=20221129
SDK_VERSION=2.5.0-snapshot.20221201.11065.0.caac1d10
LOG_LEVEL=INFO

• This example shows how to switch from the Daml Enterprise Docker images to the open source version (pulled from Docker Hub) set the variables to a similar value below. This example retrieves the v2.5.0 images for the open source repo of Daml.

CANTON_IMAGE=digitalasset/canton-open-source
CANTON_VERSION=2.6.0
SDK_VERSION=2.6.0
LOG_LEVEL=INFO

License

You may use the contents of this repository in parts or in whole according to the 0BSD license.

Copyright © 2023 Digital Asset (Switzerland) GmbH and/or its affiliates

Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted.

THE SOFTWARE IS PROVIDED “AS IS” AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.