Generate standardized PagoPA's Operational Excellence dashboards from OpenApi specs.
This tool can do the following:
Template | Description | Status |
---|---|---|
azure-dashboard | Programmatically create a Terraform representation of an PagoPA's Azure Dashboard | β OK |
azure-dashboard-raw | Programmatically create a JSON representation of a PagoPA's Azure Dashboard | β OK |
aws | Programmatically create a JSON representation of a PagoPA's CloudWatch Dashboard | βοΈ WIP |
grafana | Programmatically create a JSON representation of a PagoPA's Grafana Dashboard | β Planned |
It is distribuited as a Python package and it has two important components:
- The OpEx Dashboard client is a command-line tool for end users;
- The OpEx Dashboard library provides the logic for executing all the operations.
For each endpoint in the OpenApi spec there are three graphs:
- Availability: ratio with HTTP status codes greater than 499;
- Response codes: segmentation of all HTTP status codes;
- Response time: 95th percentile response time.
For each endpoint in the OpenApi spec there are two alarms:
- Availability: threshold at 99%;
- Response time: threshold at 1 second.
For each alarm, the following variables can be cofigured:
- Timespan (Default: 5m) The aggregation.
- Evaluation frequency (Default: 10) The frequency (in minutes) at which rule condition should be evaluated.
- Time window (Default: 20) The time window for which data needs to be fetched for query (must be greater than or equal to evalation frequency).
- Event occurrences number (Default: 1) The number of events within a time window needed to raise an alert.
Each of them can be configured at openapi level and can be overridden for each alert at endpoint level.
NOTE: Maximum number of event occurrences is defined by time window/timespan. i.e. within a time window of 30m, with a timespan of 5m, a maximum of 6 events can occur.
These values can be configured, look at Overrides paragraph.
To generate a dashbord template there are several way. You can use the opex_dashboard generate --help
to learn about this process:
Usage: opex_dashboard generate [OPTIONS]
Generate enables you to create a dashboard definition that could be imported
in a compatible provider.
Options:
-t, --template-name [azure-dashboard|azure-dashboard-raw]
Name of the template. [required]
-c, --config-file FILENAME A yaml file with all params to create the
template, use - value to get input from
stdin. [required]
--package PATH Save the template as a package, by default
it creates a folder in the current
directory.
--help Show this message and exit.
The first step is to create a configuration file:
cat <<EOF > config.yaml
oa3_spec: https://raw.githubusercontent.com/pagopa/opex-dashboard/main/test/data/io_backend.yaml
name: My dashboard
location: West Europe
timespan: 5m
data_source: /subscriptions/uuid/resourceGroups/my-rg/providers/Microsoft.Network/applicationGateways/my-gtw
resource_type: app-gateway|api-management, default app-gateway
action_groups:
- /subscriptions/uuid/resourceGroups/my-rg/providers/microsoft.insights/actionGroups/my-action-group-email
- /subscriptions/uuid/resourceGroups/my-rg/providers/microsoft.insights/actionGroups/my-action-group-slack
EOF
You can find practical example of configuration files in the examples folder.
This is the most convenient and rapid way. Generate the dashboard:
cat config.yaml | docker run -i \
ghcr.io/pagopa/opex-dashboard:latest generate \
--template-name azure-dashboard \
--config-file -
-
is a special value, it indicates stdin as input. Alternatively, it is always
possible to bind mount a volume and load the configuration file, as well as the
OA3 spec, from the volume:
docker run -v $(pwd):/home/nonroot/myfolder:Z \
ghcr.io/pagopa/opex-dashboard:latest generate \
--template-name azure-dashboard \
--config-file myfolder/config.yaml
You can also load an OA3 spec by mounting it inside the container. In this case
you must pay attention to the path of the spec beacuse it relies on the path of
the mounted volume. An example of config.yaml
in this scenario:
oa3_spec: myfolder/oa3_spec.yaml
name: My dashboard
location: West Europe
timespan: 5m
evaluation_frequency: 5
evaluation_time_window: 30
event_occurrences: 3
data_source: /subscriptions/uuid/resourceGroups/my-rg/providers/Microsoft.Network/applicationGateways/my-gtw
resource_type: app-gateway
action_groups:
- /subscriptions/uuid/resourceGroups/my-rg/providers/microsoft.insights/actionGroups/my-action-group-email
- /subscriptions/uuid/resourceGroups/my-rg/providers/microsoft.insights/actionGroups/my-action-group-slack
EOF
There is a convenient Dockerfile that you can use to build the image from scratch on your localhost:
git clone https://github.com/pagopa/opex-dashboard.git
cd opex-dashboard
docker build -t opexd .
You can choose either between cloning the repository and manually installing the package (with or without venv) or by pointing directly to this repository.
By cloning the source:
git clone https://github.com/pagopa/opex-dashboard.git && \
cd opex-dashboard && \
pip install --user -e .
Or download the dependency to the repository:
pip install --user 'opex_dashboard @ git+https://github.com/pagopa/opex-dashboard'
In any case, you'll be able to create the dashboard by using the CLI:
opex_dashboard generate \
--template-name azure-dashboard \
--config-file config.yaml
Using the option --package
with a Terraform template, the CLI creates a
package using PagoPA's conventions. The package has this structure:
<template_name>/
|- .env/
| |- dev/
| | |- backend.ini
| | |- backend.tfvars
| | |- terraform.tfvars
| |- uat/
| | |- backend.ini
| | |- backend.tfvars
| | |- terraform.tfvars
| |- prod/
| |- backend.ini
| |- backend.tfvars
| |- terraform.tfvars
|- terraform.sh
|- 01_opex.tf
|- 99_main.tf
|- 99_variables.tf
If you are running the script inside a container you can pass to --package
the path of the bind mounted volume. This is an example:
docker run -v $(pwd):/home/nonroot/myfolder:Z \
ghcr.io/pagopa/opex-dashboard:latest generate \
--template-name azure-dashboard \
--config-file myfolder/config.yaml \
--package myfolder
For each template you can overrides OpenAPI values by using the overrides
block in the configuration file, see this
example for more.
To overrides hosts add this snippet tou your config (http(s)
prefix is mandatory):
overrides:
hosts:
- example.com
- github.com
To overrides endpoint's settings add this snippet tou your config:
overrides:
endpoints:
/onboarding/info: # This is the endpoint in the OpenApi spec
availability_threshold: 0.95 # Default: 99%
availability_evaluation_frequency: 30 # Default: 10
availability_evaluation_time_window: 50 # Default: 20
availability_event_occurrences: 3 # Default: 1
response_time_threshold: 2 # Default: 1
response_time_evaluation_frequency: 35 # Default: 10
response_time_evaluation_time_window: 55 # Default: 20
response_time_event_occurrences: 5 # Default: 1
The development environment leverages on pipenv. Pipfile contains several convenient scripts to ease the development process.
To set up your localhost:
pipenv install --dev
pipenv run install_local
pipenv run opex_dashboard generate --help
This is a four steps process.
By using the Django template
language,
create a new template in src/opex_dashboard/template
. As example, consider
the following content for the file src/opex_dashboard/template/mytemplate.json
:
{% load mul %}
{
"widgets": [
{% for endpoint in endpoints %}
{
"height": 8,
"width": 8,
"x": 0,
"y": {{ forloop.counter0|mul:8 }},
"type": "log",
"properties": {
"query": "myquery#1",
"region": "eu-south-1",
"stacked": "false",
"title": "Availability - {{endpoint}}",
"view": "timeSeries"
}
},
{
"height": 8,
"width": 8,
"x": 8,
"y": {{ forloop.counter0|mul:8 }},
"type": "log",
"properties": {
"query": "myquery#2",
"region": "eu-south-1",
"stacked": "false",
"title": "Response Codes - {{endpoint}}",
"view": "timeSeries"
}
},
{
"height": 8,
"width": 8,
"x": 16,
"y": {{ forloop.counter0|mul:8 }},
"type": "log",
"properties": {
"query": "myquery#3",
"region": "eu-south-1",
"stacked": "false",
"title": "Response time - {{endpoint}}",
"view": "timeSeries"
}
}
{% endfor %}
]
}
mul
is a custom tag. You can find its implementation in tags
folder.
A builder is where you apply the business logic. There is a base Builder you must inherit from. Consider the following builder, which takes an OA3 spec to take all its endpoints:
from typing import Dict, List, Any
from urllib.parse import urlparse
from opex_dashboard.builders.base import Builder
from opex_dashboard.resolver import OA3Resolver
class MyDashboardBuilder(Builder):
_oa3_spec: Dict[str, Any]
def __init__(self, resolver: OA3Resolver) -> None:
"""Create a MyDashbordBuilder object
"""
self._oa3_spec = resolver.resolve()
super().__init__(template="mytemplate.json")
def produce(self, values: Dict[str, Any] = {}) -> str:
"""Render the template by merging base properties, given values, and information extracted form OA3 spec
Returns:
str: The rendered template
"""
endpoint_default_values = {
"availability_threshold": 0.99,
"response_time_threshold": 1,
}
if "servers" in self._oa3_spec:
self._properties["hosts"] = []
self._properties["endpoints"] = {}
for server in self._oa3_spec["servers"]:
url = urlparse(server["url"])
self._properties["hosts"].append(url.netloc)
for p in list(self._oa3_spec["paths"].keys()):
self._properties["endpoints"][f"{url.path}/{p[1:]}"] = endpoint_default_values
else:
base_path = self._oa3_spec["basePath"]
self._properties["hosts"] = [self._oa3_spec["host"]]
self._properties["endpoints"] = {}
for p in self._oa3_spec["paths"].keys():
self._properties["endpoints"][f"{base_path}/{p[1:]}"] = endpoint_default_values
return super().produce(values)
Save it as src/opex_dashboard/builders/my_dashboard_builder.py
.
Modify src/opex_dashboard/builder_factory.py
to create the new dashboard:
# ...
from opex_dashboard.builders.my_dashboard_builder import MyDashboardBuilder
# ...
def create_my_dashboard_builder(**args: Optional[Any]) -> Optional[Builder]:
inputs = normalize_params(args, {
"resolver": OA3Resolver,
})
return MyDashboardBuilder(**inputs)
# ...
def create_builder(template_type: str, **args: Optional[Any]) -> Optional[Builder]:
# ...
builders = {
# ...
"my-dashboard": create_my_dashboard_builder,
# ...
}
# ...
Update src/opex_dashboard/commands/generate.py
:
# ...
@click.option("--template-name", "-t",
required=True,
type=click.Choice([..., "my-dashboard"]),
help="Name of the template.")
# ...
In the test folder there are lots of tests, consider to upgrade it after develop a new template.
You can add the core library as a dependecy like any common Python package. unfortunately we didn't publish the package to PyPI (Python Package Index) but you can directly use GitHub:
pip install 'opex_dashboard @ git+https://github.com/pagopa/opex-dashboard'