My system monitors website availability over the network and passes these events through an Aiven Kafka instance into an Aiven PostgreSQL database.
- Python 3.8
- PostgreSQL 12 - main storage
- Kafka 12.4 - message broker between checker and db-writer instances
- Redis - for tracking last check time
- Docker - not nessesary, but it's convenient for production deployment
- Download source code to your local machine (
git clone git@github.com:petersmol/monitoring.git
) - Update the configuration file (see "Configuring" section below)
- Build the docker image:
docker build -t monitoring .
- Run consumer container:
docker run monitoring python -u run_consumer.py
- Run checker container:
docker run monitoring python -u run_checker.py
You can also use prebuilt docker image and mount your config/
folder into /app/config
.
Download source code to your local machine and run the following commands:
pip install pipenv
pipenv install
pipenv shell
Then in pipenv shell you can run scripts in root project folder:
python run_consumer.py
- start DB writer servicepython run_checker.py
- start website checker servicepython run_checker_once.py
- perform all checks once and exitpython db_statistics.py
- quick look into the existing DB entries
- Push builded docker image to AWS ECR following their instructions
- Create new ECS task with 2 containers
consumer
andchecker
. Use the same image for both and add custom entrypoints:python, -u, run_consumer.py
andpython, -u, run_checker.py
accordingly.
Copy config/config.example.yaml to config/config.yaml
, then follow the instructions below.
- Order Aiven Kafka instance and create topic.
- Download Kafka's
ca.pem
,service.cert
andservice.key
toconfig/
folder - Fill
kafka.connect.bootstrap_servers
in config.yaml with the actual host:port - Fill
kafka.topic
in config.yaml with topic naame.
- Order Aiven PostgreSQL instance. You can use the default database or create new one if needed.
- Fill
postgresql_uri
in config.yaml with your database URI.
- Fill
redis_uri
in config.yaml with your redis URI.
You can add an arbitrary number of checks into the config. Following parameters are accepted
Parameter | Required | Description |
---|---|---|
url | Yes | page to request, e.g. https://example.com/index.html . At the moment, only GET requests are supported. |
expected_code | No, default: 200 | Which HTTP response code we're expecting (all other will be treated as failure). |
regexp | No, default: "" | Check page for matching regular expression. Response code will be ignored. |
check_period | No, default: 60 | How often we want to run the check (in seconds). |
Checker runs all checks from the config file every 60 seconds, then sends results to the writer using Kafka producer.
Kafka consumer stores the data to the PostgreSQL database. Information is stored in 2 tables:
checks
– information about different checks (URL, regexp, expected return code). Checks are immutable, any change will create a new one. The only mutable field is last_check
, which is updated after each insert.
results
- all results for a given check. Besides boolean success
value, provides some additional data for analytics: response time in seconds, response length, returned HTTP code.
If you want to contact me you can reach me at pub@petersmol.ru