Deploy Prometheus + Alertmanager + Grafana stack to monitor server using Ansible

I. Requirements

Deploy Prometheus-Grafana-Alertmanager stack and Angular-Springboot web application
- Using docker, docker-compose, ansible to deploy
Define alert rules to monitoring target hosts
Configure Alertmanager to push alert to Telegram, Slack, Mail...
Create Grafana dashboards to monitoring target hosts

II. System Architecture

1. Overview

2 machines on AWS to deploy web application.
1 machine on VMWare to deploy monitoring system.
Using Ansible to deploy web app, monitoring system, package... on 3 above machines.

2. Monitor machine

I will create a monitor machine on Vmware Workstation 17 Player (Ubuntu 20.04), then I deploy Prometheus-Grafana-Alertmanager stack using docker, docker-compose and Ansible on this machine

3. Worker machines

I will create 2 EC2 instances (Ubuntu 20.04) on AWS for worker nodes

Worker1 for deploy web application and Node exporter
- Web app: http://[worker1-AWS-EC2-Public-IPv4-DNS]:4200/#
- Node exporter: http://[worker1-AWS-EC2-Public-IPv4-DNS]:9100
Worker2 for only Node exporter
- Node exporter: http://[worker2-AWS-EC2-Public-IPv4-DNS]:9100

I use Ansible to deploy web app and node exporter on 2 ec2 worker. Because I will need to get the node_exporter metrics from port 9100, so I need to change the Inbound rules of security group to allow access our EC2 instances in port 9100.

4. Diagram

Deployment diagram.

III. Deploy result

1. Run ansible-playbook to set up monitor and worker machines

Run successfully

Result after run ansible-playbook

Check image on Dockerhub

Backend image and Frontend image were pushed to Dockerhub

Test Webapp deployment on AWS EC2

Web app is hosted at http://[worker1-AWS-EC2-Public-IPv4-DNS]:4200/#
In this case web app is hosted at http://ec2-13-215-157-152.ap-southeast-1.compute.amazonaws.com:4200/#