codeur66 / terraform-flask

A CI/CD production ready pipeline for a python flask application using the DevOps tools: git for code repository, docker and docker hub for building and storing the flask application, Jenkins pipeline for continuous integration, ansible for infrastructure provisioning and Kubernetes to launch flask app in a cluster, ELK stack -monitoring.

Vijayalakshmi Kuppuswamy

Capstone Project 2 - Provisioning and Monitoring

Design Document

1. Capstone Project Technological Tool Stack:To build the python flask app using the DevOps approach in building production ready CI/CD pipeline,the tools we are going to use are listed below:
1. Python
2. Github: as a single source of truth
3. Docker:
1. Build the flask image
4. Docker hub:
1. To push the docker flask image as an artifact
2. To pull the docker flask image from the artifact
5. Jenkins: Jenkins is the heart of the entire CI/CD pipeline where we are going to:
1. Build the image from Dockerfile.
2. Push the image to the Docker-hub
3. Declare our environment credentials for Jenkins to interact with the docker hub
4. Perform a SVM checkout
5. Build the project
6. Deploy the Kubernetes cluster using terraform
6. Terraform: We are going to use terraform to provision the Kubernetes cluster
1. Using a Kubernetes.tf file we are going to set details of our deployment and service.
2. The docker hub repository and the flask application name that we are going to provision into our cluster
3. The number of repilicas of 3 flask app.
4. Terraform doesn't only create resources, it updates, and deletes tracked resources without requiring to inspect the API to identify those resource.
7. AWS: using AWS we are leveraging the cloud features:
1. Creating EC-2 instances of our linux environments.
2. AWS instances will provide the complete environment to run our CI/CD pipeline.
3. Mobaxterm and PUTTY: using this tool we can ssh into our ec2 instances and work into them.
8. Virtual Box: using ubuntu 20.04 virtual box to run our entire end-to-end tool stack having our Jenkins pipline to integrate the build and deploy.
9. ELK (Elastic-Logstash-Kibana stack) : Setting and configuring the ELK stack to monitor our CPU, memory, Disk utilization along with our flask app.
10. Prometheus and Grafana : Setting and configuring the two, to monitor our CPU, memory, Disk utilization along with our flask app.
11. Network monitoring tool: using htop to show the system utilizing and to compare with the ELK dashboard.
12. Stress tool: using the stress tool to induce load on the system and to monitor the changes we need to be alerted if the system exceeds 80% of utilization.

Flow Chart:

Setting up the environment:

Set up a virtualBox with ubuntu 20.04 iso image, keeping in mind that we need GUI utility for launching the Jenkins server and the ELK dashboard. Installed the required tool stack: docker, docker-compose, jenkins, terraform, kubectl, kind, terraform and ELK stack. At the jenkins we install all the needed plugins.

Implementing the pipeline:

1. Create a new jenkin's pipeline and configuring the job with needed plugins, configuring the global tools and setting the credentials and evironnment variable for docker hub repo and terraform file.
2. Our pipeline will clean the workspace and checkout the source code from git.Then we use the environment settings to build and push the image to docker hub vijaya81kp/tfflaskapp
3. We use terraform to use the above mentioned image from the docker hub and using the kubernetes.tf we trigger the terraform init and terraform apply to provision our kubernetes cluster with the flask application.

Deployment of the flask app into the kind cluster:

The flask app is now successfully deployed into our kubernetes cluster as shown below:

Monitoring

Elastic,Logstash and kibana are products most commonly used together for log analysis in different IT environments. Using ELK Stack we can perform centralized logging which helps in identifying the problems with the web servers or applications. It lets you search through all the logs at a single place and identify the issues spanning through multiple servers by correlating their logs within a specific time frame.

1. Logstash is the data collection pipeline tool. It the first component of ELK Stack which collects data inputs and feeds it to the Elasticsearch. It collects various types of data from different sources, all at once and makes it available immediately for further use.
2. Elasticsearch is a NoSQL database which is based on Lucene search engine and is built with RESTful APIs. It is a highly flexible and distributed search and analytics engine. Also, it provides simple deployment, maximum reliability, and easy management through horizontal scalability. It provides advanced queries to perform detailed analysis and stores all the data centrally for quick search of the documents.
3. Kibana is a data visualization tool. It is used for visualizing the Elasticsearch documents and helps the us to have an immediate insight into it. Kibana dashboard provides various interactive diagrams,data, timelines, and graphs to visualize the complex queries done using Elasticsearch. Using Kibana we can create and save custom graphs according to our specific needs

Monitoring our flask app without and with load ELK

Introducing the stress

Maximum stress and message

monitoring high stress with htop

Monitoring our flask app using Prometheus and Grafana: