tuanht12 / news-summarization-api

NEWS SUMMARIZATION APPLICATION DEPLOYED ON K8S

The purpose of this repository is to house the Final Project for Module 1 of the Machine Learning Engineering (MLE) Course.

System Architecture

This repository contains a demo news summarization application for summarizing Vietnamese news articles. The application is built using the following technologies:

• Model: The ViT5 model is used for text summarization. It is a state-of-the-art pretrained Transformer-based encoder-decoder model specifically designed for Vietnamese.
• API Endpoints: The application integrates API endpoints using FastAPI, a modern, fast (high-performance), web framework for building APIs with Python.
• Containerization: The application is containerized using Docker, which allows for easy deployment and scalability.
• Deployment: The model is deployed on Google Kubernetes Engine (GKE), a managed environment for deploying, managing, and scaling containerized applications using Kubernetes. The GKE cluster is provisioned using Terraform, an infrastructure as code tool.
• Monitoring: Prometheus, an open-source monitoring and alerting toolkit, along with Grafana, a popular visualization tool, are used to monitor the deployed application and gather performance metrics.
• CI/CD Pipeline: The project includes a CI/CD pipeline implemented with Jenkins. Jenkins is installed on Google Compute Engine (GCE) using Ansible to automate image pushes to Docker Hub and streamline the model deployment process on GKE.

Demo

API

Monitoring in Grafana

Run code in your local machine

Create virtual environment, install required packages and download pretrained checkpoints

We will use Python 3.8

make venv
source venv/bin/activate
python3.8 download_pretrained_model.py

Run the API

uvicorn main:app --host 0.0.0.0 --port 30000

Then, you can try the API at localhost:30000/docs.

Deploy to GCP

Install Gcloud Related Packages and Setup Project

• Install gcloud CLI

• Install gke-gcloud-auth-plugin

  sudo apt-get install google-cloud-cli-gke-gcloud-auth-plugin
  

• Setup Gcloud Project

  • Authorizes gcloud and other SDK tools to access Google Cloud and setup configuration

    gcloud init
    

  • Login to GCP

    gcloud auth application-default login
    

Deploy Model to GKE cluster

Create GKE Cluster with Terraform

Run the following commands to create a GKE cluster using Terraform

cd terraform
terraform init
terraform plan
terraform apply

After a while, you should see this if your cluster is successfully created

Run API in GKE cluster

• Connect to the cluster using gcloud command.

• Switch the current context using kubectx

• Create namespaces

  kubectl create ns nginx-ingress
  kubectl create ns model-serving
  

• Deploy NGINX-ingress

  kubens nginx-system
  helm upgrade --install nginx-ingress helm-charts/nginx-ingress

• Get External IP of NGINX-ingress

  kubens nginx-system
  kubectl get svc

  

• Replace the External IP above in spec/rules/host in file helm-charts/model-deployment/templates/nginx-ingress.yaml

• Deploy Model

  kubens model-serving
  helm upgrade --install news-summarization helm-charts/model-deployment
  

• You can access the API in NGINX_EXTERNAL_IP.nip.io/docs

CI/CD Jenkins in GCE

Create a Compute Instance using Ansible

• Create a new service account with Compute Admin role. In this project, the service account is named ansible-sa.

• Navigate to Service accounts section, select action Manage keys.

• Create new key of the created service account and download it as JSON file.

• Save the downloaded JSON key in ansible/secrets.

• Update the service_account_file field in ansible/playbooks/create_compute_instance.yml with your new path to the new JSON key.

• Run the following command in terminal to create a Compute Instance

  ansible-playbook ansible/playbooks/create_compute_instance.yml

• After few moments, you should see your instance is successfully created with the name vm-jenkins

Deploy Jenkins on the Compute Instance

• Generate a new SSH key using ssh-keygen in the terminal of your local machine.
• Navigate to section Settings/Metadata and select tab SSH KEYS.
• Select EDIT and copy your public key that is just created (.pub file).
• Update the external IP of your VM and the path to private key in ansible/inventory.
• Run the following command in terminal to deploy Jenkins on the VM

  ansible-playbook --inventory-file ansible/inventory ansible/playbooks/deploy_jenkins.yml

Configure Jenkins

• Connect to Jenkins UI

  • Checking Jenkins installed successfully on GCE

    • Access the GCE instance

      ssh YOUR_USERNAME@INSTANCE_EXTERNAL_IP -i PATH_TO_PRIVATE_SSH_KEY
      

    • Verify if Jenkins is running in the Compute Engine instance

      sudo docker ps
      

      

  • Access Jenkins UI via INSTANCE_EXTERNAL_IP:8081.

  • Follow the instruction to log in into Jenkins.

  • The password can be retrieved by

    # inside GCE instance
    sudo docker exec -ti jenkins bash
    cat /var/jenkins_home/secrets/initialAdminPassword
    

• Connect Jenkins to GitHub Repo

  • Add Jenkins to Repo Webhook

    • Payload URL would http://INSTANCE_EXTERNAL_IP:8081/github-webhook/

    • Event Trigger can be set to: Pushes and Pull Requests

  • Add GitHub Repo to Jenkins
    • Create new Multibranch Item in Jenkins
    • Create a new Personal Access Token. In this project we will generate a classic token.
    • Connect Repo to Jenkins
      • Add credentials with your Github username and password is just-created Personal Access Token
      • Copy the URL to your Github repo.
      • Click Validate to see whether Jenkins successfully connect to your Github repo or not.

• Add DockerHub Token to Jenkins Credential

  • Create a new DockerHub Token
  • Add the token to Jenkins' Credentials
    • Note: ID must be dockerhub to match the registryCredential in Jenkinsfile.

• Setup Cloud Connection

  • Create clusterrolebinding

    kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=system:anonymous
    kubectl create clusterrolebinding cluster-admin-default-binding --clusterrole=cluster-admin --user=system:serviceaccount:model-serving:default
  

  • Configure clouds at http://INSTANCE_EXTERNAL_IP:8081/manage/configureClouds/

    • Get Kubernetes URL and Kubernetes server certificate key by running the following in terminal of your local machine.

      cat ~/.kube/config
      

      • Kubernetes URL is server
      • Kubernetes server certificate key is certificate-authority-data

      

• Build

  • When there is a new push of new changes to a branch, the test pipeline will run.
  • When there are changes merged to main, the Deploy pipeline will run, build and push the latest image to DockerHub.
  • After that, the pipeline will use Helm to pull and deploy the latest image in GKE cluster automatically.

Monitoring System

• Create and switch to namespace monitoring

  kubectl create ns monitoring
  kubens monitoring

• Get dependencies to build kube-prometheus-stack

  cd helm-charts/k8s-monitoring/kube-prometheus-stack
  helm dependency build

• Deploy kube-prometheus-stack using helm

  cd helm-charts/k8s-monitoring
  helm install -f kube-prometheus-stack.expanded.yaml kube-prometheus-stack kube-prometheus-stack -n monitoring

• Edit hosts

  • Add 2 hosts of grafana and prometheus at the end of /ect/hosts

    sudo vim /ect/hosts
    

    NGINX_EXTERNAL_IP prometheus.newssum.monitor.com
    NGINX_EXTERNAL_IP grafana.newssum.monitor.com
    

• Grafana can be accessed at http://grafana.newssum.monitor.com/login

• Prometheus can be accessed at http://prometheus.newssum.monitor.com

TODO

• Add Model Performance Measurement: metrics and visualization
• Add Logs and Traces Storage: The application integrates with a logging and tracing system to capture and store logs and traces for troubleshooting and analysis. Elasticsearch and Jaeger are used for log storage and distributed tracing, respectively.