Continuous delivery for ML Flask and Sklearn project using Azure

Introduction

In this project, you will focus on creating a continuous delivery cycle with azure pipelines, using a given code for machine learning application (that you will not develop yourself), and azure portal to create environement on which your application will be deployed.

Project Plannig

We use for planning trello to track tasks as tickets and spreadsheet with weekly planning per year

Prerequisites

Have a azure account and Azure cloud shell activated
Have a github account

Architecture

Instructions

Create a new github repository

From azure cloud shell, clone this repo

You can run this command to check that the Makefile commands working. If you face some error during locust install, check that you have python version >=3.6

make setup && source ~/.udacity-devops/bin/activate && make all

Delete all .yml files and change the in make_predict_azure_app.sh and commands.sh by name you will give to your web app. It should be the same name in both files. Then push the project to github repo

From your github repo, add new github actions workflow

Choose Python application workflow and change the jobs part in yaml file with the following

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python 3.7
      uses: actions/setup-python@v2
      with:
        python-version: 3.7
    - name: Setup environrmrnet
      run: |
        make setup
    - name: Install dependencies
      run: |
        make install
    - name: Lint with pytest
      run: |
        make lint
    - name: Test with locust
      run: |
        make test

After you save and run. Check that the job is successfuly finished

To create azure webapp service, from azure shell run ./commands.sh from root folder. You should see like the output below if the app services deployed:

Here is the app services view in azure portal

You can check that your project is correctly deployed by visiting the URL figured in output. You should see result like:

From dev azure, create a project.

Add github repo to this project and choose your repo

Choose the azure app service already created

Create a pipeline for this project from azure pipelines. Choose the template "python app and stuff".

Update the script part of yaml file with the following

        - script: make setup
            workingDirectory: $(projectRoot)
            displayName: "setup virtual env"

          - script: make install
            workingDirectory: $(projectRoot)
            displayName: "Install requirements"

          - script: make lint
            workingDirectory: $(projectRoot)
            displayName: "lint"

          - script: make test
            workingDirectory: $(projectRoot)
            displayName: "load testing"

Save and run. You can check in your github that azurepipelines.yml already created.

Check that your pipelines is finished and already deployed the app

Good news : Your continuous delivery cycle is set. To check that your continuous delivery cycle is well set, do some changes in your app.py and push.

Replace this

@app.route("/")
def home():
    html = "<h3>Sklearn Prediction Home</h3>"
    return html.format(format)

by this

@app.route("/")
def home():
    html = "<h3>Sklearn Prediction Home - change to trigger pipeline</h3>"
    return html.format(format)

The pipeline should run automaticaly and the deployment is done successfuly.

Check the app url and check the home page. You should get this result

From azure shell, run the command and check the result

udacity@Azure:~$ ./make_predict_azure_app.sh
Port: 443
{"prediction":[20.35373177134412]}

You can from azure cloud shell run this command az webapp log tail to see the live logs and run make then see logs

To run locust test, run the command make test

Enhancements

Add different environments : dev, qa and prod
Add terrafom file to manage needed resources

Demo

https://youtu.be/3g2usvuJdd8

ysidki / flask-sklearn