flask jupyter keras miniconda python scrapy tensorflow wsl2

Duck or Cat

Duck or Cat is a binary classification model. It classifies pictures of ducks and cats.

Content of this project

This project is split into 3 differents folders:

scrapy: Scrapy project that helps with the creation of a dataset. It massively download pictures of cats and ducks.
jupyter: Jupyter project that trains and validates the model.
flask: Flask project that alows the user to try the model.

Dataset

The balance of the dataset is the following:

train: 9000 cats and 9000 ducks
test: 2000 cats and 9000 ducks

The model

The model is a CNN classification model, it has the following structure:

Result

Here is the evolution of the accuracy over 25 epochs:

Here is the predictions of cats and ducks:

And here is the confusion matrix:

The model has an accuracy of 98.21%, which seems to be very different for the deployed model.

Project setup

Install CUDA drivers on Windows 11:

Here.

Install CuDNN on WSL2:

Find the correct version of cuDNN here then run:

wget https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.0/local_installers/12.x/cudnn-local-repo-ubuntu2204-8.9.0.131_1.0-1_amd64.deb/
sudo dpkg -i cudnn-local-repo-ubuntu2204-8.9.0.131_1.0-1_amd64.deb
sudo cp /var/cudnn-local-repo-ubuntu2204-8.9.0.131/cudnn-local-D7522631-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install libcudnn8 libcudnn8-dev

Install CUDA on WSL2:

Delete the old key:

sudo apt-key del 7fa2af80

Find the correct version of CUDA here then run:

wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
bsudo apt-get update
sudo apt-get -y install cuda

Install Miniconda:

Get the correct version of Miniconda here and install it:

wget wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
sudo bash Miniconda3-latest-Linux-x86_64.sh -p /usr/bin/miniconda3

Disable Miniconda on start-up (optional):

Get into the Miniconda environement if you aren't already:

source /usr/bin/miniconda3/bin/activate

Turn off conda on start-up:

conda config --set auto_activate_base false

Create a conda environement:

Get into the Miniconda environement if you aren't already:

source /usr/bin/miniconda3/bin/activate

Create a new conda environement:

conda create --name env python=3.9

Install librairies:

Get in the environement:

conda activate env

Install cudatoolkit:

conda install -c conda-forge cudatoolkit=11.8.0

Install required librairies:

cd dev/DuckOrCat/ && pip install -r flask/requirements.txt

Set the path variables:

CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/:$CUDNN_PATH/lib
mkdir -p $CONDA_PREFIX/etc/conda/activate.d
echo 'CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/:$CUDNN_PATH/lib' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh

Usage

Flask

To develop the web interface, get into the flask/ folder and run:

flask --app  main.py --debug run

If you want to run the production version on Docker instead, you can run:

docker build -f Dockerfile -t flask . && docker run -p 8000:8000 -it flask

Then open http://localhost:8000/.

Jupyter

To develop on the model, get into the jupyter/ folder and run:

jupyter-lab

Scrapy

To download picture of ducks or cats, get into the scrapy/ folder and run:

scrapy crawl [duck|cat]

Some scripts are available in scrapy/tools/:

delete_duplicate.py: delete every picture duplicates in the dataset folder.
rename.py: rename every pictures of a folder following a pattern.
resize.py: to save on storage and RAM, resize every pictures to be a maximum of 1024 pixels of width and height.

To use any of them, get into the scrapy/tools/ and run:

python3 [script_name]

Versions

Windows 11 and WSL2 (Ubuntu 22.04.2 LTS)
Python 3.9
Flask 2.2.3
Scrapy 2.7.1
Tensorflow 2.12
CUDA tool kit 11.8
Docker version 20.10.24 (optional)

Structure

flask/
scrapy/
jupyter/
env/
.gitignore
README.md

References

About