Giters

edge-ai4cv / DeepStream-Yolo

NVIDIA DeepStream SDK 6.1 / 6.0.1 / 6.0 configuration for YOLO models

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

DeepStream-Yolo

NVIDIA DeepStream SDK 6.1 / 6.0.1 / 6.0 configuration for YOLO models

Future updates

  • Models benchmarks
  • DeepStream tutorials
  • YOLOX support
  • PP-YOLO support
  • YOLOv6 support
  • Dynamic batch-size

Improvements on this repository

  • Darknet cfg params parser (no need to edit nvdsparsebbox_Yolo.cpp or other files)
  • Support for new_coords, beta_nms and scale_x_y params
  • Support for new models
  • Support for new layers
  • Support for new activations
  • Support for convolutional groups
  • Support for INT8 calibration
  • Support for non square models
  • Support for reorg, implicit and channel layers (YOLOR)
  • YOLOv5 4.0, 5.0, 6.0 and 6.1 support
  • YOLOR support
  • GPU YOLO Decoder (moved from CPU to GPU to get better performance) #138
  • GPU Batched NMS #142
  • New documentation for multiple models

Getting started

Requirements

DeepStream 6.1 on x86 platform

DeepStream 6.0.1 / 6.0 on x86 platform

DeepStream 6.1 on Jetson platform

DeepStream 6.0.1 / 6.0 on Jetson platform

For YOLOv5 and YOLOR

x86 platform

Jetson platform

Tested models

Benchmarks

New tests comming soon.

dGPU installation

To install the DeepStream on dGPU (x86 platform), without docker, we need to do some steps to prepare the computer.

DeepStream 6.1

1. Disable Secure Boot in BIOS

2. Install dependencies

sudo apt-get update
sudo apt-get install gcc make git libtool autoconf autogen pkg-config cmake
sudo apt-get install python3 python3-dev python3-pip
sudo apt-get install dkms
sudo apt-get install libssl1.1 libgstreamer1.0-0 gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav libgstrtspserver-1.0-0 libjansson4 libyaml-cpp-dev
sudo apt-get install linux-headers-$(uname -r)

NOTE: Purge all NVIDIA driver, CUDA, etc (replace $CUDA_PATH to your CUDA path)

sudo nvidia-uninstall
sudo $CUDA_PATH/bin/cuda-uninstaller
sudo apt-get remove --purge '*nvidia*'
sudo apt-get remove --purge '*cuda*'
sudo apt-get remove --purge '*cudnn*'
sudo apt-get remove --purge '*tensorrt*'
sudo apt autoremove --purge && sudo apt autoclean && sudo apt clean

3. Install CUDA Keyring

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
sudo apt-get update

4. Download and install NVIDIA Driver

  • TITAN, GeForce RTX / GTX series and RTX / Quadro series

    wget https://us.download.nvidia.com/XFree86/Linux-x86_64/510.47.03/NVIDIA-Linux-x86_64-510.47.03.run

  • Data center / Tesla series

    wget https://us.download.nvidia.com/tesla/510.47.03/NVIDIA-Linux-x86_64-510.47.03.run

  • Run

    sudo sh NVIDIA-Linux-x86_64-510.47.03.run --silent --disable-nouveau --dkms --install-libglvnd

    NOTE: This step will disable the nouveau drivers.

  • Reboot

    sudo reboot

  • Install

    sudo sh NVIDIA-Linux-x86_64-510.47.03.run --silent --disable-nouveau --dkms --install-libglvnd

NOTE: If you are using a laptop with NVIDIA Optimius, run

sudo apt-get install nvidia-prime
sudo prime-select nvidia

5. Download and install CUDA

wget https://developer.download.nvidia.com/compute/cuda/11.6.1/local_installers/cuda_11.6.1_510.47.03_linux.run
sudo sh cuda_11.6.1_510.47.03_linux.run --silent --toolkit

  • Export environment variables

    echo $'export PATH=/usr/local/cuda-11.6/bin${PATH:+:${PATH}}\nexport LD_LIBRARY_PATH=/usr/local/cuda-11.6/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' >> ~/.bashrc && source ~/.bashrc

6. Download from NVIDIA website and install the TensorRT

TensorRT 8.2 GA Update 4 for Ubuntu 20.04 and CUDA 11.0, 11.1, 11.2, 11.3, 11.4 and 11.5 DEB local repo Package

sudo dpkg -i nv-tensorrt-repo-ubuntu2004-cuda11.4-trt8.2.5.1-ga-20220505_1-1_amd64.deb
sudo apt-key add /var/nv-tensorrt-repo-ubuntu2004-cuda11.4-trt8.2.5.1-ga-20220505/82307095.pub
sudo apt-get update
sudo apt install tensorrt

7. Download from NVIDIA website and install the DeepStream SDK

DeepStream 6.1 for Servers and Workstations (.deb)

sudo apt-get install ./deepstream-6.1_6.1.0-1_amd64.deb
rm ${HOME}/.cache/gstreamer-1.0/registry.x86_64.bin
sudo ln -snf /usr/local/cuda-11.6 /usr/local/cuda

8. Reboot the computer

sudo reboot
DeepStream 6.0.1 / 6.0

1. Disable Secure Boot in BIOS

If you are using a laptop with newer Intel/AMD processors and your Graphics in Settings->Details->About tab is llvmpipe, please update the kernel. 
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.11/amd64/linux-headers-5.11.0-051100_5.11.0-051100.202102142330_all.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.11/amd64/linux-headers-5.11.0-051100-generic_5.11.0-051100.202102142330_amd64.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.11/amd64/linux-image-unsigned-5.11.0-051100-generic_5.11.0-051100.202102142330_amd64.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.11/amd64/linux-modules-5.11.0-051100-generic_5.11.0-051100.202102142330_amd64.deb
sudo dpkg -i  *.deb
sudo reboot

2. Install dependencies

sudo apt-get update
sudo apt-get install gcc make git libtool autoconf autogen pkg-config cmake
sudo apt-get install python3 python3-dev python3-pip
sudo apt install libssl1.0.0 libgstreamer1.0-0 gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav libgstrtspserver-1.0-0 libjansson4
sudo apt-get install linux-headers-$(uname -r)

NOTE: Install DKMS only if you are using the default Ubuntu kernel

sudo apt-get install dkms

NOTE: Purge all NVIDIA driver, CUDA, etc (replace $CUDA_PATH to your CUDA path)

sudo nvidia-uninstall
sudo $CUDA_PATH/bin/cuda-uninstaller
sudo apt-get remove --purge '*nvidia*'
sudo apt-get remove --purge '*cuda*'
sudo apt-get remove --purge '*cudnn*'
sudo apt-get remove --purge '*tensorrt*'
sudo apt autoremove --purge && sudo apt autoclean && sudo apt clean

3. Install CUDA Keyring

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
sudo apt-get update

4. Download and install NVIDIA Driver

  • TITAN, GeForce RTX / GTX series and RTX / Quadro series

    wget https://us.download.nvidia.com/XFree86/Linux-x86_64/470.129.06/NVIDIA-Linux-x86_64-470.129.06.run

  • Data center / Tesla series

    wget https://us.download.nvidia.com/tesla/470.129.06/NVIDIA-Linux-x86_64-470.129.06.run

  • Run

    sudo sh NVIDIA-Linux-x86_64-470.129.06.run --silent --disable-nouveau --dkms --install-libglvnd

    NOTE: This step will disable the nouveau drivers.

    NOTE: Remove --dkms flag if you installed the 5.11.0 kernel.

  • Reboot

    sudo reboot

  • Install

    sudo sh NVIDIA-Linux-x86_64-470.129.06.run --silent --disable-nouveau --dkms --install-libglvnd

    NOTE: Remove --dkms flag if you installed the 5.11.0 kernel.

NOTE: If you are using a laptop with NVIDIA Optimius, run

sudo apt-get install nvidia-prime
sudo prime-select nvidia

5. Download and install CUDA

wget https://developer.download.nvidia.com/compute/cuda/11.4.1/local_installers/cuda_11.4.1_470.57.02_linux.run
sudo sh cuda_11.4.1_470.57.02_linux.run --silent --toolkit

  • Export environment variables

    echo $'export PATH=/usr/local/cuda-11.4/bin${PATH:+:${PATH}}\nexport LD_LIBRARY_PATH=/usr/local/cuda-11.4/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' >> ~/.bashrc && source ~/.bashrc

6. Download from NVIDIA website and install the TensorRT

TensorRT 8.0.1 GA for Ubuntu 18.04 and CUDA 11.3 DEB local repo package

sudo dpkg -i nv-tensorrt-repo-ubuntu1804-cuda11.3-trt8.0.1.6-ga-20210626_1-1_amd64.deb
sudo apt-key add /var/nv-tensorrt-repo-ubuntu1804-cuda11.3-trt8.0.1.6-ga-20210626/7fa2af80.pub
sudo apt-get update
sudo apt-get install libnvinfer8=8.0.1-1+cuda11.3 libnvinfer-plugin8=8.0.1-1+cuda11.3 libnvparsers8=8.0.1-1+cuda11.3 libnvonnxparsers8=8.0.1-1+cuda11.3 libnvinfer-bin=8.0.1-1+cuda11.3 libnvinfer-dev=8.0.1-1+cuda11.3 libnvinfer-plugin-dev=8.0.1-1+cuda11.3 libnvparsers-dev=8.0.1-1+cuda11.3 libnvonnxparsers-dev=8.0.1-1+cuda11.3 libnvinfer-samples=8.0.1-1+cuda11.3 libnvinfer-doc=8.0.1-1+cuda11.3

7. Download from NVIDIA website and install the DeepStream SDK

  • DeepStream 6.0.1 for Servers and Workstations (.deb)

    sudo apt-get install ./deepstream-6.0_6.0.1-1_amd64.deb

  • DeepStream 6.0 for Servers and Workstations (.deb)

    sudo apt-get install ./deepstream-6.0_6.0.0-1_amd64.deb

  • Run

    rm ${HOME}/.cache/gstreamer-1.0/registry.x86_64.bin
sudo ln -snf /usr/local/cuda-11.4 /usr/local/cuda

8. Reboot the computer

sudo reboot

Basic usage

1. Download the repo

git clone https://github.com/marcoslucianops/DeepStream-Yolo.git
cd DeepStream-Yolo

2. Download the cfg and weights files from Darknet repo to the DeepStream-Yolo folder

3. Compile the lib

  • DeepStream 6.1 on x86 platform

    CUDA_VER=11.6 make -C nvdsinfer_custom_impl_Yolo

  • DeepStream 6.0.1 / 6.0 on x86 platform

    CUDA_VER=11.4 make -C nvdsinfer_custom_impl_Yolo

  • DeepStream 6.1 on Jetson platform

    CUDA_VER=11.4 make -C nvdsinfer_custom_impl_Yolo

  • DeepStream 6.0.1 / 6.0 on Jetson platform

    CUDA_VER=10.2 make -C nvdsinfer_custom_impl_Yolo

4. Edit the config_infer_primary.txt file according to your model (example for YOLOv4)

[property]
...
custom-network-config=yolov4.cfg
model-file=yolov4.weights
...

5. Run

deepstream-app -c deepstream_app_config.txt

NOTE: If you want to use YOLOv2 or YOLOv2-Tiny models, change the deepstream_app_config.txt file before run it

...
[primary-gie]
...
config-file=config_infer_primary_yoloV2.txt
...

YOLOv5 usage

NOTE: Make sure to change the YOLOv5 repo version according to your model version before the conversion.

1. Copy the gen_wts_yoloV5.py file from DeepStream-Yolo/utils directory to the YOLOv5 folder

2. Open the YOLOv5 folder

3. Download the pt file from YOLOv5 repo (example for YOLOv5n 6.1)

wget https://github.com/ultralytics/yolov5/releases/download/v6.1/yolov5n.pt

4. Generate the cfg and wts files (example for YOLOv5n)

python3 gen_wts_yoloV5.py -w yolov5n.pt -c models/yolov5n.yaml

5. Copy the generated cfg and wts files to the DeepStream-Yolo folder

6. Open the DeepStream-Yolo folder

7. Compile the lib

  • DeepStream 6.1 on x86 platform

    CUDA_VER=11.6 make -C nvdsinfer_custom_impl_Yolo

  • DeepStream 6.0.1 / 6.0 on x86 platform

    CUDA_VER=11.4 make -C nvdsinfer_custom_impl_Yolo

  • DeepStream 6.1 on Jetson platform

    CUDA_VER=11.4 make -C nvdsinfer_custom_impl_Yolo

  • DeepStream 6.0.1 / 6.0 on Jetson platform

    CUDA_VER=10.2 make -C nvdsinfer_custom_impl_Yolo

8. Edit the config_infer_primary_yoloV5.txt file according to your model (example for YOLOv5n)

[property]
...
custom-network-config=yolov5n.cfg
model-file=yolov5n.wts
...

9. Edit the deepstream_app_config.txt file

...
[primary-gie]
...
config-file=config_infer_primary_yoloV5.txt

10. Run

deepstream-app -c deepstream_app_config.txt

NOTE: For YOLOv5 P6, check the gen_wts_yoloV5.py file args and set them according to your model.

  • Input weights (.pt) file path

    -w or --weights

  • Input cfg (.yaml) file path

    -c or --yaml

  • Inference size [size] or [height , weight]

    Default: 640 / 1280 (if --p6)

    -s or --size

    • Example for 1280

      -s 1280

      or

      -s 1280 1280

YOLOR usage

1. Copy the gen_wts_yolor.py file from DeepStream-Yolo/utils directory to the YOLOR folder

2. Open the YOLOR folder

3. Download the pt file from YOLOR repo

4. Generate the cfg and wts files (example for YOLOR-CSP)

python3 gen_wts_yolor.py -w yolor_csp.pt -c cfg/yolor_csp.cfg

5. Copy the generated cfg and wts files to the DeepStream-Yolo folder

6. Open the DeepStream-Yolo folder

7. Compile the lib

  • DeepStream 6.1 on x86 platform

    CUDA_VER=11.6 make -C nvdsinfer_custom_impl_Yolo

  • DeepStream 6.0.1 / 6.0 on x86 platform

    CUDA_VER=11.4 make -C nvdsinfer_custom_impl_Yolo

  • DeepStream 6.1 on Jetson platform

    CUDA_VER=11.4 make -C nvdsinfer_custom_impl_Yolo

  • DeepStream 6.0.1 / 6.0 on Jetson platform

    CUDA_VER=10.2 make -C nvdsinfer_custom_impl_Yolo

8. Edit the config_infer_primary_yolor.txt file according to your model (example for YOLOR-CSP)

[property]
...
custom-network-config=yolor_csp.cfg
model-file=yolor_csp.wts
...

9. Edit the deepstream_app_config.txt file

...
[primary-gie]
...
config-file=config_infer_primary_yolor.txt

10. Run

deepstream-app -c deepstream_app_config.txt

NMS Configuration

To change the iou-threshold, score-threshold and topk values, modify the config_nms.txt file and regenerate the model engine file.

[property]
iou-threshold=0.45
score-threshold=0.25
topk=300

NOTE: Lower topk values will result in more performance.

NOTE: Make sure to set cluster-mode=4 in the config_infer file.

NOTE: You are still able to change the pre-cluster-threshold values in the config_infer files.

INT8 calibration

1. Install OpenCV

sudo apt-get install libopencv-dev

2. Compile/recompile the nvdsinfer_custom_impl_Yolo lib with OpenCV support

  • DeepStream 6.1 on x86 platform

    CUDA_VER=11.6 OPENCV=1 make -C nvdsinfer_custom_impl_Yolo

  • DeepStream 6.0.1 / 6.0 on x86 platform

    CUDA_VER=11.4 OPENCV=1 make -C nvdsinfer_custom_impl_Yolo

  • DeepStream 6.1 on Jetson platform

    CUDA_VER=11.4 OPENCV=1 make -C nvdsinfer_custom_impl_Yolo

  • DeepStream 6.0.1 / 6.0 on Jetson platform

    CUDA_VER=10.2 OPENCV=1 make -C nvdsinfer_custom_impl_Yolo

3. For COCO dataset, download the val2017, extract, and move to DeepStream-Yolo folder

  • Select 1000 random images from COCO dataset to run calibration

    mkdir calibration

    for jpg in $(ls -1 val2017/*.jpg | sort -R | head -1000); do \
    cp ${jpg} calibration/; \
done

  • Create the calibration.txt file with all selected images

    realpath calibration/*jpg > calibration.txt

  • Set environment variables

    export INT8_CALIB_IMG_PATH=calibration.txt
export INT8_CALIB_BATCH_SIZE=1

  • Edit the config_infer file

    ...
model-engine-file=model_b1_gpu0_fp32.engine
#int8-calib-file=calib.table
...
network-mode=0
...

    To

    ...
model-engine-file=model_b1_gpu0_int8.engine
int8-calib-file=calib.table
...
network-mode=1
...

  • Run

    deepstream-app -c deepstream_app_config.txt

NOTE: NVIDIA recommends at least 500 images to get a good accuracy. On this example, I used 1000 images to get better accuracy (more images = more accuracy). Higher INT8_CALIB_BATCH_SIZE values will result in more accuracy and faster calibration speed. Set it according to you GPU memory. This process can take a long time.

Extract metadata

You can get metadata from DeepStream using Python and C/C++. For C/C++, you can edit the deepstream-app or deepstream-test codes. For Python, your can install and edit deepstream_python_apps.

Basically, you need manipulate the NvDsObjectMeta (Python / C/C++) and NvDsFrameMeta (Python / C/C++) to get the label, position, etc. of bboxes.

My projects: https://www.youtube.com/MarcosLucianoTV

About

NVIDIA DeepStream SDK 6.1 / 6.0.1 / 6.0 configuration for YOLO models

Languages

Language:C++ 69.9%Language:Cuda 15.9%Language:Python 11.8%Language:Makefile 2.4%