Media-Smart / flexinfer

A flexible Python front-end inference SDK based on TensorRT

FlexInfer

A flexible Python front-end inference SDK.

Features

• Flexible

  FlexInfer has a Python front-end, which makes it easy to build a computer vision product prototype.

• Efficient

  Most of time consuming part of FlexInfer is powered by C++ or CUDA, so FlexInfer is also efficient. If you are really hungry for efficiency and don't mind the trouble of C++, you can refer to CheetahInfer.

License

This project is released under Apache 2.0 license.

Installation

Requirements

• Linux
• Python 3.6 or higher
• TensorRT 7.1.3.4 or higher
• PyTorch 1.4.0 or higher
• CUDA 10.2 or higher
• volksdep 3.2.0 or higher

We have tested the following versions of OS and softwares:

• OS: Ubuntu 16.04.6 LTS
• Python 3.6.9
• TensorRT 7.1.3.4
• PyTorch 1.6.0
• CUDA: 10.2
• volksdep: 3.2.0

Install FlexInfer

1. If your platform is x86 or x64, you can create a conda virtual environment and activate it.

conda create -n flexinfer python=3.6.9 -y
conda activate flexinfer

2. Install volksdep following the official instructions

3. Setup

pip install "git+https://github.com/Media-Smart/flexinfer.git"

Usage

We provide some examples for different tasks.

• Classification
• Segmentation
• Object Detection
• Scene Text Recognition

Throughput benchmark

• Device: Jetson AGX Xavier
• CUDA: 10.2

Tasks		framework	version	input shape	data type	throughput(FPS)	latency(ms)
Classification (ResNet18)		PyTorch	1.5.0	(1, 3, 224, 224)	FP16	172	6.01
		TensorRT	7.1.0.16	(1, 3, 224, 224)	FP16	754	1.8
Segmentation（U-Net）		PyTorch	1.5.0	(1, 3, 513, 513)	FP16	15	63.27
		tensorrt	7.1.0.16	(1, 3, 513, 513)	FP16	29	34.03
Object Detection	RetinaNet-R50	PyTorch	1.5.0	(1, 3, 768, 1280)	FP16	8	118.79
		TensorRT	7.1.0.16	(1, 3, 768, 1280)	FP16	15	68.10
	TinaFace-R50-FPN-BN	PyTorch	1.5.0	(1, 3, 768, 1280)	FP16	3	273.60
		TensorRT	7.1.0.16	(1, 3, 768, 1280)	FP16	6	159.70
Scene Text Recognition (ResNet-CTC)		PyTorch	1.5.0	(1, 1, 32, 100)	FP16	113	10.75
		TensorRT	7.1.0.16	(1, 1, 32, 100)	FP16	308	3.55

Media-Smart toolboxes

We provide some toolboxes of different tasks for training, testing and deploying.

• Classification (vedacls)

• Segmentation (vedaseg)

• Object Detection (vedadet)

• Scene Text Recognition (vedastr)

Contact

This repository is currently maintained by Yuxin Zou (@Yuxin Zou), Jun Sun(@ChaseMonsterAway), Hongxiang Cai (@hxcai) and Yichao Xiong (@mileistone).