huyhoang17 / HybridNets

HybridNets: End-to-End Perception Network

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

HybridNets: End2End Perception Network

HybridNets Network Architecture.

Generic badge PyTorch - Version Python - Version

HybridNets: End-to-End Perception Network

by Dat Vu, Bao Ngo, Hung Phan πŸ“§ FPT University

(πŸ“§) corresponding author.

arXiv technical report (arXiv 2203.09035)


Table of Contents
  1. About The Project
  2. Getting Started
  3. Usage
  4. Training Tips
  5. Results
  6. License
  7. Acknowledgements
  8. Citation

About The Project

HybridNets is an end2end perception network for multi-tasks. Our work focused on traffic object detection, drivable area segmentation and lane detection. HybridNets can run real-time on embedded systems, and obtains SOTA Object Detection, Lane Detection on BDD100K Dataset. intro

Project Structure

β”‚                   # Model configuration
β”‚                    # Pytorch Hub entrypoint
β”‚            # Image inference
β”‚     # Video inference
β”‚                      # Train script
β”‚                        # Validate script
β”œβ”€β”€β”€encoders                      #
β”‚       ...
β”‚             # Generate new anchors by k-means
β”‚                # BDD100K dataset
β”‚                   # Focal, tversky (dice)
β”‚                  # Model blocks
β”‚       bdd100k.yml               # Project configuration
    β”‚                   # Draw bounding box
    β”‚            #
    β”‚                  # Various helper functions (preprocess, postprocess, eval...)
    └───sync_batchnorm            # 

Getting Started Open In Colab


The project was developed with Python>=3.7 and Pytorch>=1.10.

git clone
cd HybridNets
pip install -r requirements.txt


# Download end-to-end weights
mkdir weights
curl -L -o weights/hybridnets.pth

# Image inference
python -w weights/hybridnets.pth --source demo/image --output demo_result --imshow False --imwrite True

# Video inference
python -w weights/hybridnets.pth --source demo/video --output demo_result

# Result is saved in a new folder called demo_result


Data Preparation

Recommended dataset structure:

    β”‚   β”œβ”€β”€β”€train
    β”‚   └───val
    β”‚   β”œβ”€β”€β”€train
    β”‚   └───val
    β”‚   β”œβ”€β”€β”€train
    β”‚   └───val

Update your dataset paths in projects/your_project_name.yml.

For BDD100K: imgs, det_annot, da_seg_annot, ll_seg_annot


1) Edit or create a new project configuration, using bdd100k.yml as a template

# mean and std of dataset in RGB order
mean: [0.485, 0.456, 0.406]
std: [0.229, 0.224, 0.225]

# bdd100k anchors
anchors_scales: '[2**0, 2**0.70, 2**1.32]'
anchors_ratios: '[(0.62, 1.58), (1.0, 1.0), (1.58, 0.62)]'

# BDD100K officially supports 10 classes
# obj_list: ['person', 'rider', 'car', 'truck', 'bus', 'train', 'motorcycle', 'bicycle', 'traffic light', 'traffic sign']
obj_list: ['car']
obj_combine: ['car', 'bus', 'truck', 'train']  # if single class, combine these classes into 1 single class in obj_list
                                               # leave as empty list ([]) to not combine classes

seg_list: ['road',

  color_rgb: false
  dataroot: path/to/imgs
  labelroot: path/to/det_annot
  laneroot: path/to/ll_seg_annot
  maskroot: path/to/da_seg_annot

2) Train

python -p bdd100k        # your_project_name
                -c 3              # coefficient of effnet backbone, result from paper is 3
                -n 4              # num_workers
                -b 8              # batch_size per gpu
                -w path/to/weight # use 'last' to resume training from previous session
                --freeze_det      # freeze detection head, others: --freeze_backbone, --freeze_seg
                --lr 1e-5         # learning rate
                --optim adamw     # adamw | sgd
                --num_epochs 200

Please check python --help for every available arguments.

3) Evaluate

python -p bdd100k -c 3 -w checkpoints/weight.pth

Training Tips

Anchors βš“

If your dataset is intrinsically different from COCO or BDD100K, or the metrics of detection after training are not as high as expected, you could try enabling autoanchor in project.yml:

  - 640
  - 384
need_autoanchor: true  # set to true to run autoanchor
pin_memory: false

This automatically finds the best combination of anchor scales and anchor ratios for your dataset. Then you can manually edit them project.yml and disable autoanchor.

If you're feeling lucky, maybe mess around with base_anchor_scale in

class HybridNetsBackbone(nn.Module):
  self.pyramid_levels = [5, 5, 5, 5, 5, 5, 5, 5, 6]
  self.anchor_scale = [1.25,1.25,1.25,1.25,1.25,1.25,1.25,1.25,1.25,]
  self.aspect_ratios = kwargs.get('ratios', [(1.0, 1.0), (1.4, 0.7), (0.7, 1.4)])


class Anchors(nn.Module):
  for scale, ratio in itertools.product(self.scales, self.ratios):
    base_anchor_size = self.anchor_scale * stride * scale
    anchor_size_x_2 = base_anchor_size * ratio[0] / 2.0
    anchor_size_y_2 = base_anchor_size * ratio[1] / 2.0

to get a grasp on how anchor boxes work.

And because a picture is worth a thousand words, you can visualize your anchor boxes in Anchor Computation Tool.

Training stages

We experimented with training stages and found that this settings achieved the best results:

  1. --freeze_seg True ~ 100 epochs
  2. --freeze_backbone True --freeze_det True ~ 50 epochs
  3. Train end-to-end ~ 50 epochs

The reason being detection head is harder to converge early on, so we basically skipped segmentation head to focus on detection first.


Traffic Object Detection

Result Visualization
Model Recall (%) mAP@0.5 (%)
MultiNet 81.3 60.2
DLT-Net 89.4 68.4
Faster R-CNN 77.2 55.6
YOLOv5s 86.8 77.2
YOLOP 89.2 76.5
HybridNets 92.8 77.3

Drivable Area Segmentation

Result Visualization
Model Drivable mIoU (%)
MultiNet 71.6
DLT-Net 71.3
PSPNet 89.6
YOLOP 91.5
HybridNets 90.5

Lane Line Detection

Result Visualization
Model Accuracy (%) Lane Line IoU (%)
Enet 34.12 14.64
SCNN 35.79 15.84
Enet-SAD 36.56 16.02
YOLOP 70.5 26.2
HybridNets 85.4 31.6


Distributed under the MIT License. See LICENSE for more information.


Our work would not be complete without the wonderful work of the following authors:


If you find our paper and code useful for your research, please consider giving a star ⭐ and citation πŸ“ :

      title={HybridNets: End-to-End Perception Network}, 
      author={Dat Vu and Bao Ngo and Hung Phan},


HybridNets: End-to-End Perception Network

License:MIT License


Language:Python 92.9%Language:Jupyter Notebook 7.1%