MATLAB Deep Learning Model Hub
Discover pretrained models for deep learning in MATLAB.
Models
Computer Vision
- Image Classification
- Object Detection
- Semantic Segmentation
- Instance Segmentation
- Image Translation
- Pose Estimation
- Video Classification
- Text Detection & Recognition
Natural Language Processing
Audio
Lidar
Image Classification
Pretrained image classification networks have already learned to extract powerful and informative features from natural images. Use them as a starting point to learn a new task using transfer learning.
Inputs are RGB images, the output is the predicted label and score:
These networks have been trained on more than a million images and can classify images into 1000 object categories.
Models available in MATLAB:
| Network | Size (MB) | Classes | Accuracy % | Location |
|---|---|---|---|---|
| googlenet | 27 | 1000 | 66.25 | Doc GitHub |
| squeezenet | 5.2 | 1000 | 55.16 | Doc |
| alexnet | 227 | 1000 | 54.10 | Doc |
| resnet18 | 44 | 1000 | 69.49 | Doc GitHub |
| resnet50 | 96 | 1000 | 74.46 | Doc GitHub |
| resnet101 | 167 | 1000 | 75.96 | Doc GitHub |
| mobilenetv2 | 13 | 1000 | 70.44 | Doc GitHub |
| vgg16 | 515 | 1000 | 70.29 | Doc |
| vgg19 | 535 | 1000 | 70.42 | Doc |
| inceptionv3 | 89 | 1000 | 77.07 | Doc |
| inceptionresnetv2 | 209 | 1000 | 79.62 | Doc |
| xception | 85 | 1000 | 78.20 | Doc |
| darknet19 | 78 | 1000 | 74.00 | Doc |
| darknet53 | 155 | 1000 | 76.46 | Doc |
| densenet201 | 77 | 1000 | 75.85 | Doc |
| shufflenet | 5.4 | 1000 | 63.73 | Doc |
| nasnetmobile | 20 | 1000 | 73.41 | Doc |
| nasnetlarge | 332 | 1000 | 81.83 | Doc |
| efficientnetb0 | 20 | 1000 | 74.72 | Doc |
| ConvMixer | 7.7 | 10 | - | GitHub |
Tips for selecting a model
Pretrained networks have different characteristics that matter when choosing a network to apply to your problem. The most important characteristics are network accuracy, speed, and size. Choosing a network is generally a tradeoff between these characteristics. The following figure highlights these tradeoffs:
Figure. Comparing image classification model accuracy, speed and size.
Object Detection
Object detection is a computer vision technique used for locating instances of objects in images or videos. When humans look at images or video, we can recognize and locate objects of interest within a matter of moments. The goal of object detection is to replicate this intelligence using a computer.
Inputs are RGB images, the output is the predicted label, bounding box and score:
These networks have been trained to detect 80 objects classes from the COCO dataset. These models are suitable for training a custom object detector using transfer learning.
| Network | Backbone Networks | Size (MB) | Mean Average Precision (mAP) | Object Classes | Location |
|---|---|---|---|---|---|
| EfficientDet-D0 | efficientnet | 15.9 | 33.7 | 80 | GitHub |
| YOLO v4 | yolov4-coco yolov4-tiny-coco |
229 21.5 |
44.2 19.7 |
80 | GitHub |
| YOLO v3 | darknet53-coco tiny-yolov3-coco |
220.4 31.5 |
34.4 9.3 |
80 | Doc |
| YOLO v2 | darknet19-VOC darknet19-COCO tiny-yolo_v2-coco |
180 181 40 |
75.4 28.7 10.5 |
80 | GitHub |
Tips for selecting a model
Pretrained object detectors have different characteristics that matter when choosing a network to apply to your problem. The most important characteristics are mean average precision (mAP), speed, and size. Choosing a network is generally a tradeoff between these characteristics.
Application Specific Object Detectors
These networks have been trained to detect specific objects for a given application.
| Network | Application | Size (MB) | Location | Example Output |
|---|---|---|---|---|
| Spatial-CNN | Lane detection | 74 | GitHub |  |
| Single Shot Detector (SSD) | Vehicle detection | 44 | Doc |  |
| Faster R-CNN | Vehicle detection | 118 | Doc |  |
Semantic Segmentation
Segmentation is essential for image analysis tasks. Semantic segmentation describes the process of associating each pixel of an image with a class label, (such as flower, person, road, sky, ocean, or car).
Inputs are RGB images, outputs are pixel classifications (semantic maps).
This network has been trained to detect 20 objects classes from the PASCAL VOC dataset:
| Network | Size (MB) | Mean Accuracy | Object Classes | Location |
|---|---|---|---|---|
| DeepLabv3+ | 209 | 0.87 | 20 | GitHub |
Application Specific Semantic Segmentation Models
| Network | Application | Size (MB) | Location | Example Output |
|---|---|---|---|---|
| U-net | Raw Camera Processing | 31 | Doc |  |
| 3-D U-net | Brain Tumor Segmentation | 56.2 | Doc |  |
| AdaptSeg (GAN) | Model tuning using 3-D simulation data | 54.4 | Doc |  |
Instance Segmentation
Instance segmentation is an enhanced type of object detection that generates a segmentation map for each detected instance of an object. Instance segmentation treats individual objects as distinct entities, regardless of the class of the objects. In contrast, semantic segmentation considers all objects of the same class as belonging to a single entity.
Inputs are RGB images, outputs are pixel classifications (semantic maps), bounding boxes and classification labels.
| Network | Object Classes | Location |
|---|---|---|
| Mask R-CNN | 80 | Doc Github |
Image Translation
Image translation is the task of transferring styles and characteristics from one image domain to another. This technique can be extended to other image-to-image learning operations, such as image enhancement, image colorization, defect generation, and medical image analysis.
Inputs are images, outputs are translated RGB images. This example workflow shows how a semantic segmentation map input translates to a synthetic image via a pretrained model (Pix2PixHD):
| Network | Application | Size (MB) | Location | Example Output |
|---|---|---|---|---|
| Pix2PixHD(CGAN) | Synthetic Image Translation | 648 | Doc |  |
| UNIT (GAN) | Day-to-Dusk Dusk-to-Day Image Translation | 72.5 | Doc |  |
| UNIT (GAN) | Medical Image Denoising | 72.4 | Doc |  |
| CycleGAN | Medical Image Denoising | 75.3 | Doc |  |
| VDSR | Super Resolution (estimate a high-resolution image from a low-resolution image) | 2.4 | Doc |  |
Pose Estimation
Pose estimation is a computer vision technique for localizing the position and orientation of an object using a fixed set of keypoints.
All inputs are RGB images, outputs are heatmaps and part affinity fields (PAFs) which via post processing perform pose estimation.
| Network | Size (MB) | Location |
|---|---|---|
| OpenPose | 14 | Doc |
Video Classification
Video classification is a computer vision technique for classifying the action or content in a sequence of video frames.
All inputs are Videos only or Video with Optical Flow data, outputs are gesture classifications and scores.
| Network | Inputs | Size(MB) | Classifications (Human Actions) | Description | Location |
|---|---|---|---|---|---|
| SlowFast | Video | 124 | 400 | Faster convergence than Inflated-3D | Doc |
| R(2+1)D | Video | 112 | 400 | Faster convergence than Inflated-3D | Doc |
| Inflated-3D | Video & Optical Flow data | 91 | 400 | Accuracy of the classifier improves when combining optical flow and RGB data. | Doc |
Text Detection and Recognition
Text detection is a computer vision technique used for locating instances of text within in images.
Inputs are RGB images, outputs are bounding boxes that identify regions of text.
| Network | Application | Size (MB) | Location |
|---|---|---|---|
| CRAFT | Trained to detect English, Korean, Italian, French, Arabic, German and Bangla (Indian). | 3.8 | GitHub |
Application Specific Text Detectors
| Network | Application | Size (MB) | Location | Example Output |
|---|---|---|---|---|
| Seven Segment Digit Recognition | Seven segment digit recognition using deep learning and OCR. This is helpful in industrial automation applications where digital displays are often surrounded with complex background. | 3.8 | GitHub |  |
Transformers (Text)
Transformer pretained models have already learned to extract powerful and informative features features from text. Use them as a starting point to learn a new task using transfer learning.
Inputs are sequences of text, outputs are text feature embeddings.
| Network | Size (MB) | Location |
|---|---|---|
| BERT | 390 | GitHub |
Application Specific Transformers
| Network | Application | Size (MB) | Location | Output Example |
|---|---|---|---|---|
| FinBERT | The FinBERT model is a BERT model for financial sentiment analysis | 388 | GitHub |  |
| GPT-2 | The GPT-2 model is a decoder model used for text summarization. | 1.2GB | GitHub |  |
Audio
Audio Embedding pretrained models have already learned to extract powerful and informative features from audio signals. Use them as a starting point to learn a new task using transfer learning.
Inputs are Audio signals, outputs are audio feature embeddings.
| Network | Application | Size (MB) | Location |
|---|---|---|---|
| VGGish | Feature Embeddings | 257 | Doc |
| OpenL3 | Feature Embeddings | 200 | Doc |
Application Specific Audio Models
| Network | Application | Size (MB) | Output Classes | Location | Output Example |
|---|---|---|---|---|---|
| YAMNet | Sound Classification | 13.5 | 521 | Doc |  |
| CREPE | Pitch Estimation (Regression) | 132 | - | Doc |  |
Speech to Text
Speech to text pretrained models take an audio input and translate it into a text output. They are useful in digitizating audio files for downstream text processing tasks such as text summarization and sentiment analysis.
Inputs are Audio signals, outputs are audio feature embeddings.
| Network | Application | Size (MB) | Word Error Rate (WER) | Location |
|---|---|---|---|---|
| wav2vec | Speech to Text | 236 | 3.2 | GitHub |
Lidar
Point cloud data is acquired by a variety of sensors, such as lidar, radar, and depth cameras. Training robust classifiers with point cloud data is challenging because of the sparsity of data per object, object occlusions, and sensor noise. Deep learning techniques have been shown to address many of these challenges by learning robust feature representations directly from point cloud data.
Inputs are Lidar Point Clouds converted to five-channels, outputs are segmentation, classification or object detection results overlayed on point clouds.
| Network | Application | Size (MB) | Object Classes | Location |
|---|---|---|---|---|
| PointNet | Classification | 5 | 14 | Doc |
| PointNet++ | Segmentation | 3 | 8 | Doc |
| PointSeg | Segmentation | 14 | 3 | Doc |
| SqueezeSegV2 | Segmentation | 5 | 12 | Doc |
| SalsaNext | Segmentation | 20.9 | 13 | GitHub |
| PointPillars | Object Detection | 8 | 3 | Doc |
| Complex YOLO v4 | Object Detection | 233 (complex-yolov4) 21 (tiny-complex-yolov4) |
3 | GitHub |
Model requests
If you'd like to request MATLAB support for additional pretrained models, please create an issue from this repo.
Alternatively send the request through to:
David Willingham
Deep Learning Product Manager
dwilling@mathworks.com