SSD-based object and text detection with Keras
This repository contains the implementation of various approaches to object detection in general and text detection/recognition in particular.
Its code was initially used to carry out the experiments for the author's master thesis End-to-End Scene Text Recognition based on Artificial Neural Networks and later extended with the implementation of more recent approaches.
Technical background
Most of the ideas used for this project go back to the following papers:
arXiv:1512.02325
SSD: Single Shot MultiBox DetectorSSD is a generic object detector that does local regression and classification on multiple feature maps of a CNN to predict a dense population of bounding boxes, which are subsequently filtered by a confidence threshold and NMS.
arXiv:1611.06779
TextBoxes: A Fast Text Detector with a Single Deep Neural NetworkTextBoxes is a modification of SSD that uses non-square convolution kernels and prior boxes with a large aspect ratio to better detect horizontal text.
arXiv:1708.01241
DSOD: Learning Deeply Supervised Object Detectors from ScratchDSOD is a modification of SSD that uses DenseNet as backbone architecture and thus can be trained form scratch instead of depending on a pretrained VGG-16 model.
arXiv:1703.06520
Detecting Oriented Text in Natural Images by Linking SegmentsSegLink builds on SSD and detects oriented text by locally predicting text segments (objects in SSD) and there linking with each other. The segments (edges) and links (vertices) are considered as a graph and thresholded by confidence. The remaining groups are finally combined to form bounding boxes.
arXiv:1801.02765
TextBoxes++: A Single-Shot Oriented Scene Text DetectorTextBoxes++ extends TextBoxes for arbitrary oriented text by predicting horizontal bounding boxes as well as quadrilaterals and oriented bounding boxes. It additionally uses the recognition score to eliminate false positives from the detection stage (currently not implemented).
arXiv:1708.02002
Focal Loss for Dense Object DetectionThe focal loss is a dynamically weighted version of the cross entropy loss that can better handle a large imbalance between the classes and focus the training process on the difficult samples. It can be applied to the aforementioned detectors, instead of hard negative mining, to overcome the dominance of the background class.
arXiv:1507.05717
An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text RecognitionCRNN is a relatively simple architecture with some convolutional-pooling blocks, followed by two bidirectional LSTM (GRU in this implementation) layers, which can be trained with a CTC for efficient text recognition. It can be used to read the text in the cropped bounding boxes generated by the text detectors mentioned above.
Supported datasets
Currently supported datasets for object detection are
- PASCAL VOC
- MS COCO
and supported datasets related to text are
- ICDAR2015 FST
- ICDAR2015 IST
- SynthText
- MSRA TD500
- SVT
- COCO Text
For more information about the datasets, see datasets.ipynb.
Dependencies
For suitable versions of the necessary dependencies, see environment.ipynb.
Usage
The usage of the code is quite straightforward, clone the repository and run the related Jupyter notebooks. Some of the scripts (e.g. for video and model conversion) can also be executed form the command line.
Pretrained models
Pretrained SSD models can be converted from the original Caffe implementation.
Converted SSD300 VOC
PASCAL VOC 07+12+COCO SSD300* from Caffe implementation
Converted SSD512 VOC
PASCAL VOC 07+12+COCO SSD512* from Caffe implementation
Converted SSD300 COCO
COCO trainval35k SSD300* from Caffe implementation
Converted SSD512 COCO
COCO trainval35k SSD512* from Caffe implementation
SegLink
initialized with converted SSD512 weights
trained and tested on subsets of SynthText
segment_threshold 0.60
link_threshold 0.25
precision 0.884
recall 0.853
f-measure 0.868
SegLink with DenseNet and Focal Loss
trained and tested on subsets of SynthText
segment_threshold 0.55
link_threshold 0.45
precision 0.940
recall 0.904
f-measure 0.922
TextBoxes++ with DennseNet and Focal Loss
trained and tested on subsets of SynthText
threshold 0.35
precision 0.901
recall 0.931
f-measure 0.916
CRNN with LSTM
trained and tested on cropped word level bounding boxes form SynthText
mean editdistance 0.332
mean normalized editdistance 0.081
character recogniton rate 0.916
word recognition rate 0.861
CRNN with GRU
trained and tested on cropped word level bounding boxes form SynthText
mean editdistance 0.333
mean normalized editdistance 0.081
character recogniton rate 0.916
word recognition rate 0.858