Applications with inference run on local hardware, sometimes without network connections, such as Internet of Things (IoT) devices, as opposed to the cloud. Less data needs to be streamed over a network connection, and real-time decisions can be made.
TheIntel® Distribution of OpenVINO™ Toolkitenables deep learning inference at the edge by including both neural network optimizations for inference as well as hardware-based optimizations for Intel® hardware.
Computer Vision and/or AI models that are already trained on large datasets and available for use in your own applications. These models are often trained on datasets likeImageNet. Pre-trained models can either be used as is or used in transfer learning to further fine-tune a model. The OpenVINO™ Toolkit provides a number ofpre-trained modelsthat are already optimized for inference.
The use of a pre-trained model as a basis for further training of a neural network. Using a pre-trained model can help speed up training as the early layers of the network have feature extractors that work in a wide variety of applications, and often only late layers will need further fine-tuning for your own dataset. OpenVINO™ does not deal with transfer learning, as all training should occur prior to using the Model Optimizer.
A form of inference in which an object in an image is determined to be of a particular class, such as a cat vs. a dog.
A form of inference in which objects within an image are detected, and a bounding box is output based on where in the image the object was detected. Usually, this is combined with some form of classification to also output which class the detected object belongs to.
A form of inference in which objects within an image are detected and classified on a pixel-by-pixel basis, with all objects of a given class given the same label.
Similar to semantic segmentation, this form of inference is done on a pixel-by-pixel basis, but different objects of the same class are separately identified.
SSD {#ssd}
A neural network combining object detection and classification, with different feature extraction layers directly feeding to the detection layer, using default bounding box sizes and shapes/
YOLO {#yolo}
One of the original neural networks to only take a single look at an input image, whereas earlier networks ran a classifier multiple times across a single image at different locations and scales.
Faster R-CNN {#faster-r-cnn}
A network, expanding onR-CNNandFast R-CNN, that integrates advances made in the earlier models by adding a Region Proposal Network on top of the Fast R-CNN model for an integrated object detection model.
MobileNet {#mobilenet}
A neural network architecture optimized for speed and size with minimal loss of inference accuracy through the use of techniques like1x1 convolutions. As such, MobileNet is more useful in mobile applications that substantially larger and slower networks.
ResNet {#resnet}
A very deep neural network that made use of residual, or “skip” layers that pass information forward by a couple of layers. This helped deal with thevanishing gradient problemexperienced by deeper neural networks.
Inception {#inception}
A neural network making use of multiple different convolutions at each “layer” of the network, such as 1x1, 3x3 and 5x5 convolutions. The top architecture from the original paper is also known as GoogLeNet, an homage toLeNet, an early neural network used for character recognition.
Precision refers to the level of detail to weights and biases in a neural network, whether in floating point precision or integer precision. Lower precision leads to lower accuracy, but with a positive trade-off for network speed and size.
Quantization is related to the topic of precision I mentioned before, or how many bits are used to represent the weights and biases of the model. During training, having these very accurate numbers can be helpful, but it’s often the case in inference that the precision can be reduced without substantial loss of accuracy. Quantization is the process of reducing the precision of a model.
With the OpenVINO™ Toolkit, models usually default to FP32, or 32-bit floating point values, while FP16 and INT8, for 16-bit floating point and 8-bit integer values, are also available (INT8 is only currently available in the Pre-Trained Models; the Model Optimizer does not currently support that level of precision). FP16 and INT8 will lose some accuracy, but the model will be smaller in memory and compute times faster. Therefore, quantization is a common method used for running models at the edge.
https://nervanasystems.github.io/distiller/quantization.html
Freezing in this context is used for TensorFlow models. Freezing TensorFlow models will remove certain operations and metadata only needed for training, such as those related to backpropagation. Freezing a TensorFlow model is usually a good idea whether before performing direct inference or converting with the Model Optimizer.
Fusion relates to combining multiple layer operations into a single operation. For example, a batch normalization layer, activation layer, and convolutional layer could be combined into a single operation. This can be particularly useful for GPU inference, where the separate operations may occur on separate GPU kernels, while a fused operation occurs on one kernel, thereby incurring less overhead in switching from one kernel to the next.
OpenCV is an open-source library for various image processing and computer vision techniques that runs on a highly optimized C++ back-end, although it is available for use with Python and Java as well. It’s often helpful as part of your overall edge applications, whether using it’s built-in computer vision techniques or handling image processing.
There’s a lot of uses of OpenCV. In your case, you’ll largely focus on its ability to capture and read frames from video streams, as well as different pre-processing techniques, such as resizing an image to the expected input size of your model. It also has other pre-processing techniques like converting from one color space to another, which may help in extracting certain features from a frame. There are also plenty of computer vision techniques included, such as Canny Edge detection, which helps to extract edges from an image, and it extends even to a suite of different machine learning classifiers for tasks like face detection.
VideoCapture- can read in a video or image and extract a frame from it for processing
resizeis used to resize a given framecvtColorcan convert between color spaces.- You may remember from awhile back that TensorFlow models are usually trained with RGB images, while OpenCV is going to load frames as BGR. There was a technique with the Model Optimizer that would build the TensorFlow model to appropriately handle BGR. If you did not add that additional argument at the time, you could use this function to convert each image to RGB, but that’s going to add a little extra processing time.
rectangle- useful for drawing bounding boxes onto an output image
imwrite- useful for saving down a given image
See the link further down below for more tutorials on OpenCV if you want to dive deeper.
OpenCV has somepretty extensive tutorialsavailable if you want to dive deeper into this useful computer vision library. We'll look at some of the relevant material on handling camera and video inputs next.
Being able to efficiently handle video files, image files, or webcam streams is an important part of an edge application. If I were to be running the webcam on my Macbook for instance and performing inference, a surprisingly large amount of resources get used up simply to use the webcam. That’s why it’s useful to utilize the OpenCV functions built for this - they are about as optimized for general use with input streams as you will find.
We saw thecv2.VideoCapturefunction in the previous video. This function takes either a zero for webcam use, or the path to the input image or video file. That’s just the first step, though. This “capture” object must then be opened withcapture.open.
Then, you can basically make a loop by checking ifcapture.isOpened, and you can read a frame from it withcapture.read. Thisreadfunction can actually return two items, a boolean and the frame. If the boolean is false, there’s no further frames to read, such as if the video is over, so you shouldbreakout of the loop
Once there are no more frames left to capture, there’s a couple of extra steps to end the process with OpenCV.
- First, you’ll need to
releasethe capture, which will allow OpenCV to release the captured file or stream - Second, you’ll likely want to use
cv2.destroyAllWindows. This will make sure any additional windows, such as those used to view output frames, are closed out - Additionally, you may want to add a call to
cv2.waitKeywithin the loop, and break the loop if your desired key is pressed. For example, if the key pressed is 27, that’s the Escape key on your keyboard - that way, you can close the stream midway through with a single button. Otherwise, you may get stuck with an open window that’s a bit difficult to close on its own.