Bolt is a light-weight library for deep learning. Bolt, as a universal deployment tool for all kinds of neural networks, aims to minimize the inference runtime as much as possible. Bolt has been widely deployed and used in many departments of HUAWEI company, such as 2012 Laboratory, CBG and HUAWEI Product Lines. If you have questions or suggestions, you can submit issue. QQ群: 833345709
- High Performance: 15%+ faster than existing open source acceleration libraries.
- Rich Model Conversion: support Caffe, ONNX, TFLite, Tensorflow.
- Various Inference Precision: support FP32, FP16, INT8, 1-BIT.
- Multiple platforms: ARM CPU(v7, v8, v8.2), Mali GPU, X86(AVX2)
- Bolt is the first to support NLP and also supports common CV applications.
- Minimize ROM/RAM
- Rich Graph Optimization
- Efficient Thread Affinity Setting
- Auto Algorithm Tuning
- Time-Series Data Acceleration
See more excellent features and details here
Kinds of choices are provided for the compilation of bolt. Please make a suitable choice depending on your environment.
| target platform | build command | Linux | Windows | MacOS |
|---|---|---|---|---|
| Android(armv7) | ./install.sh --target=android-armv7 |  |
 |
 |
| Android(armv8+mali) | ./install.sh --target=android-aarch64 --mali |  |
|
 |
| Android(x86_64) | ./install.sh --target=android-x86_64 |  |
 |
 |
| iOS(armv7) | ./install.sh --target=ios-armv7 | / | / |  |
| iOS(armv8) | ./install.sh --target=ios-aarch64 | / | / |  |
| Linux(X86_64) | ./install.sh --target=linux-x86_64 |  |
/ | / |
| Linux(x86_64_avx2) | ./install.sh --target=linux-x86_64_avx2 |  |
/ | / |
| Windows(X86_64) | ./install.sh --target=windows-x86_64 | / |  |
/ |
| Windows(x86_64_avx2) | ./install.sh --target=windows-x86_64_avx2 | / |  |
/ |
| MacOS(X86_64) | ./install.sh --target=macos-x86_64 | / | / |  |
| MacOS(x86_64_avx2) | ./install.sh --target=macos-x86_64_avx2 | / | / |  |
NOTE: Bolt defaultly link static library, This may cause some problem on some platforms. You can use --shared option to link shared library.
Two steps to get started with bolt.
-
Conversion: use X2bolt to convert your model from caffe,onnx,tflite or tensorflow to .bolt;
-
Inference: run benchmark with .bolt and data to get the inference result.
For more details about the usage of X2bolt and benchmark tools, see docs/USER_HANDBOOK.md.
Bolt has shown its high performance in the inference of common CV and NLP neural networks. Some of the representative networks that we have verified are listed below. You can find detailed benchmark information in docs/BENCHMARK.md.
| Application | Models |
| CV | Resnet50, Shufflenet, Squeezenet, Densenet, Efficientnet, Mobilenet_v1, Mobilenet_v2, Mobilenet_v3, BiRealNet, ReActNet, Ghostnet, SSD, Yolov3, Pointnet ... |
| NLP | Bert, Albert, Neural Machine Translation, Text To Speech, Automatic Speech Recognition, Tdnn ... |
| More DL Tasks | ... |
More models than these mentioned above are supported, users are encouraged to further explore.
Everything you want to know about bolt is recorded in the detailed documentations stored in docs.
- How to install bolt with different compilers.
- How to use bolt to inference your ML models.
- How to develop bolt to customize more models.
- Operators documentation
- Benchmark results on some universal models.
- How to build demo/example with kit.
- Frequently Asked Questions(FAQ)
- 深度学习加速库Bolt领跑端侧AI
- 为什么 Bolt 这么快：矩阵向量乘的实现
- 深入硬件特性加速TinyBert，首次实现手机上Bert 6ms推理
- Bolt GPU性能优化，让上帝帮忙掷骰子
- Bolt助力HMS机器翻译，自然语言处理又下一城
- ARM CPU 1-bit推理，走向极致的道路
- 图像分类: Android Demo, iOS Demo
Bolt refers to the following projects: caffe, onnx, tensorflow, ncnn, mnn, dabnn.
The MIT License(MIT)