PULP-DroNet: Judge me by my size, do you? --Yoda, TESB

Authors: Daniele Palossi dpalossi@iis.ee.ethz.ch Vlad Niculescu vladn@iis.ee.ethz.ch Lorenzo Lamberti lorenzo.lamberti@unibo.it Copyright (C) 2021 ETH Zürich, University of Bologna. All rights reserved.

Videos

PULP Platform Youtube channel (subscribe it!)

PULP-Dronet-V1: Video1 Video2.
PULP-Dronet-V2: Video1 Video2.

Citing

If you use PULP-DroNet in an academic or industrial context, please cite the following publications:

Publications:

A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones arXiv preprint -- IEEE IoT Journal
An Open Source and Open Hardware Deep Learning-powered Visual Navigation Engine for Autonomous Nano-UAVs arXiv preprint -- IEEE DCOSS
Automated Tuning of End-to-end Neural FlightControllers for Autonomous Nano-drones IEEE AICAS
Improving Autonomous Nano-Drones Performance via Automated End-to-End Optimization and Deployment of DNNs -- IEEE JETCAS
Tiny-PULP-Dronets: Squeezing Neural Networks for Faster and Lighter Inference on Multi-Tasking Autonomous Nano-Drones -- IEEE AICAS

@article{palossi2019pulpdronetIoTJ, 
  author={D. {Palossi} and A. {Loquercio} and F. {Conti} and E. {Flamand} and D. {Scaramuzza} and L. {Benini}}, 
  title={A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones}, 
  journal={IEEE Internet of Things Journal}, 
  doi={10.1109/JIOT.2019.2917066}, 
  ISSN={2327-4662}, 
  year={2019}
}

@inproceedings{palossi2019pulpdronetDCOSS,
  author={D. {Palossi} and F. {Conti} and L. {Benini}},
  booktitle={2019 15th International Conference on Distributed Computing in Sensor Systems (DCOSS)},
  title={An Open Source and Open Hardware Deep Learning-Powered Visual Navigation Engine for Autonomous Nano-UAVs},
  pages={604-611},
  keywords={autonomous navigation, nano-size UAVs, deep learning, CNN, heterogeneous computing, parallel ultra-low power, bio-inspired},
  doi={10.1109/DCOSS.2019.00111},
  ISSN={2325-2944},
  month={May},
  year={2019},
}

@inproceedings{niculescu2021pulpdronetAICAS,
  author={V. {Niculescu} and L. {Lamberti} and D. {Palossi} and L. {Benini}},
  booktitle={2021 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS)},
  title={Automated Tuning of End-to-end Neural FlightControllers for Autonomous Nano-drones},
  pages={},
  keywords={autonomous navigation, nano-size UAVs, deep learning, CNN, heterogeneous computing, parallel ultra-low power, bio-inspired},
  doi={},
  ISSN={},
  month={},
  year={2021},
}

@ARTICLE{pulpdronetv2JETCAS,
  author={Niculescu, Vlad and Lamberti, Lorenzo and Conti, Francesco and Benini, Luca and Palossi, Daniele},
  journal={IEEE Journal on Emerging and Selected Topics in Circuits and Systems}, 
  title={Improving Autonomous Nano-drones Performance via Automated End-to-End Optimization and Deployment of DNNs}, 
  year={2021},
  volume={},
  number={},
  pages={1-1},
  doi={10.1109/JETCAS.2021.3126259}}

@INPROCEEDINGS{lamberti_tinydronet,
  author={Lamberti, Lorenzo and Niculescu, Vlad and Barciś, Michał and Bellone, Lorenzo and Natalizio, Enrico and Benini, Luca and Palossi, Daniele},
  booktitle={2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS)}, 
  title={Tiny-PULP-Dronets: Squeezing Neural Networks for Faster and Lighter Inference on Multi-Tasking Autonomous Nano-Drones}, 
  year={2022},
  volume={},
  number={},
  pages={287-290},
  doi={10.1109/AICAS54282.2022.9869931}}

1. Introduction

What is PULP-Dronet ?

PULP-DroNet is a deep learning-powered visual navigation engine that enables autonomous navigation of a pocket-size quadrotor in a previously unseen environment. Thanks to PULP-DroNet the nano-drone can explore the environment, avoiding collisions also with dynamic obstacles, in complete autonomy -- no human operator, no ad-hoc external signals, and no remote laptop! This means that all the complex computations are done directly aboard the vehicle and very fast. The visual navigation engine is composed of both a software and a hardware part.

Software component: The software part is based on the previous DroNet project developed by the RPG from the University of Zürich (UZH). DroNet is a shallow convolutional neural network (CNN) which has been used to control a standard-size quadrotor in a set of environments via remote computation.
Hardware components: The hardware soul of PULP-DroNet is an ultra-low power visual navigation module embodied by a pluggable PCB (called shield or deck) for the Crazyflie 2.0/2.1 nano-drone. The shield features a Parallel Ultra-Low-Power (PULP) GAP8 System-on-Chip (SoC) from GreenWaves Technologies (GWT), an ultra-low power HiMax HBM01 camera, and off-chip Flash/DRAM memory; This pluggable PCB has evolved over time, from the PULP-Shield , the first custom-made prototype version developed at ETH Zürich, and its commercial off-the-shelf evolution, the AI-deck.

Evolution of PULP-Dronet

PULP-Dronet-V1:

The first version of PULP-Dronet, which gave the birth to the PULP-Shield: a lightweight, modular and configurable printed circuit board (PCB) with highly optimized layout and a form factor compatible with the Crazyflie nano-sized quad-rotor. We developed a general methodology for deploying state-of-the-art deep learning algorithms on top of ultra-low power embedded computation nodes, like a miniaturized drone, and then we automated the whole process. Our novel methodology allowed us first to deploy DroNet on the PULP-Shield, and then demonstrating how it enables the execution the CNN on board the CrazyFlie 2.0 within only 64-284mW and with a throughput of 6-18 frame-per-second! Finally, we field-prove our methodology presenting a closed-loop fully working demonstration of vision-driven autonomous navigation relying only on onboard resources, and within an ultra-low power budget. See the videos on the PULP Platform Youtube channel: video.

Summary of characteristics:

Hardware: PULP-Shield
Deep learning framework: Tensorflow/Keras
Quantization: fixed-point 16 bits, hand crafted
Deployment tool: AutoTiler (early release, developed in collaboration with GreenWaves Technologies)

We release here, as open source, all our code, hardware designs, datasets, and trained networks.

PULP-Dronet-V2:

This follow-up takes advantage of a new commercial-off-the-shelf PCB design based on the PULP-Shield, now developed and distributed by Bitcraze: the AI-deck. Our work focused in automating the whole deployment process of a convolutional neural network, which required significant complexity reduction and fine-grained hand-tuning to be successfully deployed aboard a flying nano-drone. Therefore, we introduce methodologies and software tools to streamline and automate all the deployment stages on a low-power commercial multicore SoC, investigating both academic (NEMO + DORY) and industrial (GAPflow by GreenWaves) tool-sets. We reduced by 2× the memory footprint of PULP-DronetV1, employing a fixed-point 8 bit quantization, achieving a speedup of 1.6× in the inference time, compared to the original hand-crafted CNN, with the same prediction accuracy. Our fully automated deployment methodology allowed us first to deploy DroNet on the AI-Deck, and then demonstrating how it enables the execution the CNN on board the CrazyFlie 2.1 within only 35-102mW and with a throughput of 9-17 frame-per-second!