PencilNet
This repo will host dataset, models, and implementation code of PencilNet.
Abstract
In autonomous and mobile robotics, one of the main challenges is the robust on-the-fly perception of the environment which is often unknown and dynamic, like in autonomous drone racing. In this work, we propose a novel deep neural network-based perception method for racing gate detection -- PencilNet -- which relies on a lightweight neural network backbone on top of a pencil filter. This approach unifies predictions of the gates' 2D position, distance and orientation in a single pose tuple. We show that our method is effective for zero-shot sim-to-real transfer learning that does not need any real-time training samples. Moreover, compared to state-of-art methods, our framework is extremely robust to illumination changes commonly seen under rapid flight. A thorough set of experiments demonstrates the effectiveness of this approach in multiple challenging scenarios, where the drone completes various tracks under different lighting conditions.
Dependencies
- tensorflow (For training and evaluation: version 2.8. For Nvidia Jetson TX2: version 1.5)
- numpy
- tensorboard
- pyaml
- ROS melodic/noetic
It is highly recommended to use a virtual environment (e.g. anaconda. You can find an example anaconda environment in /misc/environment.yml ). If you use the provided anaconda environment, all training and evaluation should work (except for the ROS package, as it depends on whether you are using ROS Melodic (so python 2.7) or Noetic (python 3)).
Dataset
The simulation dataset generated by Unreal Engine 4 from various environments together with our drone racing gate design. The print-ready gate pattern and its 3D mesh model will be also made available for interested researchers.
-
Training dataset: 31k RGB fisheye-distorted images, annotation file (*.json) and training & testing indices (0.9 and 0.1 respectively in *.npy) can be downloaded here: RGB training dataset. Respective pencil images of the same dataset can be downloaded here: Pencil training dataset
-
Testing dataset: multiple images from different simulation environments (that do not appear in training dataset) and real world scenarios can be downloaded here: RGB test dataset. Again, the respective pencil images of the test dataset can be obtained here: Pencil test dataset
Trained models
Will be provided shortly.
Training and Evaluation
For training, inside train.py, change the config for the right directory of the dataset folder accordingly:
config["dataset_folder"] = "~/your_training_data_set"
and also the directory for the trained model:
logger.set_save_dir("./trained_model")
You can also change other parameters of the config inside train.py, for example, config['epochs'] = 500. Then you can train the pencilnet model:
python train.py
For other baselines, we will provide the training file and configurations in separate folders in this repo.
ROS detection node
Build the package with catkin build
catkin build pencil_perception_module
then, to launch the node:
roslaunch pencil_perception_module perception.launch
Bag file testing
run the bag file with the following options:
--topics /camera/image_raw /tf /tf_static /ground_truth/gate_pose_array /state_estimator/drone/pose_on_map /mavros/local_position/odom /mavros/vision_pose/pose /vicon/gate_1/gate_1 /vicon/gate_2/gate_2 /vicon/gate_3/gate_3 /vicon/gate_4/gate_4 --clock