A Robust, Real-time, RGB-colored, LiDAR-Inertial-Visual tightly-coupled state Estimation and mapping package
[Dec 31, 2021] Release of code: Our codes are now available in this repo, please kindly follow our instructions to launch our package ^_^. If you have met any bug or problem, please feel free to draw an issue and I will respond ASAP.
[Dec 29, 2021] Release of datasets: Our datasets for evaluation can now be accessed from Google drive or Baidu-NetDisk [百度网盘] (code提取码: wwxw). We have released totally 9 rosbag files for evaluating r3live, with the introduction of these datasets can be found on this page.
R3LIVE is a novel LiDAR-Inertial-Visual sensor fusion framework, which takes advantage of measurement of LiDAR, inertial, and visual sensors to achieve robust and accurate state estimation. R3LIVE is built upon our previous work R2LIVE, is contained of two subsystems: the LiDAR-inertial odometry (LIO) and the visual-inertial odometry (VIO). The LIO subsystem (FAST-LIO) takes advantage of the measurement from LiDAR and inertial sensors and builds the geometric structure of (i.e. the position of 3D points) global maps. The VIO subsystem utilizes the data of visual-inertial sensors and renders the map's texture (i.e. the color of 3D points).
Our preprint paper is available here, with our accompanying videos are now available on YouTube (click below images to open) and Bilibili1, 2.
R3LIVE is robust enough to work well in various of LiDAR-degenerated scenarios (see following figures):
And even in simultaneously LiDAR degenerated and visual texture-less environments (see Experiment-1 of our paper).
R3LIVE is able to reconstruct the precise, dense, 3D, RGB-colored maps of surrounding environment in real-time (watch this video).
To make R3LIVE more extensible, we also provide a series of offline utilities for reconstructing and texturing meshes, which further reduce the gap between R3LIVE and various 3D applications (watch this video).
Following this ROS Installation to install ROS and its additional pacakge:
sudo apt-get install ros-XXX-cv-bridge ros-XXX-tf ros-XXX-message-filters ros-XXX-image-transport ros-XXX-image-transport*
NOTICE: remember to replace "XXX" on above command as your ROS distributions, for example, if your use ROS-kinetic, the command should be:
sudo apt-get install ros-kinetic-cv-bridge ros-kinetic-tf ros-kinetic-message-filters ros-kinetic-image-transport*
Follow this livox_ros_driver Installation.
sudo apt-get install libcgal-dev pcl-tools
You can use the following command to check your OpenCV version, if your openCV version lower than OpenCV-3.3, we recommend you to update your you openCV version if you meet errors in complying our codes. Otherwise, skip this step ^_^
pkg-config --modversion opencv
We have successfully test our algorithm with version 3.3.1, 3.4.16, 4.2.1 and 4.5.3.
Clone this repository and catkin_make:
cd ~/catkin_ws/src
git clone https://github.com/hku-mars/r3live.git
cd ../
catkin_make
source ~/catkin_ws/devel/setup.bash
5.1 Download our rosbag files (r3live_dataset)
Our datasets for evaluation can be download from our Google drive or Baidu-NetDisk [百度网盘] (code提取码: wwxw). We have released totally 9 rosbag files for evaluating r3live, with the introduction of these datasets can be found on this page.
After you have downloaded our bag files, you can now run our example ^_^
roslaunch r3live r3live_bag.launch
rosbag play YOUR_DOWNLOADED.bag
If everything is correct, you will get the result that matches our paper and the results posted on this page.
R3LIVE allow you to save the maps you build at anytime you wanted. You just need to click on the "Control panel" and press 'S' or 's' key.
After you have save your offline map on your disk (default save in directory: ${HOME}/r3live_output), you can launch our utility to reconstruct and texture your mesh.
roslaunch r3live r3live_reconstruct_mesh.launch
As default, your offline map (and reconstructed mesh) will be saved in the directory ${HOME}/r3live_output, you can open it with pcl_viewer (and meshlab).
Install pcl_viewer and meshlab:
sudo apt-get install pcl-tools meshlab
Visualizing your offline point cloud maps (with suffix *.pcd):
cd ${HOME}/r3live_output
pcl_viewer rgb_pt.pcd
Visualizing your reconstructed mesh (with suffix *.ply):
cd ${HOME}/r3live_output
meshlab textured_mesh.ply
Since the LiDAR data and IMU data published by the official Livox-ros-driver is with the timestamp of LiDAR (started from 0 in each recording), and the timestamp of the image is usually recorded with the timestamp of the operation system. To make them working under the same time-based, we modified the source code of Livox-ros-driver, which is available at here. We suggest you replace the official driver with it when sampling your own data for R3LIVE.
We know our packages might not totally stable in this stage, and we are keep working on improving the performance and reliability of our codes. So, if you have met any bug or problem, please feel free to draw an issue and I will respond ASAP.
For reporting our problems and bugs, please attach both your hardware and software environment if possible (printed by R3LIVE, see the following figure), which will be a great help for me in locating your problems.
In the development of R3LIVE, we stand on the shoulders of the following repositories:
- R2LIVE: A robust, real-time tightly-coupled multi-sensor fusion package.
- FAST-LIO: A computationally efficient and robust LiDAR-inertial odometry package.
- ikd-Tree: A state-of-art dynamic KD-Tree for 3D kNN search.
- LOAM-Livox: A robust LiDAR Odometry and Mapping (LOAM) package for Livox-LiDAR.
- openMVS: A library for computer-vision scientists and especially targeted to the Multi-View Stereo reconstruction community.
- VCGlib: An open source, portable, header-only Visualization and Computer Graphics Library.
- CGAL: A C++ Computational Geometry Algorithms Library.
The source code is released under GPLv2 license.
We are still working on improving the performance and reliability of our codes. For any technical issues, please contact me via email Jiarong Lin < ziv.lin.ljrATgmail.com >.
If you use any code of this repo in your academic research, please cite at least one of our papers:
[1] Lin, Jiarong, and Fu Zhang. "R3LIVE: A Robust, Real-time, RGB-colored, LiDAR-Inertial-Visual tightly-coupled state Estimation and mapping package."
[2] Xu, Wei, et al. "Fast-lio2: Fast direct lidar-inertial odometry."
[3] Lin, Jiarong, et al. "R2LIVE: A Robust, Real-time, LiDAR-Inertial-Visual tightly-coupled state Estimator and mapping."
[4] Xu, Wei, and Fu Zhang. "Fast-lio: A fast, robust lidar-inertial odometry package by tightly-coupled iterated kalman filter."
[5] Cai, Yixi, Wei Xu, and Fu Zhang. "ikd-Tree: An Incremental KD Tree for Robotic Applications."
[6] Lin, Jiarong, and Fu Zhang. "Loam-livox: A fast, robust, high-precision LiDAR odometry and mapping package for LiDARs of small FoV."
For commercial use, please contact me < ziv.lin.ljrATgmail.com > and Dr. Fu Zhang < fuzhangAThku.hk >.