Welcome to the final project of the camera course. By completing all the lessons, you now have a solid understanding of keypoint detectors, descriptors, and methods to match them between successive images. Also, you know how to detect objects in an image using the YOLO deep-learning framework. And finally, you know how to associate regions in a camera image with Lidar points in 3D space. Let's take a look at our program schematic to see what we already have accomplished and what's still missing.
In this final project, you will implement the missing parts in the schematic. To do this, you will complete four major tasks:
- First, you will develop a way to match 3D objects over time by using keypoint correspondences.
- Second, you will compute the TTC based on Lidar measurements.
- You will then proceed to do the same using the camera, which requires to first associate keypoint matches to regions of interest and then to compute the TTC based on those matches.
- And lastly, you will conduct various tests with the framework. Your goal is to identify the most suitable detector/descriptor combination for TTC estimation and also to search for problems that can lead to faulty measurements by the camera or Lidar sensor. In the last course of this Nanodegree, you will learn about the Kalman filter, which is a great way to combine the two independent TTC measurements into an improved version which is much more reliable than a single sensor alone can be. But before we think about such things, let us focus on your final project in the camera course.
- cmake >= 2.8
- All OSes: click here for installation instructions
- make >= 4.1 (Linux, Mac), 3.81 (Windows)
- Linux: make is installed by default on most Linux distros
- Mac: install Xcode command line tools to get make
- Windows: Click here for installation instructions
- Git LFS
- Weight files are handled using LFS
- For Linux:
- Install using
sudo apt install git-lfs - Once the repo directory has been cloned, run
git lfs pullto pull large files in.
- Install using
- OpenCV >= 4.1
- This must be compiled from source using the
-D OPENCV_ENABLE_NONFREE=ONcmake flag for testing the SIFT and SURF detectors. - The OpenCV 4.1.0 source code can be found here
- This must be compiled from source using the
- gcc/g++ >= 5.4
- Linux: gcc / g++ is installed by default on most Linux distros
- Mac: same deal as make - install Xcode command line tools
- Windows: recommend using MinGW
- Clone this repo.
- Make a build directory in the top level project directory:
mkdir build && cd build - Compile:
cmake .. && make - Run it:
./3D_object_tracking.
Provide a Writeup / README that includes all the rubric points and how you addressed each one. You can submit your writeup as markdown or pdf. The writeup / README should include a statement and supporting figures / images that explain how each rubric item was addressed, and specifically where in the code each step was handled. This README file will detail how I solved this project.
Implement the method "matchBoundingBoxes", which takes as input both the previous and the current data frames and provides as output the ids of the matched regions of interest (i.e. the boxID property). Matches must be the ones with the highest number of keypoint correspondences.
I completed this by filling out matchBoundingBoxes(...). In this method I looped through the list of matches and checked if they were in both frames. If they were I stored the value of that match in a matrix and incremented the match value.
Finally, I counted the occurances in that matrix and selected the one with more keypoints tallied.
Compute the time-to-collision in second for all matched 3D objects using only Lidar measurements from the matched bounding boxes between current and previous frame.
This was completed in the computeTTCLidar(...) method. In this method I average the x values of the previous and current frame ensuring that they were within the lane by checking the y values of each point. I then used the difference in the x values divided by 1 / frameRate to find the current speed. I then calculated TTC by divided the current frame's x value by the calculated speed.
Prepare the TTC computation based on camera measurements by associating keypoint correspondences to the bounding boxes which enclose them. All matches which satisfy this condition must be added to a vector in the respective bounding box.
This was implemented in the clusterKptMatchesWithROI method. This method checks to see if the bounding box contains the keypoint. If so, it is pushed into the bounding box's list of keypoint matches.
Compute the time-to-collision in second for all matched 3D objects using only keypoint correspondences from the matched bounding boxes between current and previous frame.
This was implemented in the computeTTCCamera method. This method follows the example exercise from Lesson 3: Engineering a Collision Detection System, Concept 3: Estimating TTC with a Camera.
Find examples where the TTC estimate of the Lidar sensor does not seem plausible. Describe your observations and provide a sound argumentation why you think this happened.
Generally, it seems like the Lidar provided a very good estimate of the time-to-collision. Ostensibly, this is because lidar can provide a very precise distance measurement. If you can filter the point cloud data to only concentrate on a specific target, this can be really effecty for getting the distance to the next car. Framerate tends to be fairly consistent, so it becomes realtively straight forward to estimate relative velocity and the time to collision.
Run several detector / descriptor combinations and look at the differences in TTC estimation. Find out which methods perform best and also include several examples where camera-based TTC estimation is way off. As with Lidar, describe your observations again and also look into potential reasons.
Generally, the camera based method is not very good. Often, it yields negative times to collision which isn't useful. When this method does yield sensible and accurate results is when the distance ratio is greater than one.