This repository walks you through building a driverless car. It is composed of two components:
- A raspberry PI powered car
- A laptop for the ML training
Set up Python 2 environment on the Raspberry PI. To run the data collection script, you will need to set up Python 2.7 on Anaconda; whereas, to run the script that will do the model training, you will need to use Python 3.6 on Anaconda (Follow the instructions below to set up the environment, bearing in mind the version of Python to be used).
Make sure to assemble the car beforehand as per the instructions below (this diagram was created by TheAssembly and can also be found on TheAssembly/IBM_selfdriving repository.
1.Install Anaconda
You can either install Anaconda from here, or get miniconda, which is a lighter version of Anaconda, on your machine. Detailed instructions for miniconda can be found here
2.Setup the environments
Create an environment, for our purposes we will name it sdc, or you can name as you wish.
conda create -n sdc python=3.6
Running this command will create a new conda environment that is provisioned with all libraries you need to run the codes in this repo successfully. Note that you will need to create another environment using a different name(sdc27 for instance) that uses Python 2.7 to run the data collection script (or you can modify the script such that you can run it on Python 3.6)
To uninstall the environment:
conda env remove -n sdc3. Activate the sdc environment:
$ source activate sdcOn Windows, just run:
$ activate sdcActivate sdc27 in the same manner to run the data collection script.
4. Install Tensorflow
conda install -c conda-forge tensorflow
5. Install Keras
conda install -c conda-forge keras
6. Install OpenCV
conda install -c conda-forge opencv
7. Install Numpy
conda install -c conda-forge numpy
Note that you need to install Tensorflow and Keras only on the environment that uses Python 3.6.
As seen in the architecture image, the car collects sensor data and images, then they are all sent to a local server for processing. For training, the images are sent to train a model built with Keras, however when deployed, the images are passed on to this model which returns an instruction. The instruction is then passed back to the raspberry pi which in turn sends commands to the motors.
In the Raspberry Pi Folder, there are 2 files:
control_rc.py: receives commands from the computer to control the car.send_data_inti.py: sends data from the car back to the computer for processing.
In the Arduino Folder, there is 1 file:
rc_control_arduino.ino: sends commands from the raspberry pi to the arduino to instruct the motor to rotate.
In the Computer Folder, there are 4 files, two scripts and two model files:
collect_training_data.py: Receives the data sent from the car and saves it on the computer.model.py: used to train the models based on the data collected from the car. The model is built from the image data collected.move_model.h5andsteer_model.h5: pretrained models to control the car based based on lane tracking.
Figure Above: The trained network is used to generate steering commands from a single front-facing center camera
The trained Machine Learning model is built with Keras on top of Tensorflow. It was developed by Nvidia for their self driving cars, and was adapted for the purpose of this workshop.
Figure Above: CNN architecture
In principle, the model takes in the images from the camera and then produces a command dependant on the position of the lanes in the camera frame.
- Place
control_rc.pyandsend_data_inti.pyon the RaspberryPI and make sure to replace the IP address to that of your machine. - Compile
rc_control_arduino.inoon the arduino. - Place
collect_training_data.pyon your machine. - Run
control_rc.pyandsend_data_inti.pyon the RaspberryPI andcollect_training_data.pyon your machine using the conda environment that uses Python 2.7 (remember to change the IP address incollect_training_data.pyto that of your machine). Notice that incollect_training_data.py, msvcrt used to get the record key stroked (use a different library on on other Operating Systems) - Take the car for a drive around the track, allowing the collection of the data to be used to tarin your model.
- Run
model.pymaking sure you place the data to be consumed by the script in the appropriate location. The result of running the script should bemove_model.h5andsteer_model.h5 - Create the testing script, which is similar to the
collect_training_data.py, and add it to your machine. The modification to the script will include reading the model:steer_model = load_model('steer_model.h5') move_model = load_model('move_model.h5')
and adding the snippet that will make the prediction, based on which commands will be sent to the Raspberry PI, which in turn relays them to the Arduino.
# Start predictions from the model
steer_predictions = steer_model.predict(temp_arr, batch_size=1)
print(steer_predictions)
move_predictions = move_model.predict([self.ultrasonic, self.ir, self.pir], batch_size=1)
print(move_predictions)
# if msvcrt.kbhit():
# key = msvcrt.getch()
#print(key)
if move_predictions > 1.2 and move_predictions <= 2.2:
self.conn.sendall('ff')
elif move_predictions > 0.5 and move_predictions <= 1:
self.conn.sendall('ss')
elif move_predictions <= -0.5 and move_predictions <= -1:
self.conn.sendall('bb')
elif move_predictions >= -0.5 and move_predictions <= 0.5:
self.conn.sendall('no')
if steer_predictions <= -0.5 and steer_predictions <= -1:
self.conn.sendall('ll')
elif steer_predictions > 0.5 and steer_predictions <= 1.1:
self.conn.sendall('rr')
elif steer_predictions >= -0.5 and steer_predictions <= 0.5:
self.conn.sendall('no')Check out this link to learn how to create a a Node-RED application that is supposed to grab car probe data from a Raspberry PI that is positioned inside an RC car and visualize both the data and the driver behavior analysis (the car was assembled slightly differently in the repository linked here).