Spothole - Artificial Intelligence Powered Pothole Detection, Reporting and Management Solution

Spothole Core Backend (Object Detection + Flask API)

This repository covers the Core Backend End Segment for the Capstone Project for my Master's in Information Sciences and Technologies at Rochester Institute of Technology.

Project Committee Members:

Dr. Charles Border
Prof. Edward Holden
Prof. Bryan French

Introduction

A major problem being faced by municipalities around the world is maintaining the condition of roads be it summer, the monsoons (when it is at its worst) or any weather condition as a matter of fact. And although it’s the responsibility of the authorities to make sure the roads are free of damage, at times they overlook the problem, and most times don’t even know that the problem exists. According to “Safety Resource Center”, approximately 3 Billion US Dollars are spent by motorists for repair of blown tires, busted axles, and other damage to their vehicles. Over the past five years around 16 million drivers across the U.S. have suffered damage from potholes as per an article from “American Automobile Association (AAA)”

Maintaining the road condition is a challenge with constant weather changes, wear and tear, low budgets for the municipalities. Also, not to forget keeping people informed is a task. So, this is an app aimed at solving the challenges mentioned. A reporting system, where the citizen can capture the scene of an area, which will be fed to a machine learning model that will geocode, validate and track down potholes in the scene. This has been achieved by training for object tracking on multiple images and developed using convolutional neural networks. Users can see the damage on the roads using a mobile application or through their browsers. A dynamic report is also generated for the closest authority of concern which they can view and update to create and manage work orders using their own jurisdiction-based web/mobile app-based dashboard.

Motivation

In articles covered by Guardian News & Media potholes took a deadly toll in 2017, claiming almost 10 lives daily. IndiaTimes stated that "Bereaved Father Mr. Dadarao" filled 600 Potholes in Mumbai in memory of son he lost in a road accident! Inshorts reported, potholes killed more people than terrorists reporting 14,926 deaths in road accidents.

When we look at the other side of the world, there is a similar situation as reported by American Automobile Association.

Keeping the roads in good condition along with tracking damages is a challenge with constant changes in weather, low budgets for the municipalities. Not to forget keeping the people informed is a task. This project was aimed at solving the challenges mentioned.

Important URLs

Home (Landing Page)

Link to Landing Page: https://nirbhay.me/spothole/home

Demo

Link to Citizen App Demo: https://nirbhay.me/spothole/
Link to Authority App Demo: https://nirbhay.me/spothole.authority/

Presentation

Link to Canva Presentation: https://bit.ly/spothole-ppt

Process Description

The backend comprises of three main sections. The first being the object detection model built using Darkflow. The second being the database for the application. The third being the Flask API for data exchange between the model built, the database and the frontend.

1. Object Detection Model

As the focus of the application was to create a rest api to automate the process of pothole validation with media files, from the beginning itself a cloud server was used for implementation. A EC2 Amazon Web Services Instance was used for this purpose.
AWS EC2 C5 Instance (model: c5.xlarge) after choosing Ubuntu 18 which features the Intel Xeon Platinum 8000 series and offers a set of 4vCPUs each with 8 GiB of memory was chosen for training the object detection model. Please refer the deployment section to learn more about setting this up.
The model is trained on top of Darkflow and built on top of pretrained weights which were obtained from Darknet.
For crawling images relevant to our label ‘pothole’, images were crawled using the open source google image search package, along with using the serpapi image search tool.
In addition to this freely available pothole video feeds to create the dataset were also used. For a near to decent detection result, we should look to collect at least 500+ images. The dataset has been provided with the repository.
In case of video files, we can upload it to our server using an sftp client like Filezilla or by directly using ssh on the terminal of our local machine. The next step was to write a script to slice this uploaded video to images. Python’s OpenCV (cv2 package and Video Capture Module) to divide the feed into frames.
Using ‘pip install’ the following dependencies need to be added. [pillow, lxml, jupyter, matplotlib, protobuffer]
After cloning the DarkFlow repository, to prepare the input files for DarkFlow we need to consider two things. Firstly, we need an RGB image which is encoded as jpeg or png and secondly, we need a list of bounding boxes (xmin, ymin, xmax, ymax) for the image and the class of the object in the bounding box.
Our class, in this case, is ‘pothole’. We then need to label our images with a tool like LabelImg to identify areas of interest with bounding boxes.
LabelImg is a graphical image annotation tool that is written in Python and uses Qt for the graphical interface.
It supports Python 2 and 3. The annotations are saved as XML files in the Pascal VOC format We can split the data to train and test sets before running the training command.
Now the datasets needed for feeding darkflow package in the required format are available.
We need to first configure Darkflow by modifying the configuration file and labels.txt file.
Then we need to make a copy from cfg/tiny-yolo-voc.cfg and create a cfg/tiny-yolo-voc-1c.cfg file with the same content. Change the line 114 to filters=30 [num * (classes + 5)] and set classes=1 as we have only one class ‘pothole’.
In the label.txt file remove all the labels and just keep the pothole label.
Once done, refer to the training command on this link. Train Yolo with Darkflow, and start training with the dataset created earlier.
Using the test sets the model can be verified to check the accuracy of our newly trained object tracking model. This can now also be applied to various snippets of video to highlight the potential of object detection on potholes. Refer to the link mentioned in the above step for the testing command. It also contains options to restart training from a previous checkpoint.
We will then proceed to saving the built graph as a protobuf file. Read this link on Darkflow to understand this better.

flow --pbLoad built_graph/yolo.pb --metaLoad built_graph/yolo.meta --imgdir sample_img/

The dataset, the annotation files, the built model have all been provided with the repository for reference. You can follow the steps mentioned above only if you wish to have a fresh start to training and testing the model.

2. Database

The database is comprised of 4 main entities. (Authority, Public User, Reports, Comments).
Please refer the database scheme to know more about the attributes associated with each entity.
The authority is someone that has the authorization to manage reports in their zone. A zone is specified as a 5000 mts radius.
A public user is someone that can create a report using the citizen app.
A report is basically details of a submission that has been validated through the object detection model.
Comments are basically communication exchange between an authority and a user on a report.
A service layer file in python has been created with different methods which are used by the api to communicate with the database for performing CRUD operations.

Schema Diagram

3. Flask API

The Flask API has been built for data exchange between the front end applications, the object detection model and the database.
For understanding the front end side of things please refer the important urls section of this read me file.
A app controller script has been created to communicate with the service layers (mysql, mail service and parent(object detection))
The following endpoints have been created for the API.

End Point	Description	Method	Required Params (Keys)	Application
/api/profile/authority/update	For updating a authority's profile details	POST	authorityId, emailId, name, photoURL	Authority
/api/authority/check	For validating auhtority credentials	POST	emailId	Authority
/api/authority/reports/geonear	For querying reports in an authority's zone	POST	authorityId	Authority
/api/authority/update/report	For updating a report's status (for authority)	POST	severity, status, caseId	Authority
/api/profile/authority/data	For retreiving a authority's profile details, location and address	POST	authorityId	Authority
/api/authority/update/user/status	For changing the status of a user (blocked / allowed)	POST	userId, status	Authority
/api/authority/send/email	For notifying a user via email	POST	emailId, message, subject	Authority
/api/reports/all	For retrieving all users reports	POST	(N/A)	Citizen, Authority
/api/reports	For retrieveing a particular user's reports	POST	userId	Citizen, Authority
/api/submit/report/comment	For submitting a comment on a report	POST	userType, commentText, caseId	Citizen, Authority
/api/reports/comments	For retrieving all comments on a report	POST	caseId	Citizen, Authority
/api/submit/report	For submitting a new report	POST	description, location, latitude, longitude, imageURL, severity, userId	Citizen
/api/upload	For uploading files to the server	POST	bytes (file data)	Citizen
/api/detect/single	For detecting whether an image has a pothole. (Object Detection)	POST	image_url	Citizen
/api/profile/update	For updating a user's basic profile details	POST	userId, emailId, name, photoURL	Citizen
/api/user/validate	For validating a user's status (allowed / blocked)	POST	emailId	Citizen

System Diagram

Key Features

Feature Name	App Usage
Darkflow, Darknet Incorporation	Backend
Pothole Detection / Validation (Images, Video)	Citizen
Protobuf Compilation (Model)	Backend
Image Annotations	Backend
Rename and Rearrange Files (Script)	Backend
Image Crawler on top of PyImage Search	Backend
Video File Slicing (Script)	Backend
Output File Storage	Backend
Route 53 (Hosted Zone) on Amazon Web Services (AWS)	Backend
SSL Configuration	Backend
Security Groups (AWS)	Backend
Media Uploads	Citizen
CRUD Operations Authorities Entity	Authority, Citizen
CRUD Operations Public Users Entity	Authority, Citizen
CRUD Operations Comments Entity	Authority, Citizen
CRUD Operations Reports Entity	Authority, Citizen
Email Service	Authority
Allow / Block User Status	Authority
App User Authorization Check	Authority, Citizen
Geo Near Reports Querying	Authority
Flask API Controller	Authority, Citizen

Tools, Libraries and Languages Used

The Backend Application has been built with Python at its Core.
The REST API has been built with Flask.
The Object Detection Model has been built using DarkFlow on top of Darknet.
The Final Model is a Protobuf Compilation (Minified).
Mysql has been used for Database needs.
Package smptlib has been used for Sending Emails.
Package flask_cors has been used to manage Cross Origin Requests.
Package mysql is used to interact between Python and MySql.
Package cv2 is used to create Bounding Boxes at Runtime.
The Parent Service which Communicates with the Deep Learning Model uses the TensorFlow Model created.
Amazon Web Services has been used for Deployment Purposes.
Certbot has been used for HTTPS Configuration
Freenom has been used for Domain needs.
Apache is used for Server needs.
FileZilla and GitBash are used for SFTP and SSH needs.
Chrome Dev Tools and Postman are used for API Manual Testing.

Technology Stack (Complete Frontend and Backend)

Architectural Tiers (Complete Frontend and Backend)

Instructions to Set-Up and Deploy the Backend Application

Amazon Web Service Set-Up

The authority's application has been hosted through Amazon Web Services. To set up the application on a EC2 Instance and a Route 53 Hosted Zone for your choice of Domain Name from a site like Freenom for free refer the following screenshots.
Login to the Amazon AWS Console. Once registered and logged in successfully click on Launch EC2 Instance. Follow the steps in the screenshots.

Your console should now look similar to this after a successful launch.

For connecting your domain, search for Route 53 in the app bar and create a hosted zone. Create a new hosted zone and then record sets for connecting your domain. Use this name servers in your domain's settings to form a connection both ways.

Darkflow Set-Up

Follow the discussion in the process description above. In addition to that refer the README.md file for the darkflow repository. View the links mentioned in the process description and finally review documentaion on Darknet.
Here are some important links. Darknet, Darkflow, Training YOLO with DarkFlow

MySQL Set-Up

Please follow a setup guide like this to setup MySQL on to your instance.
Import the .sql file provided in the repository to finish setting up mysql for the purpose of this application.

Flask Set-Up

You can follow this guide to learn about setting up Flask. Guide
You will have to make changes in the utility file under the flask app's directory by changing the directory constants according to your project structure.
For running on https you will have to setup certbot and use your key files in the app controller. Or you can ignore that run on http.

Apache Set-Up

Follow this guide to learn more. Apache Setup.

Certbot Set-Up

Follow this guide to setup a ssl certificate for your ubuntu ec2 instance Guide

Enjoy using the app. Feel free to make contributions and raise pull requests. Run the app_controller python file to see your api running.

Future Work

Pothole Dimension (Width / Height) Analysis
Machine Level Severity Analysis
OAuth JWT Server Authorization
Worker App Backend
Push Notifications
Authority Hierarchy Management (Database, API)
Permission Level System (Database, API)
Analytics API
Extended Analytics Section (Hierarchy Based)
News Feed (Database, API)
Reward System (Database, API)
User Blocking (Region Level)

Note

For developers out there, if you wish to contribute to the project, feel free to do so. Please review the future work section and create pull requests for ideas and thoughts. Once approved, we can follow up with more discussions.

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Developer

Nirbhay Pherwani

GitHub - @nirbhayph - https://github.com/nirbhayph
LinkedIn - https://linkedin.com/in/nirbhaypherwani
Profile - https://nirbhay.me
Email - pherwani37@gmail.com

Acknowledgements and Mentions

@github - https://github.com
@githublfs - https://git-lfs.github.com/
@smtplib - https://docs.python.org/3/library/smtplib.html
@tensorflow - https://www.tensorflow.org/install/pip
@darkflow - https://github.com/thtrieu/darkflow
@darknet - https://pjreddie.com/darknet/
@dhirensc - https://github.com/dhirensc
@filezilla - https://filezilla-project.org/
@postman - https://www.postman.com/
@chrome-dev - https://developers.google.com/web/tools/chrome-devtools
@aws - https://aws.amazon.com/
@certbot - https://certbot.eff.org/
@freenom - https://www.freenom.com/en/index.html?lang=en
@labelimg - https://github.com/tzutalin/labelImg
@stackoverflow - https://stackoverflow.com/
@mysql - https://www.mysql.com/
@lucidchart - https://www.lucidchart.com/
@flask - https://flask.palletsprojects.com/en/1.1.x/
@cv2 - https://pypi.org/project/opencv-python/
@dbdiagram.io - https://dbdiagram.io/home
@letsencrypt - https://letsencrypt.org/