This project was created as a submission for the Estée Lauder Companies Accessible Beauty Hackathon.
We created an application that makes it easy for blind and visually impaired users to gain information about products by using the device camera.
According to the Vision Loss Expert Group, in 2015 there were over 250 million people with moderate-to-severe visual impairment (MSVI), including 36 million people who were totally blind -- with these statistics only expected to grow over time.
Many things that sighted people take for granted are especially difficult for the visually impaired. Something seemingly as simple as shopping presents a wide array of problems, including accessing basic information about the products they buy.
Improving technologies offer the ability to improve the experience for non-sighted shoppers -- so we have created an application which leverages smartphone technology to improve the shopping experience for all visually impaired individuals
We created a smartphone application that makes it easy for people with low vision to get information about products using just the device camera.
When you open the app, it will try to identify any objects visible to the camera and speak them to you. If you point the camera at a product's barcode, it will give you all the product's information.
- Pick up an item at the store
- Open the app and point the camera at the item
- Listen for object recognition feedback, e.g. "water bottle"
- Move the item around until the barcode scans
- Listen to the product's details spoken aloud
While there are already applications that perform barcode scanning, current solutions have some significant issues:
- Prohibitive costs: KNFB Reader costs $99.99 to download.
- Limited availability: Microsoft's Seeing AI is only available for iOS, which comprises only about a quarter of the global smartphone market.
- Poor UX: current solutions do not provide feedback on visible objects.
Our solution addresses these issues and more:
- Completely free to download and use without advertisements.
- Available for Android and iOS, which together make up more than 99% of the global smartphone market.
- Recognized objects are spoken out loud for better user experience
- Database has over 600 million registered products.
We built the user interfeace for the mobile application using Expo / React Native. This makes it easy to run our project on any Android or iOS smartphone without having to duplicate code, and also easy to distribute during development without having to go through each platform's app store.
We are leveraging several device API's that allow for using the camera, text-to-speech, audio feedback.
By using Google's MobileNet models, we are able to run fast and lightweight object recognition on-device. This enables users to gain more details about any items they are looking at with the camera
To obtain product information once we have a barcode number, we can use barcodelookup.com to query their database with over 600,000,000 product entries. The application requests the product's page and retrieves the product's name, description, image, and attributes.
Note: Some distinct products have the same barcode, which can occasionally result in incorrect product details.
Once presented with product details, the details are read aloud using text-to-speech or the user's configured accessibility tools such as iOS's VoiceOver.
There are certain issues with the current version of the application we would like to address in upcoming versions.
- It's hard to scan a barcode if you can't see where it is.
- This can be solved by using a lightweight Tensorflow.js model trained to recognize the position of blurry or occluded barcodes.
- Based on the bounding box coordinates returned by the model, it would be possible to say "move camera down" or "move closer" to get a clearer view of barcodes.
- Object recognition is prone to mistakes.
- MobileNetV2 is rather outdated as it was released in 2018. Using a newer model is sure to provide more accurate results.
Shopping Assistant is available now with Expo Go -- a platform used to distribute and test React Native applications on mobile devices.
To be released shortly to the Google Play and Apple App Stores
| Julian Hecker | Geraldine Turcios |
|---|---|
 |
 |
| Full Stack Engineer | iOS & UX Engineer |
| React Native Application | User Experience, Accessibility |