FEAST, short for Food & Ingredient AI Suggestion Technology, is an innovative project designed to assist individuals in meal planning and preparation by leveraging advanced deep learning techniques. The project employs object detection and text generation models to analyze images of ingredients and generate personalized recipes based on the detected ingredients. This system aims to minimize food waste, optimize meal planning, and enhance the overall cooking experience.
FEAST can significantly enhance food delivery apps by offering personalized meal suggestions based on the ingredients users have at home. By integrating FEAST, apps like Uber Eats and DoorDash can:
- Provide users with meal options that complement their existing pantry items, encouraging users to order specific ingredients they lack.
- Suggest healthier meal alternatives by identifying available ingredients and recommending balanced recipes.
- Enhance user engagement through personalized meal planning features, potentially increasing order frequency and customer satisfaction.
From a consumer's perspective, FEAST offers several advantages:
- Personalized Recipes: Users receive customized meal suggestions tailored to the ingredients they have, reducing the time spent deciding what to cook.
- Reduced Food Waste: By utilizing available ingredients, users can minimize food wastage.
- Cost Savings: Encourages home cooking with available ingredients, potentially lowering the frequency of takeout orders.
- Convenience: Simplifies the meal planning process, making it easier for users to prepare meals without extensive planning.
FEAST integrates object detection and text generation to create a seamless user experience:
- Image Analysis: Users upload images of their ingredients.
- Object Detection: The system identifies and labels the ingredients using trained YOLO models.
- Recipe Generation: A transformer model generates recipes based on the detected ingredients.
- User Interaction: The generated recipes are presented to the user through application.
The project uses two types of datasets:
- Object Detection Data: Collected through web scraping using the Bing image downloader, including images of 100 different ingredients, each class containing 50 images.
- Text Generation Data: The RecipeNLG dataset, encompassing over 2 million recipes, including titles, ingredients, directions, links, and Named Entity Recognition (NER) labels.
The preprocessing steps include:
- Data Cleaning: Ensuring consistency in image formats and directory names.
- Data Augmentation: Augmenting underrepresented classes to ensure uniformity.
- Annotation: Automating the annotation process using Grounding DINO for object detection.
- Tokenization: Tokenizing the recipe data for text generation using the BART tokenizer.
The project trained multiple versions of YOLO models:
- YOLO_v7_Version_1
- YOLO_v7_Version_2
- YOLO_v7_Version_3
- YOLO_v9_Version_1
- YOLO_v9_Version_2
| Version | Precision | Recall | mAP@0.5 |
|---|---|---|---|
| Version 1 | 0.528 | 0.504 | 0.527 |
| Version 2 | 0.526 | 0.49 | 0.547 |
| Version 3 | 0.607 | 0.476 | 0.557 |
| Version | Precision | Recall | mAP@0.5 |
|---|---|---|---|
| Version 1 | 0.416 | 0.391 | 0.394 |
| Version 2 | 0.477 | 0.438 | 0.461 |
The project initially used the BART transformer model for text generation but switched to the ChefTransformerT5 from Hugging Face due to overfitting issues with the BART model.
- Inconsistent Data Formats: Resolved by standardizing image formats and directory names.
- Data Augmentation: Addressed underrepresented classes through augmentation techniques.
- Annotation Difficulties: Automated the process using Grounding DINO for efficient annotation.
- Model Overfitting: Switched to a pre-trained model, ChefTransformerT5, to overcome overfitting issues in text generation.
The locally deployed application, developed using Streamlit, allows users to upload images of ingredients and receive generated recipes. Below is working video showcasing the application's functionality, including ingredient detection and recipe generation.
Extented to structure the recipe and get nutritional value of the food.
- Anil, A. K. "GroundingDino Auto-Annotation." GitHub Repository. GroundingDino Auto-Annotation
- Gaurav, G. "Bing Image Downloader." GitHub Repository. Bing Image Downloader
- Hugging Face. "T5 Recipe Generation." Flax Community
This project in this repository is intended solely as an inspiration for your future projects and should be referenced accordingly. It is not meant for students to fulfill their academic project requirements. If a student uses this project for such purposes, the creators are not responsible. The student will be solely accountable for violating academic integrity. We explicitly state that this repository should not be used to meet academic requirements. Therefore, any academic integrity issues should be addressed with the student, not the creators.