Caddie AI Backend - README

This README provides an overview of the Caddie AI backend, including its structure, architecture, key pipelines, and technologies used.

Project Overview

Caddie AI is a comprehensive golf swing analysis and coaching application that leverages artificial intelligence (AI) to provide personalized feedback and drills for golfers of all levels. This repository contains the backend code for Caddie AI, which powers the core functionalities of the application.

Directory Structure

└── new_repo/
    ├── app.py
    ├── chronos/
    │   ├── chronos_event_filter.py
    │   ├── distance_matrix.py
    │   ├── xbot_with_chronos.py
    │   ├── xbot_trim_nocsv.py
    │   ├── xbot_mp_mixer.py
    │   ├── dtl_buddy_v2/
    │   │   ├── mean_percentages.py
    │   │   ├── predict_events.py
    │   │   ├── entropy.py
    │   │   ├── universal_infer_dtlbudv2.py
    │   │   ├── resample_spline.py
    │   │   ├── train_utils.py
    │   │   ├── dict_utils.py
    │   │   ├── swing_trimmer.py
    │   │   ├── speech_calls.py
    │   │   ├── mediapipe_video_parse.py
    │   │   ├── tnf_utils.py
    │   │   ├── process_live_dataframe.py
    │   │   ├── main_dtl_bud_v2.py
    │   │   ├── side_bend/
    │   │   │   └── __pycache__/
    │   │   ├── takeaway_encode/
    │   │   ├── Arm/
    │   │   ├── over_the_top/
    │   │   ├── Hips/
    │   │   ├── backswing_encode/
    │   │   ├── backup/
    │   │   │   ├── backup_main_dtl_bud_v2.py
    │   │   │   └── backup_dict_utils.py
    │   │   └── path_encode/
    │   ├── dtl_buddy/
    │   │   ├── infer_hip.py
    │   │   ├── predict_events.py
    │   │   ├── infer_takeaway.py
    │   │   ├── fix_impact_address.py
    │   │   ├── layers.py
    │   │   ├── infer_sidebend.py
    │   │   ├── dtl_buddy.py
    │   │   ├── infer_over_the_top.py
    │   │   ├── universal_infer_dtl_bud.py
    │   │   ├── swing_trimmer.py
    │   │   ├── get_mean_event_columns.py
    │   │   ├── resample_bk_imp.py
    │   │   ├── infer_path.py
    │   │   ├── infer_arm.py
    │   │   ├── process_live_dataframe.py
    │   │   └── __pycache__/
    │   │       └── ready_models/
    │   │           ├── over_the_top/
    │   │           │   ├── layers.py
    │   │           │   ├── tnf_ovthtop.py
    │   │           │   └── infer_tnf_6_10_sn_overthetop.py
    │   │           ├── arm/
    │   │           │   ├── infer_tnf_6_10_sn.py
    │   │           │   ├── layers.py
    │   │           │   ├── tnf_6_10n_sn.py
    │   │           │   ├── tnf_10_10n_sn.py
    │   │           │   └── tnf_arm.py
    │   │           ├── hips/
    │   │           │   ├── layers.py
    │   │           │   ├── tnf_hips.py
    │   │           │   └── infer_tnf_6_10_sn_hips.py
    │   │           ├── path_encoded/
    │   │           │   ├── infer_tnf_6_10_sn_path_encoded.py
    │   │           │   ├── layers.py
    │   │           │   └── tnf_path_encoded_3label.py
    │   │           └── takeaway/
    │   │               ├── infer_tnf_6_10_sn.py
    │   │               ├── layers.py
    │   │               ├── tnf_takeaway_adbk_lite.py
    │   │               ├── tnf_takeaway_lite.py
    │   │               └── tnf_takeaway_adbk.py
    │   ├── wrist_model/
    │   │   ├── wrist_prepare.py
    │   │   ├── wrist_flexion_predictor.py
    │   │   └── __pycache__/
    │   ├── hermes_openai/
    │   │   ├── chat_engine.py
    │   │   ├── hermes_helpers.py
    │   │   └── __pycache__/
    │   ├── angle_calculations/
    │   │   ├── neweventscoring.py
    │   │   ├── event_range_workings.py
    │   │   ├── frame_scoring.py
    │   │   ├── event_scoringt.py
    │   │   ├── new_scoring.py
    │   │   └── __pycache__/
    │   ├── standard_event_classifier/
    │   └── __pycache__/
    │       └── grip_transfg/
    │           ├── modeling.py
    │           ├── old_grip_inference.py
    │           ├── wrist_inference.py
    │           ├── grip_inference.py
    │           ├── configs.py
    │           └── __pycache__/
    │               ├── modeling.py
    │               ├── old_grip_inference.py
    │               ├── wrist_inference.py
    │               ├── grip_inference.py
    │               └── configs.py
    ├── misc/
    │   ├── shell_scripts/
    │   └── misc_modules/
    │       ├── debug_json.py
    │       ├── automate_recording.py
    │       └── twilio_test.py
    ├── migrations/
    │   ├── env.py
    │   └── versions/
    │       └── __pycache__/
    └── __pycache__/
        └── core/
            └── database.py
    ├── core/
    │   ├── database.py
    │   ├── openai_speaker.py
    │   ├── db_ops.py
    │   ├── dtl_flow.py
    │   ├── language_helper.py
    │   ├── audionet.py
    │   ├── helpers.py
    │   ├── highlight_helper.py
    │   ├── ip_helpers.py
    │   ├── dtl_detector.py
    │   ├── blueprint_routes/
    │   │   ├── auth.py
    │   │   ├── klaviyo.py
    │   │   ├── __init__.py
    │   │   ├── cloud_storage_routes.py
    │   │   ├── init.py
    │   │   ├── stripe.py
    │   │   ├── main_routes.py
    │   │   ├── live_routes.py
    │   │   ├── hermes_chat.py
    │   │   ├── db_file_routes.py
    │   │   ├── youtube.py
    │   │   ├── influencer.py
    │   │   ├── json_routes.py
    │   │   ├── subscription.py
    │   │   ├── native_live_swing_upload.py
    │   │   └── __pycache__/
    │   └── __pycache__/
    │       └── routes/
    │           ├── auth.py
    │           ├── __init__.py
    │           ├── auth_api.py
    │           ├── main_routes.py
    │           ├── email_route.py
    │           ├── stripe_routes.py
    │           ├── main_apis.py
    │           ├── json_routes.py
    │           └── __pycache__/
    ├── templates/
    │   └── share_html.py
    ├── static/
    │   ├── revcat/
    │   └── uploads/
    │       └── 0_base/
    │           ├── dustin_johnson_driver_slow_t/
    │           ├── SUZANN PETTERSEN_dtl/
    │           ├── brooks_koepka_s_t/
    │           ├── tiger_trim/
    │           ├── justin_thomas_4i/
    │           ├── LYDIA_KO_dtl/
    │           ├── ADAM_SCOTT_dtl/
    │           ├── rory_driver_face_trim_slow/
    │           ├── HENRIK STENSON_dtl/
    │           ├── tour_videos/
    │           │   └── anthonycritface/
    │           ├── MICHELLE_WIE_dtl/
    │           ├── adamscot_face_trim/
    │           ├── LEXI_THOMPSON_dtl/
    └── misc/
        ├── shell_scripts/
        └── misc_modules/
            ├── debug_json.py
            ├── automate_recording.py
            └── twilio_test.py
    ├── openvino_dir/
    │   ├── infer_images.py
    │   ├── infer_images_backend.py
    │   ├── utils.py
    │   ├── last_openvino_model/
    │   └── __pycache__/
    ├── elo_classf/
    │   ├── elo_classf.py
    │   ├── cnn_elo.py
    │   └── __pycache__/
    ├── core/
    │   ├── config/
    │   │   ├── __init__.py
    │   │   ├── settings.py
    │   │   ├── celery_instance.py
    │   │   ├── load_config.py
    │   │   └── __pycache__/
    │   ├── models/
    │   │   ├── __init__.py
    │   │   ├── user.py
    │   │   ├── subscription_db.py
    │   │   ├── token_blocklist.py
    │   │   ├── academy_video.py
    │   │   ├── swing_elo_ladder.py
    │   │   ├── arabic_dictionary.py
    │   │   └── __pycache__/
    │   ├── dtos/
    │   │   ├── user_dto.py
    │   │   └── __pycache__/
    │   ├── views/
    │   │   ├── __init__.py
    │   │   ├── login_form.py
    │   │   ├── register_form.py
    │   │   ├── checkbox_field.py
    │   │   └── __pycache__/
    │   └── prediction/
    │       ├── __init__.py
    │       └── uploaded_swing_prediction_call.py
    └── db_to_csv/
        └── scripts_for_db_ops/
            ├── import_sqlite.py
            ├── db_queries.py
            ├── csv_to_db.py
            ├── csv_event_images.py
            ├── variables.py
            ├── base_folder_parse_db.py
            ├── drop_sqlite.py
            ├── csv_thai_vid.py
            ├── db_to_csv.py
            ├── calc_percentile.py
            ├── dum_infl.py
            ├── users_db_to_csv.py
            ├── modified_datetime_correction.py
            ├── delete_users_db.py
            └── hash_user.py

Architecture

The Caddie AI backend is designed as a Flask-based application with a layered architecture. Here's a breakdown of the main components, along with associated files:

Frontend (Not Included): The frontend is built using React and handles user interaction, data visualization, and communication with the backend via APIs.
API Layer (core/routes): This layer defines and exposes APIs that are consumed by the frontend. The APIs handle authentication, user management, data processing, and swing analysis.
- core/routes/auth_api.py: Handles user registration, login, logout, password reset, and related API calls.
- core/routes/json_routes.py: Provides endpoints for fetching and managing user videos, accessing academy videos, and retrieving swing analysis data.
- core/routes/email_route.py: Handles sending OTPs for password resets.
- core/routes/main_apis.py: Manages API calls for the dashboard, updating user profiles, payment integration, and other core functionalities.
- core/routes/auth.py: Contains routes for user registration and login via the UI.
- core/routes/main_routes.py: Handles UI routes for the dashboard and user video display.
Business Logic Layer (core/helpers, core/dtl_detector, core/audionet, core/models, chronos): This layer implements the core logic of the application. It includes functionalities like:
- User authentication and authorization (core/models/user.py, core/routes/auth_api.py)
- Data storage and retrieval (core/database.py, core/models)
- Swing analysis pipelines (core/helpers.py, core/dtl_detector.py, chronos)
- Natural language processing for generating feedback (chronos/hermes_openai/chat_engine.py, core/openai_speaker.py)
- Integration with external services (core/routes/main_apis.py)
Data Layer (core/models): This layer provides access to the SQLite database, storing user data, swing analysis results, and other application information.
- core/models/user.py: Defines the User model, storing user information, swing analysis data, and subscription details.
- core/models/user_video.py: Represents user-uploaded videos and their associated data.
- core/models/user_analysis.py: Stores user-specific swing analysis results.
- core/models/subscription_db.py: Manages user subscriptions.
- core/models/academy_video.py: Represents academy videos for learning purposes.
- core/models/academy_thai.py: Represents academy videos in Thai language.
- core/models/arabic_dictionary.py: Stores Arabic translations of golf terms.
- core/models/swing_elo_ladder.py: Represents the swing ELO ladder data.
- core/models/token_blocklist.py: Handles revoked JWT tokens.
- core/models/live_session_data.py: Stores data for real-time practice sessions.
Chronos Folder: The chronos folder is an integral part of the architecture, containing various scripts and modules responsible for specific tasks related to the golf swing analysis process. These tasks include:
- ELO Classification: This module implements a Random Forest classifier to predict swing quality based on the user's pose data.
  - chronos/elo_classf/elo_classf.py
  - chronos/elo_classf/cnn_elo.py
- Event Detection and Filtering: This module contains scripts and functions to detect swing events and filter erroneous data.
  - chronos/chronos_event_filter.py
  - chronos/xbot_with_chronos.py
- XBot Animation Generation: The XBot animation module processes pose data and uses a blend shape approach to generate 3D animations.
  - chronos/xbot_with_chronos.py
  - chronos/xbot_trim_nocsv.py
  - chronos/xbot_mp_mixer.py
- Grip and Wrist Analysis:
  - chronos/grip_transfg/modeling.py
  - chronos/grip_transfg/configs.py
  - chronos/grip_transfg/grip_inference.py
  - chronos/grip_transfg/wrist_inference.py

Key Pipelines with Associated Files

Swing Analysis Pipeline:
- Video Processing (core/helpers.py): Handles video upload, keyframe extraction, and pose data generation using Mediapipe.
- DTL & Face-on Detection (core/dtl_detector.py): Detects video perspective using a TensorFlow model.
- Event Detection (chronos/xbot_with_chronos.py, chronos/dtl_buddy/predict_events.py): Detects key swing events using TensorFlow models.
- Angular Analysis (core/helpers.py, angle_calculations/new_scoring.py): Calculates angles between body parts.
- Score Calculation (angle_calculations/new_scoring.py): Scores angles based on predefined ranges.
- Feedback Generation (chronos/hermes_openai/chat_engine.py): Generates personalized feedback using OpenAI.
- ELO Rating (chronos/elo_classf/elo_classf.py, chronos/elo_classf/cnn_elo.py): Calculates a swing ELO rating using a Random Forest model.
- Pose to Text Generation (chronos/pose_text/pose2text.py, chronos/pose_text/pose2text_new.py): Generates text descriptions of the golfer's swing using a language model.
Live Session Pipeline:
- Real-Time Analysis (chronos/dtl_buddy_v2/main_dtl_bud_v2.py, chronos/dtl_buddy_v2/predict_events.py): Performs real-time swing analysis using Mediapipe and TensorFlow models.
- Session Management (core/models/live_session_data.py, chronos/dtl_buddy_v2/main_dtl_bud_v2.py): Tracks session data and user focus.
- Interactive Guidance (chronos/dtl_buddy_v2/main_dtl_bud_v2.py, chronos/dtl_buddy_v2/speech_calls.py): Provides interactive swing thoughts using OpenAI.
- Live Video Processing (chronos/dtl_buddy_v2/mediapipe_video_parse.py): Parses and processes live video frames for analysis.
- Data Trimming and Resampling (chronos/dtl_buddy_v2/swing_trimmer.py, chronos/dtl_buddy_v2/resample_spline.py): Trims and resamples live video data for more accurate analysis.
User Management Pipeline:
- Authentication & Authorization (core/routes/auth_api.py, core/models/user.py): Handles user management, including registration, login, logout, and token management.
- User Profiles (core/routes/main_apis.py, core/models/user.py): Allows users to create and manage their profiles.
- Subscription Management (core/routes/main_apis.py, core/models/subscription_db.py): Integrates with Stripe for subscription payments and access control.
- Influencer Management (core/models/influencer.py, core/models/influencer_signup.py): Handles influencer signups and referral programs.

Technologies

The Caddie AI backend uses a combination of Python libraries and frameworks, including:

Flask (app.py): Web framework for building the backend application.
SQLite (core/database.py): Database for storing user data and application information.
Mediapipe (core/helpers.py): Used for pose estimation from video frames.
TensorFlow (chronos/dtl_buddy, chronos/dtl_buddy_v2, core/dtl_detector.py): Machine learning framework for model training and inference.
OpenAI (chronos/hermes_openai/chat_engine.py, core/openai_speaker.py): Natural Language Processing API for generating personalized feedback and advice.
Stripe (core/routes/main_apis.py): Payment processing API for subscription management.
Celery (app.py, config.py): Asynchronous task queue for background processing (video analysis, NLP requests, etc.).

Running the Application

Environment Setup: Install the required Python libraries listed in the requirements.txt file.
Database:
- make sure users.db exists .
Configuration:
- API Keys: Add your API keys for external services (Stripe, OpenAI, Google Cloud) in the config.py file.
- Database Connection: Update the SQLALCHEMY_DATABASE_URI in the app.py file to match your database file path.
Run the Application: Run the app.py file.

Deployment

The backend can be deployed using various methods, including:

Docker: Containerize the application and deploy it to a container orchestration platform like Docker Swarm or Kubernetes.

To build and run the Docker container, use the following commands:
```
docker build -t backend_app .
```
```
docker run -p 5001:5001 backend_app
```

App.py: Python=3.9.13

pip install -r requirements.txt

python app.py

Frontend: Now you are ready to go to the React Frontend repo

Further Development

The backend can be further enhanced by:

Adding New Features: Implement features like golf course search, swing comparison with other users, and personalized training plans.
Improving AI Models: Continue training and refining the TensorFlow models to increase accuracy and enhance feedback generation.
Scaling the Database: Use a more robust database like PostgreSQL or MySQL for better performance and scalability.
Improving Performance: Optimize the code and the database queries for efficient processing.

Contact

For any questions or issues, please feel free to contact Yahya Ghani at me@yahyaghani.com

yahyaghani / publicrepo_caddie_ai_backend