nagarajpandith / aido

3D multilingual chatbot with interactive gestures, contextual conversations, persistent chat history, and personalized medical and mental health support.

Aido

🏆 Awarded the 'Most Innovative Approach' at NivHack 2023 hosted at Niveus Solutions Pvt. Ltd., Mangalore.

• Live URL
• VertexAi API Microservice was built with Flask -- Repo
• Both of the repos were dockerized & deployed on Google Cloud Run
• Final Presentation Pitch Deck

NOTE The deployments might be down right now as the Cloud access was given only during the hackathon.

Features

• Virtually Interactive 3D Character with speech & hand gestures
• Multilingual Chat Sessions in different languages
• Follow-up Questions based on previous conversation
• Persistent chat sessions considering chat history of user
• Personalised medical query & mental health support
• Multilingual Voice Chat Support

Architecture

Tech Stack

• Google PaLM2 based on model chat-bison using Google vertex AI
• Deployed both client & server on Google Cloud Run
• Database on Cloud SQL (PostgreSQL)
• Google OAuth
• Deepgram STT & ElevenLabs TTS
• CI/CD using Google Cloud source repositories & Cloud build
• Google Translate
• T3 Stack: Next.js, tRPC, Prisma, Typescript
• Flask Backend API

Screenshots

Challenges Faced

• No finetuning available for chat-bison model
• It was our first time using & configuring GCP & Gitlab
• No Direct support of CI/CD of Gitlab on Cloud Run
• Didn't get access to Google TTS & STT due to Quota Limit
• The TTS generation & lipsyncing had a lot of latency

Future Scope

• MedPaLM integration for better accuracy.
• Using 3D model for other usecases like kids' trainer, lifestyle expert etc.
• Break the sentences according to fullstops & punctuations before processing further (to avoid latency)
• Can extend the concept to Metaverse domain

Team Members - t3-tribe

• Nagaraj Pandith
• Swasthik Shetty
• Nidheesha T