dluc / miyagi

Sample for Intelligent app dev workshop to demonstrate the potential of integrating SoTA foundation models in user experiences and backend workflows. Orchestrated by Semantic Kernel and built on Azure primitives.

Project Miyagi - Financial coach

“start with the customer experience and work backwards for the technology” - Steve Jobs

Systems with AI will replace systems without AI

Project Miyagi serves as the foundation for an envisioning workshop that reimagines the design, development, and deployment of intelligent applications. It demonstrates that integrating intelligence transcends a simple chat interface and permeates every aspect of the end-user experience, utilizing enriched data to generate personalized interactions and effectively address individual needs. Through a comprehensive exploration of a financial coaching use case, Miyagi offers hands-on experience with cutting-edge programming paradigms that harness the power of foundation models in every workflow. Additionally, it introduces traditional software engineers to emerging design patterns in prompt engineering (chain-of-thought, few-shot, in-context), vectorization for long-term memory, and the affordances provided by LLMs.

The project also includes examples of usage for popular frameworks and orchestrators like Semantic Kernel, LangChain, Jarvis, vector stores (Qdrant, Chroma, Milvus, Pinecone), and generative image utilities such as DreamFusion and ControlNet. Additionally, it features foundation Models from Azure OpenAI and Cognitive Services. Utilize this project to gain insights as you modernize and transform your applications with AI and fine-tune your private data to build your own Copilot.

Embedded with intelligence and built on a scalable event-driven architecture, Project Miyagi emphasizes customer-centricity. It challenges you to rethink how AI can curate and create hyper-personalized user interactions, whether in a line-of-business (LOB) or consumer-facing app. It features an easy-to-understand use case that offers concrete examples of how these new AI platforms and architectures, supported by Azure, can be leveraged for valuable insights.

This polyglot codebase relies on a multitude of microservices, choreographing retail and high-frequency trading. It includes generative text and images for personalized financial coaching, summarization, and agent-like orchestration. Built on a cloud-native EDA backbone, the architecture and codebase ensures enterprise-grade quality attributes such as availability, scalability, and maintainability.

Embark on a journey to transform your applications into cutting-edge, intelligent systems with the self-guided workshop and discover the art of the possible.

Work in Progress

Partial Implementations

Due to the rapid pace of advancements in foundation models, we are incrementally implementing use cases for Miyagi in the experiments folder. So far, we have the following implemented:

1. Knowledge Graph memory using Langchain's entity cache
2. Qdrant vector store for embeddings via Langchain
3. MS Graph API intent invoked via Semantic Kernel's skills
4. Miyagi prompt engineered chat interaction using LangChain's PromptTemplate
5. Azure OpenAI GPT-3.5 basic flow
6. GPT-3.5-turbo and Whisper-1 usage to transcribe audio and demonstrate few-shot example
7. Reliability & Verifiability of model output (Responsible AI: Grounding, Retrieval augmented Generation (RaG) and Alignment) - coming soon
8. DeepSpeed Chat MiyagiGPT (BYO Weights w/ RLHF - Reinforcement Learning from Human Feedback) - coming soon

Frontend

Interaction with foundation models is more than chat. This sample shows a few use cases

Architecture

High-level logical architecture

Semantic Kernel Orchestration for Miyagi usecase

In-context learning flow

30k foot view

Initial ideation for EDA + SK flow

Generative image use case architecture with Dreambooth

This will be similar to reddog product image generation use case.

Tech Stack

• Azure OpenAI
  • gpt-4
  • gpt-35-turbo
  • text-embedding-ada-002
• Semantic Kernel
• Azure HuggingFace Inference Endpoints
• Jarvis
• LangChain
• Foundation Models from CogServices
• Qdrant
• Microsoft DeepSpeed Chat
• Azure Web PubSub
• Azure Communication Services (ACS)
• Azure Functions
• AKS / ACA
• Azure Monitor
• Cosmos DB
• Azure DB for PostgreSQL
• Azure Redis Cache
• Azure Storage
• Apache Kafka on Azure Event Hubs
• Github Actions

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [e-mail] with any additional questions or comments.