A tech demo for scheduling an appointment by speaking with an AI healthcare call center.
This project creates a server instantiating a publicly exposed AI call center to schedule doctor appointments. Upon calling the number, the patient will be speaking with a conversational AI agent (by default "Rachel") which will:
- Guide them through sharing relevant healthcare intake info for scheduling
- Offering example availabilities (time, place, doctor)
- Confirming appointment based on availability selection
- Sending text to confirm said appointment
- Local Python FastAPI RESTful server with ngrok tunnel for exposing it
- AI vocoding and telephony managed by vocode.dev, which orchestrates with the following services:
- Transcribing (caller voice to text) by Deepgram
- Synthesizing (AI text to voice) by ElevenLabs
- Agent (the AI model itself) by OpenAI
- Telephony (in/outbound calling) by Twilio
- Redis for in-memory storage
While there are already choice links to tech sites and docs throughout this README, the following are some more specific links which I found to be particularly useful.
- Vocode python quickstart
- Vocode action agents
- Streaming examples
- Application examples
- App examples hosted on Replit
- Sourcecode for AI agents
- Sourcecode for AI actions
- Searching the Vocode-python repo
- Python 3.8+
- Audio processing: ffmpeg
- Python dep management: Poetry
- Python virtual environment: venv
- Memory: Redis
- Server: ngrok
- Containerization (optional): Docker
- API Tokens (See Installation on where to put the details):
- Deepgram
- OpenAI
- ElevenLabs
- Twilio (must have phone number)
.
├── .env.template # Environment template file
├── .gitignore # Which folders and files git will not track
├── README.md # This file
├── callcenter # Source folder for additional Python modules
│ ├── action_factory.py # Factory to handle creating custom actions in subfolder below
│ └── actions # Custom actions for the AI agent (anything we want it to do beyond talking)
│ ├── best_times.py # Return best appointment availabilities to user (info required)
│ └── send_text.py # Final step (info required + appointment accepted) of sending confirmation text
├── config.yaml # Project-specific values (right now only used for constants)
├── docker-compose.yml # Docker configuration
├── Dockerfile # The Docker container "recipe"
├── main.py # The entry point and core logic of the server application
├── poetry.lock # List of the installed versions of Poetry dependencies
├── pyproject.toml # Defines project information and poetry dependencies
└── requirements.txt # Pip-friendly list of dependencies
Based on the very useful vocode Python docs If you find yourself wanting more details check that out.
- Clone / download this repo and
cd
into it - Create environment file out of template then fill with corresponding values with
cp .env.template .env
- Set up tunneling with
ngrok http 3000
- Be sure to configure ngrok
- This is crucial for communicating with the outside world and services, especially webhooks
- In
.env
changeBASE_URL
to<yoururl>.ngrok.app
generated byngrok
- Be sure to add a webhook to your Twilio number with the url
https://<yoururl>.ngrok.app/inbound_call
to enable our server defined inmain.py
to handle incoming calls with itsinbound_call
endpoint
- Open another terminal window, setup a virtual environment:
python3 -m venv env; source env/bin/activate
- Install dependencies with
poetry install
- Make sure you install Poetry
- Launch Redis:
brew services start redis
- Run the actual server:
poetry run uvicorn main:app --port 3000
- Test it out by calling your Twilio number!
- Vocode easily connects with services for transcription, synthesizing, and AI agents
- ActionAgents (used here) are very powerful, a docstring can tell ChatGPT what action to trigger and what to pass it
- Actions and Agents are highly customizable
- Open source library nascent, and there's a lack of documentation for especially newer and more rapidly changing features such as actions
- There were some very ad-hoc changes required to get certain custom actions to work, such as needing to modify the
ActionFactory
in original the Python module file itself (dev confirmed this is currently the way to do this)
- Using
redis
and kubernetes on cloud deployments not only great for data managament but scalability of service- SortedSets can make use cases for scheduling faster (e.g. pulling best times)
- For HIPPA compliance and competitive edge, self-managed AI agents can be used instead (See the list here)
- If CICD and resource usage can be optimized, a custom LLM agent such as Llama can be trained and used
- Use of
langchain
could be very powerful to handle autonomous actions such as contacting insurance to validate information (think of the applications beyond scheduling)