AndrewVeee / nucleo-ai

An AI assistant beyond the chat box.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add speech recognition

harish0201 opened this issue · comments

This is an idea that just popped up, while I was trying to search something.

Maybe we can integrate whisper.cpp or something like that, for interaction? It would be awesome if the Assistant can search the web then, and pass it through the summarization engine.

Sorry for the delay in responding - had a little vacation after the release.

I think it would be great to test this out using the API instead of building it into the app from the start. I've seen a few posts on r/LocalLlama about using whisper with LLMs, so it doesn't seem too difficult. You would just have to read in the whisper speech-to-text, then using the openai lib with model="assistant", messages=[{"role": "user", "content": whisper_stt}]), and it should do its thing.

Of course, it would also be great to have tts as well.