AI Voice Assistant
A voice-first assistant with LLM intent classification and a low-latency speech pipeline.
Voice commands span very different intents — search, media, utilities — and each needs a different action.
I wanted to go past a speech-to-text demo. Hearing the words is the easy part — the interesting problem is deciding what the person actually wants, and doing the right thing with it.
- Q1
"Play something", "search this", "set a timer" — how do I tell them apart without writing a rule for every phrasing?
- Q2
How do I keep the loop fast enough to feel like a conversation?
- Q3
How do I add a new skill without rewriting the core?
Google Gemini with prompt engineering classifies each spoken request into an intent, which is routed to a pluggable command handler.
An LLM as the intent classifier
Prompt-engineered Gemini calls map free-form speech to an intent, so new phrasings work without new rules.
Speech handled in the browser
The Web Speech API runs speech-to-text and text-to-speech on the client, cutting round trips so replies come back quickly.
Pluggable command handlers
Each intent routes to its own handler — adding a capability means adding a handler, not touching the classifier.
Prompt-engineered Gemini calls classify requests across search, media, and utility commands.
Web Speech API handles speech-to-text and text-to-speech for a low-latency loop.
- Modular Node.js backend with MongoDB and JWT-based auth.
- Pluggable command handlers make new capabilities easy to add.
- Responsive UI built with Tailwind CSS.
Prompts that classify reliably. Spoken requests are short and ambiguous, so each intent has to be defined sharply enough that search, media and utility commands never bleed into each other.
Using an LLM well often means using it narrowly: one clear decision, with ordinary code doing the rest.