Personal projectAI project2025

AI Voice Assistant

A voice-first assistant with LLM intent classification and a low-latency speech pipeline.

01Why I built it

Voice commands span very different intents — search, media, utilities — and each needs a different action.

I wanted to go past a speech-to-text demo. Hearing the words is the easy part — the interesting problem is deciding what the person actually wants, and doing the right thing with it.

Questions I had to answer
  • Q1

    "Play something", "search this", "set a timer" — how do I tell them apart without writing a rule for every phrasing?

  • Q2

    How do I keep the loop fast enough to feel like a conversation?

  • Q3

    How do I add a new skill without rewriting the core?

02Approach

Google Gemini with prompt engineering classifies each spoken request into an intent, which is routed to a pluggable command handler.

03Key decisions
  1. An LLM as the intent classifier

    Prompt-engineered Gemini calls map free-form speech to an intent, so new phrasings work without new rules.

  2. Speech handled in the browser

    The Web Speech API runs speech-to-text and text-to-speech on the client, cutting round trips so replies come back quickly.

  3. Pluggable command handlers

    Each intent routes to its own handler — adding a capability means adding a handler, not touching the classifier.

04Architecture
intentVoice inputmicrophoneWeb Speechspeech-to-textNode.js APIJWT authGeminiintent classifierSpoken replytext-to-speechHandlerssearch · media · utilityMongoDBpersistence
AIServiceDataInterfaceData flow
05AI components
01
Intent classification

Prompt-engineered Gemini calls classify requests across search, media, and utility commands.

02
Speech pipeline

Web Speech API handles speech-to-text and text-to-speech for a low-latency loop.

06Implementation
  • Modular Node.js backend with MongoDB and JWT-based auth.
  • Pluggable command handlers make new capabilities easy to add.
  • Responsive UI built with Tailwind CSS.
07Tech stack
React.jsNode.jsGoogle Gemini APIWeb Speech APIMongoDBJWTTailwind CSS
08Looking back
The hard part

Prompts that classify reliably. Spoken requests are short and ambiguous, so each intent has to be defined sharply enough that search, media and utility commands never bleed into each other.

What I learned

Using an LLM well often means using it narrowly: one clear decision, with ordinary code doing the rest.