Personal projectAI product2026

Cueline

A real-time interview copilot: live transcript, question detection and streamed answers grounded in your resume.

median from end of question to first answer word
~2 smedian from end of question to first answer word
median time to first token (Claude Haiku)
747 msmedian time to first token (Claude Haiku)
layers of question detection
4layers of question detection
01Overview

Cueline listens to the interviewer in a Google Meet or Zoom tab, transcribes them live, works out when a real question has been asked, and streams back a short, speakable answer based on your resume and the job description.

02Why I built it

In a live interview the answer is only useful in the next two or three seconds. That turns an “AI chat” problem into a real-time systems problem: audio capture, streaming speech-to-text, deciding when a question is finished, and an LLM that starts talking fast enough.

Questions I had to answer
  • Q1

    When has the interviewer actually asked a question, and not just said “can you hear me?”

  • Q2

    Which speech-to-text gives live partial results and end-of-speech signals?

  • Q3

    Which model can start answering inside a 3-second budget?

  • Q4

    How do I keep every vendor swappable?

03Key decisions
  1. Deepgram over raw WebSocket, no SDK

    It streams interim and final results and gives end-of-speech signals for free. The protocol is small, so writing it directly keeps every byte visible. An SttStream interface keeps the server provider-neutral.

  2. Measure the model, don't guess

    A benchmark script sent five interview questions through the real prompt: Claude Haiku 747 ms median TTFT, GPT-4.1 mini 1,422 ms, Opus (low effort) 5,303 ms. Detection takes ~1.1 s, so only Haiku leaves real headroom. Opus is one env var away.

  3. Detection layers that get more expensive only when needed

    Rules on every turn (~0 ms), a short hold when a turn trails off on “and / the / to”, an LLM classifier only when rules can't decide, and a stable-interim backstop when STT is slow to finalise.

  4. One shared WebSocket contract

    A shared package holds the message types and audio constants, so client and server can't drift apart.

04Screens
Live transcript with a detected question and a streaming answer
Cueline setup screen in light mode
Cueline in dark mode
Past interview sessions
Cueline on a phone
05Features
Live
  • Captures tab audio (Chrome/Edge) or the microphone
  • AudioWorklet resamples to 16 kHz PCM and streams 50 ms chunks
  • Live transcript with interim and final text
Answers
  • Question detection with rules, hold, classifier and backstop
  • Short, speakable answers grounded in resume + job description
  • Streamed token by token, with latency metrics in the UI
Pluggable
  • Claude, OpenAI, Gemini, Groq or DeepSeek behind one LlmClient interface
  • Deepgram behind an SttStream interface
  • History of past sessions
06Under the hood
01
Latency budget

Measured on a bundled 71-second mock interview (5 questions): Deepgram endpointing ~1.0–1.4 s, then detection, then first token. Median first word ≈ 2 s.

02
Session state machine

One Session per socket runs the pipeline and records latency metrics for every answer.

03
Config validation

Environment is validated with zod at startup; provider selection is one setting.

04
Known limits, written down

Tab audio needs Chrome or Edge; microphone mode has no speaker separation. Next step: speculative generation and a labelled evaluation set.

07Tech stack
Next.jsTypeScriptNode.jsWebSocketAudioWorkletDeepgram Nova-3Claude HaikuZod
08Looking back
The hard part

Answering at the right moment: not on small talk, not halfway through “tell me about a time when…”, and not seconds late. That is why detection is layered instead of one prompt.

What I learned

Latency is a budget you spend stage by stage. Writing the budget down (endpointing, detection, first token) showed exactly where the next second could come from: speculative generation on stable interim text.