Angelina YangFounder and Host, TwoSetAI · AI builder for visibility and discoverability

How AI Predicts What You'll Say Before You Finish

3 Aug 2026 · 1 min

Watch on YouTube

0:43

In this brief dialogue, Speaker A explains a technique for cutting response latency in voice‑based AI assistants: as soon as the speech‑to‑text engine returns a partial transcript, the large language model (LLM) begins generating a reply. When additional transcript chunks arrive, the ongoing generation is cancelled and a new one starts, ensuring the AI stays ahead of the speaker. Speaker B draws a parallel to human conversation, noting that we often anticipate what the other person will say before they finish. The discussion frames this as the prevailing hypothesis that predictive modeling of the interlocutor’s utterance enables faster, more natural interactions.

One of the tricks to reduce the latency is every time the user's speech comes in from the speech-to-text engine, you start having the LLM generate a response even before the user has finished speaking. Every time you get a chunk of transcripts from the user, you start generating a response with the LLM. The user will keep speaking and you'll have to cancel the previous LLM generation and start a new generation as you get more speech coming in.
Think about how our human brains work. As you're speaking, I'm already thinking.
The prevailing hypothesis is that we're modeling out and predicting what the other person is going to say before they finish speaking and already starting to generate a response.