Writing
AI Engineering
August 10, 202614 min read

Bypassing Voice Wrappers for Raw Google APIs

Building Low-Latency Conversational Agents That Actually Stick to the Plan

A Senior Product Manager & Builder's guide to engineering direct-stream conversational bots: architecting low-latency WebSockets, avoiding expensive wrappers, and establishing boundary systems to guarantee strict execution goals.

The Voice Bot Illusion

For most product managers, building a conversational voice agent starts and ends with a third-party wrapper. You sign up for an API key on a voice middleware platform, drop a widget on your page, and map it to a basic system prompt. It looks great in a five-minute demo.

But when you try to ship it to production, the reality of the developer-builder hits you. First, the latency is horrific. Every extra hop—from browser to wrapper, wrapper to speech-to-text (STT), STT to LLM, LLM to text-to-speech (TTS), and back—adds 500ms to 1.5s of lag. In conversational voice, 1.5 seconds is the difference between a natural flow and an awkward, stuttering mess. Second, you have zero control. When the agent wanders off-topic or hallucinates a fake phone number, you can't surgically intervene because the agent's turn-taking, interruption logic, and prompt state are black-boxed inside the wrapper's backend.

As a product manager who actually builds, I refused to accept these compromises. We decided to build our voice twin from scratch, bypassing the wrappers and integrating directly with Google's native APIs. Here is how we did it, the limits we hit, and how we engineered them.



1. The Stack: Going Raw with Google's APIs

We bypassed the entire middleware industry and went straight to the source: Google's Multimodal Live API.

Unlike standard REST APIs, the Gemini Live API runs over a persistent bidirectional WebSocket. It ingests raw PCM audio streams directly from the user's microphone and outputs raw audio streams to the user's speakers, performing speech-in, speech-out, and interruption detection in a single, low-latency loop.

Here is what the architecture looks like:
Audio Pipeline: We capture the user's microphone using browser-native AudioContext and a lightweight AudioWorklet running at a clean 16kHz mono.
WebSocket Gateway: We establish an encrypted WebSocket tunnel from the user's browser directly to our edge gateway, which proxies the stream securely to the Gemini Live endpoint.
Direct Streaming: We send raw float32 audio arrays, convert them to 16-bit linear PCM, base64-encode them, and stream them continuously. The model responds in real-time with base64 PCM audio chunks, which we queue and play immediately in the user's browser.


By removing the intermediate STT/TTS layers, we cut round-trip latency from a sluggish 1.8 seconds down to a crisp 320 milliseconds. It feels like talking to a human on a high-quality phone line.



2. The Limits of Unbounded Conversational Agents

In conversational AI, the biggest product risk is deviation from the target user path. When an agent is speaking in real-time, it lacks the luxury of "thinking" steps before answering. It must stream tokens immediately.

During early testing, we observed two critical failure modes:

1
The Interruption Loop: Users would interrupt the bot with a casual remark ("Oh, interesting..."), and the bot would abandon its current plan (e.g., collecting a user's contact information) to dive deep into a tangential discussion about the remark.
2
Goal Drift: If a user asked a complex question, the bot would answer correctly but forget to return to the original product flow, leaving the user stranded in an open-ended conversational dead-end.


To solve this, we couldn't just rely on a standard system prompt. System prompts are easily overridden by strong user intents. We needed structural boundaries.



3. Engineering Steerability: How We Force the Bot to Stick to the Plan

To guarantee our conversational agents execute their target goals, we implemented a three-tier steering and constraint architecture:

A. Structured State Machines in the Edge Proxy

Instead of leaving the conversational state entirely to the model's memory, we wrap the WebSocket session in an edge-based state coordinator.
The coordinator maintains a JSON schema representing the conversational plan (e.g., Phase 1: Welcome, Phase 2: Ingest Problem, Phase 3: Collect Email, Phase 4: Confirm Schedule).
As the conversation progresses, the coordinator uses lightweight semantic classification on the incoming text transcription (which Gemini provides alongside the audio stream) to determine if a state transition has occurred.
If the model attempts to skip a phase or wanders off-topic, the coordinator dynamically injects a high-priority system instruction override (a "nudge") into the WebSocket frame before the next token is generated.


B. The "Anchor" Prompt Pattern

We structured our system instructions using what we call the Anchor Pattern. Every system prompt is divided into three distinct segments:

1
The Core Persona: Who the bot is (concise, professional).
2
The Current Milestone: The specific, single goal the bot must accomplish right now (e.g., "Collect the user's email address").
3
The Anchor Clause: A strict instruction repeating at the end of the prompt: "Regardless of the user's input, answer their question concisely, and then immediately ask for their email address to proceed."


By dynamically updating the "Current Milestone" and "Anchor Clause" via our state machine as the conversation progresses, we keep the model's attention focused on the immediate task.

C. Active Interruption Management

When the user interrupts the bot, the browser instantly stops playing the audio buffer and sends an interruption signal over the WebSocket. The gateway immediately sends a cancel frame to the Gemini Live endpoint, aborting the generation of the current turn.
Before restarting the next turn, our proxy inspects the canceled turn's progress.
If the bot was interrupted mid-sentence while trying to deliver a crucial plan step, the proxy appends a tracking tag to the next user message: [SYSTEM: User interrupted while delivering Milestone 3. Re-assert Milestone 3 immediately.].
This ensures the agent does not lose track of the plan just because the user spoke over it.




The Builder's Takeaway

Building conversational voice bots that feel premium requires owning the pipeline. If you rely on third-party voice wrappers, you are paying a 300% markup for higher latency and zero control over steering.

By building directly on Google's native Multimodal Live API and structuring our own edge-based state routing, we created a system that is incredibly fast (sub-400ms latency), highly steerable, and resilient to conversational drift. As product managers and builders, our job isn't just to make the tech work—it's to design the boundaries that keep the tech working for the user.
EOF
0

DEVESHJOSHI.COM