Bud WaaV 1.1: voice agents that keep talking while they work

Interaction mode splits a voice agent in two: a 2B-parameter front model that keeps the conversation going, and a background that does the work as tasks. It is open source, runs on your own hardware, and on Full-Duplex-Bench v3 it starts speaking sooner than every published system.

Mean time to first word on Full-Duplex-Bench v3: Bud WaaV 1.1 at 2.48 seconds, against 3.88 seconds for the next fastest published system.

A voice agent that has to do something on a call, such as look up an order or change a booking, fails in one of two ways. A turn-based pipeline goes silent while its model and tools run. A fast end-to-end speech model talks over the caller every time they pause to think. WaaV Gateway 1.1.0, released on 27 October 2026, adds interaction mode: an open, self-hosted way to have both a responsive conversation and real work running behind it.

2.48s
mean time to first word on Full-Duplex-Bench v3, the soonest of every published system (3.88–8.78 s)
9.7%
of pauses in real conversation where it took the floor, the lowest on the Full-Duplex-Bench v1 leaderboard
44
speech vendors behind the same mode: 31 speech-to-text, 34 text-to-speech and 11 speech-to-speech

We introduced WaaV in August as an open-source audio AI gateway: one Rust binary between an application and the speech vendors it uses. That post ended on the hard part, which is making a cascade of speech-to-text, a language model and text-to-speech feel like a conversation. Interaction mode is our answer to it. Everything below comes from the technical paper that accompanies the release, and the numbers were measured on the released binary, not taken from design targets.

Why voice agents go quiet, or talk too much

People take turns fast. Across ten languages, answers to yes/no questions most often start within about 200 ms of the question ending. Listeners also make small listening sounds while the other person talks, give way when interrupted, and do not treat every pause as an invitation to speak. A voice agent with tools has it harder: the work can take seconds or minutes, the caller keeps talking while it runs, and some of what they say changes the work ("actually, make it eight").

On Full-Duplex-Bench v3, a public benchmark of tool use under real, disfluent speech, a Whisper → GPT-4o → TTS cascade spoke its first words 8.78 s after the caller stopped. End-to-end full-duplex models remove most of that delay but give up floor control: on the v1 leaderboard Moshi starts replying 0.265 s after the user stops, yet takes the floor in 98.5% of synthetic pauses. Neither end of that trade-off works for an agent that also has to act.

In 2026 a third design became prominent: a fast interaction model that stays in the conversation, paired with a slower background model that does the reasoning and tool use. Thinking Machines Lab's interaction models, OpenAI's GPT-Live and Google's Gemini 3.8 Live all work this way. They are closed and hosted, they use the vendor's own voices and recognisers, and they do not expose the decision they make on each turn. What was missing was an implementation an operator can run on its own hardware, with its own choice of vendors, models and agents, and inspect.

Three roles: front, background, gateway

Architecture of interaction mode. Clients reach the WaaV gateway over WebSocket, OpenAI Realtime, speech-to-speech, LiveKit and SIP, and a task API. Inside, the caller's audio passes echo control, noise suppression, voice activity and end-of-turn detection into a floor and barge-in machine. A Qwen3.5-2B front model names one act per caller turn, a policy engine schedules speech on three lanes, and a task registry runs requests on background language models, orchestrators, webhooks, A2A agents, MCP tools or the OpenAI Responses API. Speech-to-text, text-to-speech and speech-to-speech providers sit underneath.
Figure 1 — one interaction actor per call. The front decides and acknowledges, the background works, the gateway owns timing and the words that bind.

Each call gets one interaction actor that owns its state, and the work is divided three ways:

  • The front is a small model, Qwen3.5-2B quantised to 4 bits on llama.cpp. It decides what each caller turn is and says a short line at once.
  • The background does the requested work as a task, on any language model, an orchestrator or a remote agent.
  • The gateway owns timing and the words that bind. It relays the background's answers word for word, and it writes every confirmation, question and failure notice from templates.

That last split is the safety property. The front's prompt forbids it to say a number, date, name or availability the caller did not say, to predict an outcome, or to claim that something is done. A 2B model keeps the conversation moving; it never reports facts.

One explicit act for every caller turn

The front answers every turn with one JSON object naming one act. A grammar built for each decision admits only the acts that are legal at that moment. No two act names share a first letter, so the gateway knows the act from the first generated character and can move before the line is complete. Every decision is logged with what the model was shown.

ActWhen the front chooses itWhat the gateway does
delegateThe caller wants something found out or done.Starts a task and says the acknowledgement.
updateThe caller changes a running task ("no, Thursday").Revises the task and restates only the new value.
abortThe caller wants a task stopped.Cancels it and states the true outcome.
follow_upThe caller asks about a finished task.Answers from the stored result or continues on the same agent.
progressThe caller asks how a task is going.Says the status.
controlThe caller steers the call: stop, hold, repeat, slower.Applies the control.
social, refuseGreetings and small talk, or out-of-scope requests.Says the persona's matching line.
waitMid-thought, "uh-huh", side talk, nothing new.Says nothing.

Table 1 — the front model's acts. Which acts are allowed depends on the state of the call.

The short line that goes with delegate is limited to the persona's openers plus the caller's own words: "Okay, let me check whether your bag is included in the fare." It can restate the request but cannot assert an outcome. A decision has 1.5 s. A late decision is never guessed; the caller is asked to repeat.

Work runs as tasks, not as a blocking call

A task is one piece of work the caller asked for, with a lifetime of its own. It survives barge-in, corrections and a dropped connection, and if it may have side effects, it survives the end of the call. Its states follow the A2A protocol, and it records its effects: none committed, a list of what was done, or unknown.

  • Speculative start. The background run begins on the caller's words while the front is still deciding. Nothing from it is spoken, and no side effect runs, until the act settles as delegate.
  • Corrections. Each correction raises the task's revision. A gate refuses a side-effecting call made under a stale revision, and a request that nearly repeats a recent action is held with "Do you want me to change it, or is this a second one?"
  • Truthful cancellation. Stopping a run never undoes an effect. If the backend cannot confirm the cancel, the caller hears "I've stopped it, but I can't confirm whether it had already gone through."
  • Life after the call. A caller whose connection drops can return within 60 s to hear the outcome; otherwise it is pushed, signed, to the operator's endpoint.

A task can run on the session's own model (any OpenAI-compatible server such as llama-server or vLLM, or OpenAI, Anthropic or Gemini) or on a remote backend: a signed webhook, an A2A 1.0 agent over JSON-RPC, HTTP+JSON or gRPC, an MCP task tool, or an OpenAI Responses model in background mode. With an orchestrator configured, a larger model such as gpt-oss-20b treats each agent as a tool and starts it as a child task, so several agents can work in parallel. Answers are spoken in the background's own words, sentence by sentence as they arrive, and each sentence is dropped if it carries a link, a long number nobody gave, or leaked instructions.

A "yes" only counts if the caller heard the question

A tool declared with side effects is held before its request is sent. The gateway reads the call back from its exact arguments ("To confirm: book a table for 4 at Rosa's. Do you want me to do that?"), and a spoken yes counts only if it began at least 150 ms after the caller heard the read-back in full. That timing comes from a playout timeline the gateway keeps of what the caller actually heard. A decline or a timeout means the request is never sent.

Card numbers, PINs, passwords and one-time codes never go over the voice channel. They go to the client's screen, and while one is being asked for, the speech-to-text provider receives silence of the same length, so a value said aloud anyway reaches neither a transcript nor a vendor. On a phone line the caller can confirm on the keypad.

Duplex behaviour inside a cascade

The audio layer gives a cascade the behaviours a caller expects from a person:

  • It ducks. When the caller starts speaking over the agent, its voice drops 15 dB within 60 ms, a reaction to sound before any words are read.
  • It yields. After 400 ms of overlap the agent pauses, and by 900 ms the overlap is classified as an interruption, a backchannel or background speech.
  • It holds through thinking pauses. A turn that ends on "and" or "to" while the end-of-turn model is unsure is held for another 800 ms.
  • It makes listening sounds. "Mhm" and "I see", rendered in the session's voice, play quietly on their own lane during long turns, and never while private details are being given.
  • It ignores its own echo. A correlation gate, a transcript filter and WebRTC's AEC3 canceller keep the agent from hearing itself as the caller.

All of this runs over the vendor set WaaV already had: 31 speech-to-text vendors, 34 text-to-speech vendors plus self-hosted synthesis, and 11 speech-to-speech integrations, on WebSocket sessions, the OpenAI Realtime protocol, LiveKit rooms and SIP calls.

How it measures

We ran WaaV 1.1.0 on two public benchmarks, using each benchmark's own scoring scripts. Full-Duplex-Bench v3 plays 100 real, disfluent spoken requests to an agent with 12 mock tools; here Claude Opus 5.5 was the background.

SystemResp. qualityPass@1First word (s)Task done (s)
GPT-Realtime 1.50.7920.6006.366.89
Gemini Live 3.10.7180.5403.954.25
Gemini Live 2.50.5540.4907.037.26
Cascade (Whisper + GPT-4o + TTS)0.6000.4508.7810.12
Grok0.6170.4305.976.65
Ultravox v0.70.5100.4103.888.40
Bud WaaV 1.1.0 + Claude Opus 5.50.7300.39 (0.31 strict)2.4814.19

Table 2 — Full-Duplex-Bench v3, 100 examples, as of Q4 2026. Published systems from the benchmark authors; latencies are means from the end of the user's speech.

WaaV speaks first sooner than every system in the table (median 1.84 s) because the front acknowledges at once while the background works, and its response quality is second only to GPT-Realtime. On Full-Duplex-Bench v1, which scores turn-taking across 727 samples, it took the floor in 2.9% of synthetic pauses and 9.7% of pauses in real conversation, the latter the lowest on the leaderboard (Gemini 3.1 10.2%, GPT-Realtime 12.0%). It also makes listening sounds more often than any listed system, 0.054 per second, with the second-closest timing to human listeners.

Where it falls short

Task success is the lowest in the table. Pass@1 is 0.39 judged and 0.31 strict, tool-selection F1 is the lowest at 0.763, and mean task completion is the slowest at 14.2 s (median 7.4 s). Of 61 failed examples, 18 started a task on half a request because a disfluent pause ended the turn, and 13 split one request into several tasks. Those causes are in the gateway's turn handling, not in the models, and are listed for repair in the issue tracker.

Turn-taking and interruptions are slower. On v1 it took the floor at 78% of real turn ends after a mean of 1.28 s, and answered interruptions after 2.21 s; only dGSLM is slower there. A cascade has a latency floor that a single model trained on audio does not, and Thinking Machines Lab's self-reported v3 Pass@1 of 68.0% is well ahead of every system we measured, including ours.

The runs used one shared NVIDIA GB10 with other workloads holding the GPU above 90% for long stretches, and speech over the public internet. On an idle host the front model's first byte arrived in 135–182 ms; on the shared host, with three sessions at once, 410–941 ms.

A voice coworker that uses a computer

Benchmarks score one turn at a time. The launch demonstration shows what the mode is for: a conversation with an agent that does real work on a computer while the talk goes on. Claude Opus 5.5 orchestrated three agents on a sandboxed Linux desktop: a browser, a terminal and LibreOffice Writer. The take was recorded in one go, with no edits.

  1. "Can you pull up the GitHub release for WaaV one point one and check it all looks right?" Acknowledged 1.8 s after the caller stopped. The browser opened the release page, scrolled it and highlighted the Latest label and the release note.
  2. "Remind me, how many speech vendors do we support now?" Small talk while nothing ran: "Forty-four speech vendors. That's thirty-one for speech-to-text, thirty-four for text-to-speech and eleven for speech-to-speech."
  3. "Can you also get the download numbers in the terminal?" The terminal typed a curl | jq query against the GitHub API and reported the downloads for each file.
  4. "Can you draft the launch announcement now, with those numbers in it?" The writer typed the announcement into Writer on screen. Saving is a side effect, so it was read back and saved only after a spoken "Yes, go ahead."

Every request was acknowledged 1.5–1.8 s after the caller stopped, and the call lasted 168 s. Two defects are visible in the take and reported as they happened: the front garbled its spoken summary of the download numbers, and the orchestrator corrected it on its own 25 s later. The 27 takes before it found defects in the gateway that were fixed in PR #49 on the repository.

What comes next

Only the English language pack ships today; a Hindi pack is waiting on review by native speakers. The design documents name the next steps: reviewed language packs beyond English, a front model fine-tuned on teacher-labelled decisions, voice approvals for agents that run their own tools, and evaluations on TurnBench and Full-Duplex-Bench v2, whose correction tasks exercise the update act directly.

Trying it

Interaction mode is opt-in and ships in WaaV Gateway 1.1.0 under the Apache-2.0 licence. The repository is at github.com/BudEcosystem/WaaV. The interaction mode reference covers every setting, event and message, and the design chapters record why each mechanism exists. The front model needs a GPU or a fast accelerator; CPU-only serving was too slow for its 1.5 s budget. Issues, language packs and benchmark runs on other hardware are welcome.

Benchmark figures are from the WaaV interaction mode technical paper (Bud Ecosystem, 27 October 2026), measured on the WaaV Gateway 1.1.0 release binary. Full-Duplex-Bench v3 was run over WaaV's WebSocket adapter rather than the official LiveKit client, with confirmations off and a local gpt-oss-20b as the LLM judge instead of GPT-4o, so the judge-free strict Pass@1 is reported alongside. Full-Duplex-Bench v1 used Qwen2.5-3B as the background and Deepgram for speech. Figures for other systems are from the benchmark authors' published tables and leaderboard; Thinking Machines Lab's figures are self-reported on its own setup.

Get the next one by email

Product releases, benchmarks, and deployment patterns. Monthly, one email, unsubscribe any time.

Unsubscribe any time.

BN
Written by
Bud Newsroom
Bud Ecosystem
Get started with Bud

Put your data on it.

The fastest way to see what an integrated AI operating system does for your enterprise is a proof-of-concept on your infrastructure, with your data.

01 Identify a use case where complexity, cost, or governance is a known pain point.
02 Joint discovery — Bud maps your AI pain points to platform capabilities.
03 POC in days, on your hardware, with your data.