We’ve open-sourced WaaV Audio AI Gateway. It is a real-time voice AI gateway written in Rust, sitting between an application and the fragmented ecosystem of speech providers, offering one WebSocket and one REST interface in place of thirty different SDKs. This article explains what the project does, why it was built the way it was, and what problem each part of it solves.

WaaV is a single Rust binary that an application talks to instead of talking to speech vendors. It accepts audio over WebSocket, REST, WebRTC or SIP, cleans it, sends it to whichever speech-to-text, text-to-speech or audio-to-audio service the configuration names, works out when the speaker has finished, and streams the reply back. Around that core sit the things any production voice system eventually needs, including connection pooling, response caching, rate limiting, authentication and recording, all built once rather than once per vendor. The application never learns which provider answered, and switching from one to another is an edit to a configuration file.

What that buys is freedom of movement across a market that is genuinely fragmented. No single speech vendor is good at every language, and the ones that handle Malayalam, Vietnamese or Gulf Arabic well are rarely the ones with the best English voices. WaaV carries more than seventy integrations precisely so a deployment can use several at once, choosing per request rather than per project, and can point at a model running inside your own data centre with equal ease. On top of that sits a routing engine for describing the pipeline itself, so the chain of speech recognition, language model, tool calls and synthesis can branch, run in parallel and fall back without any of that logic living in the application.

An audio gateway is not an LLM gateway with a microphone attached

Teams already running a text gateway in front of their language models often assume audio can be handled by the same component. The two are structurally different pieces of software.

A text gateway is largely a request-and-response system. It multiplexes API calls, handles keys, retries, budgets and fallbacks, and the unit of work is a message. An audio gateway has to hold open bidirectional streams, manage continuous PCM frames in both directions, deal with codecs and sample rates, run signal processing on the audio before it ever reaches a provider, and decide in real time when a person has stopped speaking. None of that fits the shape of a text gateway, and bolting it on produces something that works in a demo and falls apart on a bad mobile connection.

The coverage problem

The first reason WaaV exists is coverage, and this is the part of the project easiest to underestimate.

Setting out to build a genuinely global voice agent with existing open source tooling gets blocked fairly quickly. Frameworks such as Pipecat provide a pipeline abstraction, but the set of cloud integrations available through them clusters around the same handful of well-known Western providers. That works if your users speak English, Spanish, French or German. It stops working the moment you need Malayalam, or Vietnamese, or a Gulf Arabic dialect, or one of the many African languages served well by regional providers and served poorly or not at all by the global ones.

WaaV ships with more than seventy cloud provider integrations, selected to maximise coverage across languages, capabilities and features rather than by brand recognition. Alongside Deepgram, Google Cloud, Azure, OpenAI, ElevenLabs, AssemblyAI, Cartesia, AWS and IBM Watson, the gateway carries Sarvam AI, Gnani, Reverie and Bhashini for India; iFlytek, Alibaba, Baidu, Tencent and Huawei for China; Yandex, Tinkoff and SberDevices for Russia and the CIS; NAVER CLOVA and AmiVoice for Korea and Japan; and Zalo, FPT, Viettel, Prosa and NECTEC across South East Asia. The current build supports 27 speech-to-text providers, 32 text-to-speech providers and two audio-to-audio realtime providers.

What this buys in practice is the ability to mix. A single deployment can route English through ElevenLabs for voice quality, Indian languages through Sarvam or Bhashini for accuracy and compliance reasons, and Middle Eastern dialects through a third provider entirely, without the application code knowing anything has changed. Provider selection becomes a configuration value rather than an integration project.

The latency problem

The second reason is performance, and the argument here concerns user experience rather than infrastructure cost.

Voice is unforgiving in a way text is not. A chat interface taking two seconds to start responding feels normal. A voice agent taking two seconds to start responding feels broken, and users disengage from it almost immediately. The perceptual budget for a natural-feeling spoken response sits under a few hundred milliseconds end to end, which leaves every component in the chain competing for a very small allowance.

This is why the gateway is written in Rust rather than Python. Published benchmarks put gateway overhead at a P50 of 0.343ms and a P99 of 1.384ms, with peak throughput of 112,528 requests per second and a resident memory footprint of 38MB. Those numbers are not end-to-end latency. They measure only what the gateway adds on top of whatever the provider takes. The useful comparison is that a typical inference gateway adds somewhere in the region of 1.5 to 3ms per request, and in a budget where the whole interaction needs to feel instantaneous, returning two milliseconds to the model rather than the plumbing is worth caring about.

The scaling table shows latency rising gently up to ten thousand concurrent users and then jumping sharply at twenty-eight thousand, where P99 reaches 28.7 seconds. That cliff reflects the compute ceiling of the single machine used for the test, which ran without horizontal scaling. Error rate stayed at zero throughout, so the failure mode under saturation is queueing rather than dropping.

The orchestration problem, and why conversation feels unnatural

The third reason goes beyond routing, and it is the most interesting part of the project.

A voice agent is almost never a single model. A realistic customer support flow involves voice activity detection, possibly speaker diarisation, a speech-to-text model, a language model that may call tools and wait on their results, and then a text-to-speech model to produce the reply. Each step adds latency, and the sum of them is what makes so many voice agents feel stilted. You finish speaking, there is a pause while the machinery grinds, and then a fully formed answer arrives. Real conversation does not work like that. People interrupt, backchannel, and think out loud while they talk.

Thinking Machines Lab set this problem out clearly in its May 2026 work on interaction models, arguing that most real-time systems only emulate interactivity by wrapping turn-based models in a harness that predicts turn boundaries, and that native full duplex, processing input and generating output in continuous micro-turns, produces something closer to human conversational rhythm.

WaaV takes the pragmatic route to the same destination. Rather than requiring a natively full-duplex model, it emulates that class of interaction behaviour using models that already exist, through the gateway. Turn detection runs as an ONNX post-processing stage using LiveKit’s SmolLM-based turn-detector model, quantised to INT8, with a configurable probability threshold and a sub-50ms prediction target. Noise suppression runs as a DeepFilterNet pre-processing stage with SNR-adaptive filtering, so clean audio is left largely alone and noisy audio gets treated. Above those, the orchestration layer allows a larger model to work on a tool call in the background while a smaller, faster path keeps the conversation moving.

Full duplex here means what it means in telecommunications: both directions are live at once. Start speaking over the agent and it yields, the way a person would. Most deployed speech systems cannot do this, because they are strictly half duplex and stop listening while they speak.

The DAG engine

Underneath the conversational behaviour sits a directed acyclic graph engine. Nodes come in six categories: inputs, provider nodes for speech-to-text, text-to-speech and language models, processors for transformation, filtering and aggregation, routers for switching, conditionals and split and join patterns, outputs, and endpoint nodes reaching out to HTTP, gRPC, WebSocket, IPC and LiveKit destinations. Conditional logic is expressed in Rhai. Graphs are pre-compiled and pass data between nodes without locking, which keeps the abstraction from eating the latency budget it exists to protect.

Practical uses include sending short utterances to a small fast model and longer ones to a larger model, running two providers in parallel and taking whichever returns first, A/B testing a new voice against an incumbent based on API key identity, or fanning out to a transcription branch and a summarisation branch simultaneously.

Getting audio in and out

None of this is useful if the audio cannot reach the gateway. WaaV integrates with LiveKit for WebRTC rooms and SIP telephony, works with Daily, and exposes plain WebSocket streaming for applications preferring to handle transport themselves. SDKs exist for TypeScript and Python, both organised around three primitives: stt, tts and talk for bidirectional sessions. The TypeScript SDK includes a metrics collector for time-to-first-token and connection timing, and automatic reconnection with exponential backoff, which matters more than it sounds when the client is a phone on a patchy network. There is also a development dashboard for inspecting transcripts, latency and WebSocket traffic, and an embeddable web component for dropping a voice interface into an existing page.

Operationally, the gateway carries HTTP/2 connection pooling, TTS response caching keyed by XXH3 content hashes, per-IP token bucket rate limiting, optional JWT authentication against an external validation service, optional TLS via rustls, and S3-backed recording storage. Configuration is YAML with environment variable override.

On-premise, sovereign and guarded

Because the gateway abstracts providers rather than hosting models, it does not care whether the model it calls is a commercial cloud API or something running in your own data centre. Organisations with audio models deployed on-premise can route to them through the same interface, making the gateway usable in environments where audio cannot leave the jurisdiction or the building. Combined with the regional provider coverage, this is what makes the project relevant to sovereign deployments rather than only to consumer applications.

Guardrails work along two paths. Audio-specific guardrails handle the usual cases directly on the audio stream. Where the pipeline converts speech to text and passes it to a language model, the converted text flows through whatever text-based guardrail policies are already in place, so existing controls are inherited rather than rebuilt.

Trying it

The repository is at github.com/BudEcosystem/WaaV, licensed under Apache 2.0. The gateway builds with Cargo, optional features are gated behind dag-routing, turn-detect, noise-filter and openapi flags so the base binary stays small, and there is a Docker build for anyone preferring not to compile Rust. Contributions, provider additions and issue reports are welcome, and provider coverage in particular improves fastest when people who actually speak the language in question get involved.