Real conversation broke 80% of the voice-agent speech models
Voice agents are sold on benchmarks recorded in quiet rooms by one person at a time. A new one measured them on two people actually talking — and the leaderboard changed.
Ocular AI published a speech-to-text benchmark this week showing that 12 of the 15 models it tested transcribed real two-person conversation worse than the clean public audio everyone else scores on, and four of them more than doubled their word error rate.
The benchmark, called Converse-STT and run with evaluation startup Cekura, used studio-grade full-duplex recordings of two-person American English conversations made by human speakers, then re-scored the same models on the 1,000 public voice-agent clips from Pipecat's open STT dataset as a control. The gap is the finding. AssemblyAI's Universal 3.5 Pro goes from 1.93% word error rate on the public audio to 4.23% on real conversation, dropping from first place to ninth. Speechmatics' Linden goes from 2.46% to 3.93%. The models that held up best were not the ones topping the clean-audio tables: Reson8 leads the real-conversation set at 2.93%, ahead of Cartesia's Ink 2 at 3.09% and Smallest's Pulse at 3.46%. OpenAI's GPT Realtime Whisper sits at 3.53%.
The caveat is that the two halves are not directly comparable — Ocular scores standard word error rate, while Pipecat scores semantic WER, which forgives formatting differences and only counts errors that change meaning for a downstream agent. So the absolute numbers move for methodological reasons as well as acoustic ones. The direction of the gap is the part that travels: the clean-audio ranking stops predicting the real-conversation ranking as soon as two people are in the room.
That matters because the failure mode is asymmetric. A speech-to-text error on a place name or an account number does not cost you accuracy points, it costs you a turn. The caller repeats themselves, the agent apologises, and the interaction that was supposed to save a phone call takes three. As Ocular puts it, 80% of the models it tested were less accurate once the speech became a real exchange.
The same week, Pipecat's own open benchmark — the one both Ocular and Speechmatics build on — published a fresh run across 23 real-time speech-to-text models, and its headline result is that there is no winner to pick.
Pipecat, the open-source voice-agent framework maintained by Daily, scores 1,000 utterances for latency and semantic accuracy and publishes the code and dataset so anyone can rerun it. Plot latency against accuracy and you do not get a ranking, you get a curve: Meta's muse-voice-transcribe-1.0 is the most accurate on the frontier at 0.83% semantic WER but sits at 392ms median time to final segment, while NVIDIA's Nemotron 3.0 ASR is the fastest at 221ms with 1.95%. Deepgram's nova-3-general lands in between at 247ms and 1.62%. Several models are defensible choices and the benchmark cannot narrow it further — that is what a Pareto frontier means.
The honest reading of both results together is that these benchmarks are good for building a shortlist and bad for making a decision. Pipecat's own methodology notes say so: the audio is English, one speaker at a time, no telephony, no accents switching mid-sentence, and the reference transcripts were generated by Gemini and then human-reviewed — so the "correct" answers start out as one model's output. Vendor-submitted benchmarks are worse, because they are marketing with a table attached. Speechmatics' write-up of the Pipecat results, for instance, is badged by HackerNoon as a vested interest. If you are buying speech recognition for a product that takes real calls, the useful next step is running the open framework against your own recordings, weighted the way your callers are actually distributed.
We have been tracking how fast the voice stack is moving underneath these numbers — Gemini 3.8 Live topped the voice benchmarks at a sixth of GPT-Live's price earlier this month, and that price pressure is exactly why choosing on a leaderboard position is a bad idea.
Chinese automotive software company Banma Intelligent used last week's Apsara Conference to launch AutoOmni 2.0-23B-A3B, a mixture-of-experts on-device model it says runs omni-modal tasks inside a car cockpit without a round trip to the cloud.
Banma claims the 23B-parameter model, with 3B active, matches a cloud model roughly ten times its size on ordinary tasks and reaches 80–90% of that model's performance on complex ones — with inference five to six times faster, memory use cut by more than half, and eight or more tasks running concurrently on the same chip. The company says it worked with ten domestic and international chip makers to get the model optimized at the silicon level, and that 63% of automakers putting on-device models into cars have chosen its platform. Pilots have been shown on the Denza Z9GT, IM LS6 and Hongqi Tiangong 06. There is no production date and no independent benchmark — the numbers are the company's own, announced at its own conference, so treat them as claims rather than measurements.
The strategic logic is more interesting than the spec sheet. Running the model locally removes the network from the loop entirely, which solves latency, connectivity and data-residency problems at once — the same bet BYD made when it put Alibaba's Qwen in its cockpit agent. The hard part is not the model, it is that every car platform has a different chip, and Banma is trying to be the layer that spans them.
What to watch: whether any of the models Ocular found collapsing on real conversation publish a response — and whether the next Pipecat run adds telephony audio, which is where actual voice agents live.
If a speech model scores 99% on clean audio and 96% on a real phone call, which number should go on the box? Tell us in the comments.
Sources: Ocular AI — Converse-STT benchmark · Pipecat STT benchmark (GitHub) · Speechmatics — what the Pipecat benchmark says · Soniox — Pipecat benchmark results and pricing · Securities Daily — Banma AutoOmni 2.0 · TMTPost — Banma's on-device model
