NVIDIA's 100M-param diarizer tops the speaker-label leaderboard
NVIDIA quietly shipped the missing half of speech AI this week: not what was said, but who said it.
Nemotron 3 Diarization is a 100-million-parameter open-weight model that labels up to eight speakers in live or recorded audio, and it arrives ranked first on VoiceArena's initial Diarization-Bench. Across 139 English conversations totalling roughly 22 hours, it scored a 14.72% diarization error rate against 19.3% for the next-best of the 12 systems evaluated — about a 24% relative gap — with overlapping speech included and no boundary tolerance. That is a small model, openly licensed for commercial use, beating systems that until now lived behind a per-minute API.
The interesting engineering is that one checkpoint covers both live and offline processing without embedding or clustering stages. The previous generation of open diarizers split the problem in two: segment the audio, then cluster voice embeddings to decide which segments belong to the same person. Nemotron 3 does it in a single pass, emitting an eight-channel tensor of per-speaker activity probabilities, with output channels ordered by whoever spoke first. Two pieces of state — a cache that remembers earlier voices and a short queue of recent frames — carry speaker identity across chunks in streaming mode, which is why a speaker who pauses for a minute doesn't come back as a stranger. NVIDIA reports an average 41% relative error reduction at the 1.04-second setting against its own four-speaker predecessor, and a much wider margin on the hardest recordings: 11.00 versus 30.49 on NOTSOFAR1, 11.14 versus 21.42 on AMI. The same checkpoint serves four latency profiles from 30.4 seconds down to 0.32 seconds, so a developer can pick responsiveness per request rather than per deployment.
The honest caveat is in NVIDIA's own numbers. On the two-person CALLHOME subset, the new model is marginally worse than the old baseline — 5.98% against 5.68% — before it wins the dataset as a whole at 9.10%. Eight speakers is also the ceiling: anything larger gets split, missed or mislabelled. And the model returns anonymous speaker channels, not names; mapping "speaker_2" to an actual person is still your problem.
The deployment numbers are what make this a product story rather than a paper. Baseten added same-day serving and reports a single RTX PRO 6000 holding more than 500 concurrent hour-long diarization streams at the low-latency profile, about 190 when transcription is attached, at roughly a cent per audio hour. Its transcription preset pipes speaker-activity vectors straight into NVIDIA's 600-million-parameter Parakeet recogniser, which is how two people talking over each other get transcribed instead of collapsed into one garbled turn. The bigger unlock is that speaker activity, tracked every 10 milliseconds, doubles as a signal for voice agents: knowing when one specific person has stopped talking is what separates an agent that waits politely from one that answers background chatter.
We covered the closest thing earlier this month — Meta ships Muse Voice Transcribe, its first real-time audio model — a closed, API-only real-time audio system. Open weights with published error rates is the version teams can actually build on.
What to watch: whether the open diarizers hold up in the messy acoustic conditions — far-field mics, heavy crosstalk, non-English meetings — that the benchmark set only partly covers. NVIDIA trained on roughly 10,000 hours of real conversations plus 82,611 hours of simulated multi-talker audio; the failure modes in real rooms are usually the ones nobody simulated.
If your meeting notes attribute the wrong decision to the wrong person, is that a bug worth paying to fix — or would you rather the tool just stopped guessing? Tell us in the comments.
Sources: NVIDIA — Know Who Spoke When: Build Real-Time, Multi-Speaker AI with Nemotron 3 Diarization · Baseten — NVIDIA Nemotron 3 Diarization: real-time speaker labels at a cent per audio hour · Unite.AI — NVIDIA Releases Nemotron 3 Diarization Open-Weight Speaker Model · AI Midday — Meta ships Muse Voice Transcribe, its first real-time audio model
