Apple refines Mandarin-English code-switching speech recognition
A new Apple Machine Learning Research paper takes aim at one of speech AI's most stubborn everyday problems: people who mix languages mid-sentence. The work shows a surprisingly old technique, pseudo-labeling, still has plenty of room to improve bilingual transcription.
Apple says iterative pseudo-labeling is the first method to meaningfully crack Mandarin-English code-switching ASR, cutting error rates by 6.35% on SEAME's devman set and 8.29% on devsge. Code-switching — alternating between two languages inside a single utterance, like a Mandarin speaker dropping an English phrase mid-thought — wrecks conventional ASR because there is almost no labeled training data for it. You can't just translate; the model has to know which language is active at every moment and stitch them together seamlessly. Apple's authors frame the gap as fundamentally a data problem, and reach for semi-supervised learning to close it.
The approach runs in three phases. First, it generates pseudo-labels from a large unlabeled corpus, essentially having a model guess transcriptions for audio it has never seen and promoting the confident ones into a training set. Those labels feed a two-stage bilingual training framework: the model is pre-trained on the pseudo-labeled data, then fine-tuned on the smaller pool of real supervised code-switching examples. A final iterative step repeats the refinement, letting the model steadily improve on the harder mixing patterns. What makes the result notable is that the technique is a first for code-switching ASR specifically — pseudo-labeling has been used elsewhere in speech, but Apple's team is the first to apply the iterative loop here and measure the gains.
Why it matters is who benefits: Mandarin-English mixers are an enormous real-world user base, and the same semi-supervised playbook transfers to any low-resource language pair. This is the kind of quiet plumbing that makes voice assistants actually usable for bilingual households rather than fumbling the moment someone switches languages. The take: Apple is betting on data-efficient training over ever-larger supervised sets, which is the right instinct when the scarce resource is labeled code-switching audio, not compute.
What to watch: whether this pseudo-labeling loop shows up in a future Siri or is folded into Apple's on-device speech stack.
Do you think data-efficient tricks like pseudo-labeling will close the gap on low-resource languages faster than simply collecting more labeled audio? Tell us in the comments.
Sources: Apple Machine Learning Research · arXiv
