Meta's distilled byte models break through the token ceiling
Two research results and one big European round: distillation gets a smaller vocabulary, physical AI gets open hardware, and embedded-device security gets unicorn money.
Meta FAIR found the way to break the distillation ceiling is to shrink the vocabulary to 256 bytes. A paper posted to arXiv on September 11 (authored with a University of Washington team) trained 1.28-billion-parameter byte-level students — models that predict raw bytes instead of tokenizer chunks — from larger teachers, and the scaling curves cross. Token models start ahead and plateau; byte models start behind and climb a steeper slope to a higher asymptote, with the distilled byte model predicted to beat the distilled token model by up to 4% average accuracy at the limit. Byte students also match token students on one-sixth of the data and cut logit storage roughly fivefold, since a 256-symbol output needs no top-k truncation. The paper's most useful finding for anyone running evals is unglamorous: identical validation BPB maps to different benchmark results depending on tokenizer and training objective, so a byte model with better loss can still lose on tasks — cross-tokenizer comparisons need a downstream calibration step. This lands one week after we covered the other side of distillation — Anthropic locks Claude's thinking blocks to kill API distillation.
Exein raised $270 million led by Headline at a $1.7 billion valuation, making it one of Europe's priciest cybersecurity bets of the year. The Financial Times reports the Rome-based firm sells runtime security for connected devices — cars, home routers, industrial controllers — and frames the pitch as fighting AI hackers with AI: autonomous attack tooling is moving faster than signature databases, so the defense has to be a model too. The round more than doubles the roughly €700 million valuation Exein got on a €100 million raise in December 2025, per European startup press. Europe keeps exporting this category — embedded and operational-technology security is where its industrial base gives startups a customer list US rivals can't copy.
Enactic shipped OpenArm, a fully open-source 7-DOF humanoid arm aimed at physical AI research. Every layer is published: CAD under CERN-OHL-S-2.0, firmware and control code under Apache-2.0, plus ROS 2 drivers, teleoperation stacks, MuJoCo and Isaac Lab simulations, and a manipulation dataset. The arm has backdrivable, force-feedback joints for contact-rich imitation learning, and it sells as a DIY kit or a pre-built unit. The most interesting piece is the OpenArm Cell — a standardized evaluation rig with fixed cameras, lighting, arm position, and a lift, so "model A beats model B" claims become reproducible across labs instead of anecdotes. If embodied AI keeps inheriting the benchmark-replication mess that plagued LLM evals, cheap verifiable infrastructure like this is the fix.
What to watch: whether the byte-model result holds on mixture-of-experts students — the authors flag sparsity as the next open axis.
Distillation's next lever might be a smaller vocabulary rather than a bigger teacher — does that change how you'd build a small model? Tell us in the comments.
Sources: arXiv — Breaking the Token Ceiling · AILog · Financial Times · Techmeme · OpenArm (GitHub) · OpenArm project site
