PerceptionBench: no frontier model breaks 60% on basic vision
Frontier models can hold their own on math and code, but Moonshot AI's new benchmark suggests most of them still can't reliably see — and what gets blamed as a reasoning failure may actually start at the eyeball.
Moonshot AI, the lab behind Kimi, released PerceptionBench, a benchmark that isolates visual perception from reasoning — and no frontier model clears 60 percent accuracy on it. The test breaks vision into ten atomic skill domains built from real-world errors: counting, localization, depth and 3D, fine-grained recognition, OCR, comparison, context integration, and hallucination detection, among others. Of the 16 models evaluated, GPT-5.6 Sol leads at 59.7 percent, followed by Kimi K3 at 58.5 percent, Claude Fable 5 at 57.2 percent, and Gemini 3.1 Pro at 56.2 percent. Open-weights models trail badly: Qwen3.5-397B-A17B scores 47.5 percent and GLM-4.6V just 32.5 percent.
The authors' central argument is that many errors labs chalk up to reasoning failures actually happen at the first step — misreading the image itself. PerceptionBench is designed to catch that: every question can be answered just by looking, with no outside knowledge or inference required. The team analyzed 42 open-source benchmarks, found their error profiles barely overlapped, and distilled them into the ten domains. From an internal pool of more than 17,000 verified questions, they publish 3,000 tasks, 60 percent of them traced to attributed model errors and the rest reformulated from augmented images.
The category-level results are the sharpest part. Hallucination is the weakest skill across the board: GPT-5.6 Sol, despite topping the overall ranking, scores just 26.9 percent on questions where the correct answer is "zero" — while the weaker Gemini 3.5 Flash lands among the best at 50.6 percent. Models with nearly identical aggregate scores diverge sharply by category, which suggests leaderboard totals are hiding very different failure profiles.
This is the second time Moonshot has pushed on the perception question: its earlier WorldVQA testing had the best model, Gemini 3 Pro, below 50 percent. The implication for the field is uncomfortable. If frontier models are quietly failing at basic seeing, then multimodal benchmarks that mix perception with reasoning are overstating capability, and agentic systems reading screenshots, maps, or camera feeds inherit those errors before any "thinking" begins. The fix may not be a longer chain of thought — it may be better vision, or at least knowing when vision is the weak link.
What to watch: whether the next generation of multimodal models actually closes the perception gap, or keeps masking it behind aggregate reasoning scores.
Which is the bigger bottleneck for multimodal agents — seeing or thinking? Tell us in the comments.
Sources: The Decoder · Moonshot AI PerceptionBench (GitHub) · MarkTechPost
