South Korea's national AI race faces benchmark-rigging claims
Seoul's sovereign AI project — a government-funded contest to crown one "national representative" foundation model — heads into its second-round verdict under a cloud: foreign firms allegedly offered to juice the entrants' benchmark scores.
Allegations that overseas companies offered to inflate benchmark scores for South Korea's government-backed AI foundation model race surfaced days before its second-round elimination. According to industry sources reported by The Hankyoreh, US-based training specialist AfterQuery and others approached Korean developers at ICML 2026 in Seoul last month, pitching data and post-training services that could lift performance metrics. The ministry plans to drop one of four finalists — LG AI Research, Upstage, SK Telecom, and Motif Technologies — as early as this week, with the evaluation weighting benchmarks at 40 points, expert review at 35, and public user testing at 25. The core suspicion, per the reporting, is "benchmaxxing": tuning models so aggressively toward known test data that they memorize exam questions rather than learn to solve problems — which would make inflated scores collapse in real-world use. All four companies deny the allegations, and the ministry insists benchmarks are only one input among expert and citizen-user evaluations, where a 200-person public panel tested the models August 8–11.
The stakes make this more than a Korean tempest. Motif 3 — from the smallest entrant, which joined via a supplementary call in February — scored highest on the Artificial Analysis Intelligence Index at 47 points, ahead of Upstage's Solar Open 2 (37), SK Telecom's A.X K2 (35), and LG's K-ExaOne 2.0 (31). Epoch AI just listed all four models as "Notable," lending the contest global visibility right as its integrity is questioned. The uncomfortable lesson here is that benchmark gaming is no longer a lab curiosity — it's a service US firms sell at machine-learning conferences, and a national program spending public money on a "sovereign" model has to design evaluations that can't be gamed, or its winner arrives pre-discredited.
A community-abliterated Qwen3.8-27B drops refusal rates from 64–99% to 0–6% while scoring almost identically on capability benchmarks. A new "uncensored" FP8 build of the open model Qwen3.8-27B, published on Hugging Face as red-team material, reportedly takes refusal on harmful-instruction sets (AdvBench, HarmBench, StrongREJECT) from near-total down to near-zero with thinking disabled — while MMLU moved just 84.3 → 84.7 and GSM8K 90.0 → 88.7, a shift under 1.3 points. If the capability-side numbers hold under independent testing, it's another data point for the Arditi single-direction result: refusal can be carved out of a model without dragging the rest of the network down with it. Caveats apply — the refusal figures come from a rule-based classifier the model card itself calls indicative rather than publication-grade, and no KL-divergence measurement against the base model is included.
What to watch: whether the ministry's elimination criteria — and the promised transparency around them — survive the week, and whether any participant's technical report is independently audited for test-data contamination.
If a national government is picking an AI "champion" on benchmarks that vendors can sell score-boosts for, what would make you trust the result? Tell us in the comments.
Sources: The Hankyoreh · AI Times · BigGo Finance · The Chosun Daily (English) · OrcaRouter Qwen3.8-27B-Uncensored-FP8 (Hugging Face) · Refusal projection repo (GitHub) · r/LocalLLaMA discussion
