Peking University's 591-run RL study finds uniform sampling wins

A Peking University group built the comparison platform, ran the experiments, and reported the boring answer: with everything else held constant, the data policy is not what moves the score. The sharper finding is that the benchmark summary you choose can flip which method looks best.

Peking University's Data-Centric AI group ran 591 reinforcement-learning experiments to test whether cleverer data policies beat plain uniform sampling, and could not find one that reliably does. The paper, DataFlex-RL, was posted to arXiv on 5 September with collaborators at UCAS, the Shanghai Institute for Advanced Algorithms and Zhongguancun Academy, landed second on Hugging Face's daily papers board with 164 upvotes, and reached Chinese AI media today, where it ran as the day's number-two item on BAAI's 智源 hub.

Start with what the thing actually is, because the release is arguably more useful than the result. DataFlex-RL is a plugin for verl, the open-source RL framework behind much of the current post-training work, and it turns three decisions researchers normally hard-code into a trainer — which rollouts contribute to the update, how heavily each sample or token counts, and which domain supplies the next batch — into configurable parts that read signals the training loop already produces: rewards, advantages, token log-probabilities, prompt groups. A scorer computes a number, an actuator converts it into an action, and both are registered by name and swapped in config. The stated payoff is that comparing two ideas stops meaning forking the trainer.

The controlled study is the payload, and it is a negative result. Holding everything else fixed — 13 configurations, 12 matched seeds, Qwen2.5-7B-Base, 12 mathematics, logic and science benchmarks — uniform GRPO lifted the domain-balanced average by 7.76 percentage points over the untrained checkpoint, from 42.01 to 49.77 by the authors' numbers. None of the eight rollout-selection or reweighting methods produced a paired 95% confidence interval that excludes zero against uniform sampling, and none of the three adaptive domain mixtures beat a fixed equal split at the same precision, moving the average by 0.45, 0.61 and 0.16 points. An extension to Llama-3.1-8B-Base, where training added 10.25 points, put the extra methods on the same score scale without a consistent winner.

Then there is the part that should worry anyone publishing these comparisons. Rescoring nine Qwen2.5-7B-Instruct runs under two summary rules — a domain-balanced 12-benchmark average and a math-heavy six-benchmark one that drops logic entirely — produced negatively correlated rankings, a coefficient of −0.33, while summaries that keep all 12 benchmarks largely agree. If the ordering of your methods depends on which benchmark bundle you happened to average, a good share of the "our data policy wins" literature is a reporting artifact rather than a training effect.

Two caveats worth keeping. Selection is enforced after rollouts are generated, through 0/1 weights on the policy loss, so filtering that throws away half the batch does not give back the generation cost. And the study covers two model families at 7B and 8B under GRPO only — narrow enough that a real effect at larger scale, or under a different RL recipe, would not show up here.

That is the shape of this result: a matched-seed, matched-budget comparison in which a family of published techniques collapses into the noise floor. We hit the same pattern in inference — plain LRU beat three smarter KV-cache policies on 68,000 real agent requests — and the discipline it argues for is the same one behind seven PhD students who trained a 7B model and left every log in: run the boring baseline at the same budget, then say what changed.

What to watch: whether the authors' negative result survives at 30B-plus scale and non-GRPO recipes, and whether RL papers start reporting matched-seed intervals as a default.

A data policy that only wins under one benchmark bundle — has it won anything at all? Tell us in the comments.

Sources: arXiv: DataFlex-RL · Hugging Face paper page · DataFlex-RL (GitHub) · 智源社区 / 新智元