Marin opens a 535B-model training run in live public view
A Chinese lab just started what it bills as the largest fully-open large-model training run ever — and it wants the world to watch it happen in real time, not read about the results after the fact.
Marin has begun training a 535-billion-parameter MoE model with its code, data mixture, and live loss curves published as it goes. Called Marin 535B-A23B, the model activates about 23 billion of its total parameters per token and is being trained on 18.75 trillion tokens across 11 NVIDIA GB200 NVL72 racks — roughly 792 GPUs — over about three months. The plan splits the token budget 80/20 between pretraining and midtraining, and the full effort is pegged at around 2.7×10²⁴ FLOPs. The run is live as of this week: there are no final weights yet, which is exactly the point. Instead of releasing a clean technical report only after a successful run, Marin published its operating plan, engineering risks, scaling methodology, and contingency procedures in an open GitHub issue, with a Weights & Biases board tracking training loss and per-domain data composition along the way.
The openness is the story. Marin describes the effort as an experiment in doing frontier-scale training in public — and as a transparency argument for open-weights AI. Rather than have outsiders take the team's word that the model was trained cleanly on the announced mixture, the project exposes the evidence during the run: sampled documents, the exact pretraining mixture per domain, and forecasts that let anyone check whether the model is tracking its expected loss trajectory in real time. The venture has drawn prominent backing from the research community, including Andrew Ng, who publicly endorsed the approach, alongside senior figures in open-model development.
What makes this more than a publicity stunt is the engineering groundwork. Before launching the full run, Marin trained a four-rung scaling ladder from 1.6 billion to 27.7 billion parameters, costing about 1% of the main run's compute. That ladder doubles as both a loss forecast and an early-warning system — a material deviation in the full run can trigger an investigation before months of compute are wasted. It already paid off once: the ladder surfaced gradient-norm growth above four on longer horizons, which led the team to adopt logit z-loss regularization to keep high-batch configurations from diverging. Marin also documented a custom all-to-all implementation for the expert-parallel transport, and honestly reports the failed designs along the way, not just the wins.
There's a legitimate question of how much can be inferred from a partial record — extrapolating from small runs to a 535B model is one of the experiment's central uncertainties, and a short one-rack gate test doesn't establish the final token-drop rate. But the framing matters. Marin is betting that radical transparency becomes a credibility weapon in the open-weights race, letting the community audit a frontier-scale training run while it's happening instead of trusting a retrospective paper.
What to watch: whether Marin hits its reported ~250,000-tokens-per-second design throughput across all 11 racks over the coming weeks — the first real test of whether the roadmap survives at full scale.
Open training at this scale is a meaningful bet on radical transparency. If it works, does it set a new bar for how every major open model should be built — or will short-term loss wobbles give skeptics ammunition? Tell us in the comments.
