Deep Dive — Marin's 535B run puts open model training on livestream

A 535-billion-parameter model is training right now with its code, data mixture, loss curves, and even its failures posted in public as they happen. The Marin project started the run this week and wants the world to watch the experiment unfold rather than read about it in a retrospective paper three months from now. It is the starkest test yet of a question the open-weights community has never fully answered: can transparency at frontier scale be real, or is it just branding?

We covered the launch in this morning's brief — Marin opens a 535B-model training run in live public view. The deep dive is the part that brief could not hold: the architecture and engineering decisions behind the run, why "open development" is a materially different claim from open weights, and the specific reasons to be skeptical of how far the openness really goes.

What is actually running

Marin 535B-A23B is a mixture-of-experts model with roughly 535 billion total parameters, of which about 23 billion are active for every token. It has 48 transformer blocks, each pairing an attention branch with a sparse expert branch containing 384 routed experts — the router picks eight per token, while two shared experts run densely on every token as a stabilizing backbone. The announced plan calls for 18.75 trillion tokens across 11 NVIDIA GB200 NVL72 rack systems (792 Blackwell GPUs) over roughly three months, with the token budget split 80/20 between pretraining and midtraining; the full effort is pegged at about 2.7×10²⁴ FLOPs.

The run was announced publicly by Stanford professor Percy Liang, who is also founder of the reasoning lab Simile, and it has drawn explicit endorsement from Andrew Ng, who called the project "a precious demonstration of openness in model training" in the fight to defend openness in AI. The core novelty is the operating record. Before there is a finished model at all, Marin has published the voyage plan in a GitHub issue, pinned the code to a specific commit, opened a Weights & Biases dashboard tracking live training loss and per-domain data composition, and logged the failures alongside the wins.

That last part is unusual enough to matter. The hero issue documents an earlier one-rack gate test that hit 250,691 tokens per second of median throughput, but it also documents the configurations that failed — a direct fixed-expert-cell design that blew past its XLA memory estimate, an out-of-memory on a 123 GiB CUDA allocation, a six-expert receiver bank that also ran out of memory. Teams rarely publish the negative results at this level of detail while a multi-week run is still on the road.

The scaling ladder is the part worth stealing

The most transferable piece of the project is not the 535B model. It is the scaling ladder Marin trained before committing to the hero run. The team ran the same recipe at four smaller sizes — from a 1.6-billion-parameter MoE (61 million active, 48 billion tokens) up to a 27.7-billion-parameter rung (1.2 billion active, 926 billion tokens) — for about 1 percent of the kickoff run's compute.

The ladder does two jobs. It is a loss forecast, letting the team predict where the 535B run's loss should sit at every training step and compare it against the measured curve; if the hero deviates from the projection, that is the early-warning signal that something is wrong before months of compute are wasted. And it is a debugging rig. Marin says an earlier ladder surfaced gradient-norm growth above four over longer token horizons, which is what led the team to adopt logit z-loss regularization to stop high-batch configurations from diverging.

The honesty here is twofold. The largest ladder rung, the 27.7B model, crashed at roughly 81 percent of its planned trajectory and was not resumed — the issue records that plainly rather than polishing over it. And the project is upfront that the ladder does not guarantee the extrapolation: predicting what a 535B model does from four smaller runs is one of the experiment's central uncertainties. A short one-rack gate test does not establish the eventual token-drop rate at full scale, and the projection method itself (a power-law fit with a fixed loss asymptote of 1.5, extrapolating the missing last chunk of the ladder from a 60–80 percent window) is a set of modeling choices, not measured fact.

Transparency, not open weights

The framing matters because "open development" is a different promise from "open weights." Releasing final weights tells you nothing about how they were trained — the data mixture, the decisions, the dead ends. Marin's stated lifecycle preregisters an experiment in an issue, ties it to reviewable code, exposes it through public telemetry, and returns the analysis, including the failures, to the same thread.

For the 535B run that record is remarkably complete: the model spec, the pinned code commit, a frozen data-provenance table, deduplication and decontamination reports (the team removed about 2.5 trillion tokens from the mixture through fuzzy dedup and decontamination against common benchmarks), the expert-parallel transport design, and a live tracking board. As the audit-focused review at TeqVolt put it, the question the project forces is not "can I download the weights?" but "can I audit how this system was built and where it might fail?"

That is a shift with real stakes for the open-weights conversation this site has been tracking. The strategic logic behind many open releases has been that downloadable weights are advertising for the hardware that runs them and the ecosystem that adopts them — a dynamic we traced when Alibaba released its 2.4-trillion-parameter Qwen3.8 as open weights, and one that has pulled Nvidia fully into the open-model lane, as we covered in the analysis of Nvidia's $6 billion plan to meet China at open weights. Marin is trying to move the competition one step earlier in the pipeline, from who releases the best weights to who lets you audit the training itself. For readers still placing the pieces, our explainer on what open-weight models actually are makes the baseline clear: weights being downloadable is the beginning of openness, not the end.

The hard engineering problem: token dropping

Beneath the transparency narrative is a genuinely tricky distributed-systems problem, and Marin has been unusually candid about it. With 384 routed experts spread across an expert-parallel domain of 64 GPUs per rack, getting the right tokens to the right experts without floods of dropped assignments is the bottleneck.

Marin's answer is a hand-rolled fixed pooled-wave all-to-all implementation written for JAX and XLA. Instead of one variable-size buffer per expert (which created prohibitive memory and dynamic-shape traffic), each sender builds one fixed pool per destination GPU and transmits in three static waves carrying expert identifiers in-band, so no separate metadata exchange is needed. A sender-capacity factor of 1.10 and a receiver capacity factor of 1.15 bound the buffers, and a latent projection halves the activation width from 6,144 to 3,072 before it crosses the network — a LatentMoE compression that roughly halves the transport bytes.

Token dropping is the thing that threatens the whole plan. Earlier implementations saw dropping jump from about 7 percent at 4K context to roughly 40 percent at 65K. The new pooled-wave design drops about 3 percent at 4K, which is why the run deliberately starts at a 4,096-token sequence length: twice as many sequences per batch means better expert balance. But Marin expects the drop rate to climb sharply again when context extends. The plan calls for staged extensions to 8K, then 65K, then a target of 262K — all conditional on whether a one-to-two-day early cooldown branch around 10 to 20 days in proves the routing stays stable. If dropping stays too high, the fallbacks are a dropless ragged implementation, a bigger capacity factor with its memory cost, or sequence-level balancing that risks making experts respecialize mid-run.

What the skeptics get right

The case for treating the run with cold eyes is strong, and it is not that the transparency is fake. It is that transparency and reproducibility are different standards, and the project's own record exposes the gap.

Start with the numbers. The announcement says 18.75 trillion tokens; the GitHub hero issue and the live launcher say roughly 18.0 trillion (390,000-plus steps at a batch of 11,264 sequences). The 18.75 trillion figure looks like the reference allocation of the Harrier data mixture design — 15 trillion of pretraining plus 3.75 trillion of cooldown — while the configured run sums to about 18.0 trillion. That is a difference of roughly 4 percent, too large to dismiss as rounding, and the live config's target-budget fields are null. It is, in an odd way, a feature of the transparency that an outsider can find the discrepancy — but it is also evidence that the project's own public materials are not fully reconciled.

More fundamental: the data has an access problem. An unresolved comment in the hero issue reported that the referenced Harrier data store was not anonymously downloadable from its advertised object-storage endpoint as of late August. The provenance metadata is valuable, but if an independent team cannot retrieve the inputs, the "open" record is easier to inspect than to reproduce. The compute is an even harder boundary: eleven GB200 NVL72 racks with custom kernels and a multi-month schedule are simply not a reproduction target for almost anyone. The realistic test is whether outsiders can rerun the smaller ladder stages and regenerate the analysis, and that is a lower bar than the marketing implies.

Then there is the temptation to conflate scale with capability. A 535B total number is not a benchmark result. As of now there are no final weights, no post-training evaluation, and the projected "dropless Paloma macro-loss of about 2.04" is a conditional scaling-law extrapolation, not a score. The precedent for at least some of this goodwill is real — BigScience's BLOOM 176B exposed its training metrics and chronicles, EleutherAI's Pythia released sixteen models with shared checkpoint structure, and AI2's OLMo 2 opened its data and recipe. Marin's contribution is pushing that style of development to a 535-billion-parameter cross-rack scale for the first time, and treating failures as research output. But scale is not completion, and live-streaming a training run is not the same as demonstrating a finished model works as advertised.

What to watch next

The run's real test comes over the next month. Four things will decide the project's lasting weight, in rough order of importance. First, whether the schedule survives contact with hardware reality — Marin's own delay plan says that if failures eat more than roughly a quarter of the token budget, the default is to shorten the horizon, retune the linear learning-rate decay, and adjust the data mixture, so the announced plan is an operating target rather than an immutable specification. Second, whether the public record gets versioned cleanly as context length and data mixture change over the run, rather than leaving readers to reconcile drifting step counts and token totals from scattered comments. Third, whether the artifact layer — processed data, permanent checkpoints, evals — becomes retrievable or rebuildable with clear licenses. Fourth, whether the preregistered loss forecast actually lands where the final training loss ends up, deviations explained rather than hidden.

If in three months the failures, changes, and deviations are all still on the public record and the final model, evaluations, and post-training recipe are released against the same frozen timeline, Marin will have given the field something more durable than a benchmark table: a navigational chart for what a frontier-scale open run actually takes. If the run quietly stops updating, or the forecast quietly stops matching, the transparency bet will have produced the opposite of the community it claims to serve — evidence not that radical openness works, but that it was never the real constraint.

Is publishing a training run's failures mid-flight genuinely more auditable than open weights, or is it a credibility play that can't hold up under pressure? Tell us in the comments.

Sources: Lamjin Lab — Marin announcement · Marin hero-run issue #8435 (GitHub) · AlphaLab analysis · TeqVolt audit · HuggingNews