Google rebuilt AX so a million agent tasks never hit Kubernetes' etcd

Share
Google rebuilt AX so a million agent tasks never hit Kubernetes' etcd

Google's open-source agent runtime just traded its Kubernetes plumbing for Redis — and the reasoning in the design doc is the most interesting part of the release.

Google restructured AX, its open-source runtime for AI agents, into a general-purpose orchestration layer for agentic tasks, and moved task state out of Kubernetes entirely. The rewrite shipped as version 0.3.0 on Sunday, a 151-file commit that describes a deliberate turn away from "a single CLI with an embedded Python harness" toward a system built for scale. The design notes are blunt about the trigger: storing millions of short-lived agent tasks as Kubernetes custom resources pushes etcd past its comfort zone — single-digit-gigabyte storage limits, write-rate bottlenecks, control-plane degradation. AX now keeps task state in Redis and uses Redis Streams as the work queue between its API server and a horizontally scaled pool of controllers, so adding replicas is how you add throughput. The old Python harness, its event log and the skill examples are gone.

What is left is four small primitives and three services. A task is the smallest unit of isolated execution — a sandbox with compute limits, environment variables, and mounts, cheap enough to create and throw away that an agent can spawn a tree of them to decompose its own work. A workspace makes environment setup declarative: Git repositories cloned at the right revision, MCP servers and skill registries the agent may call, plus a plain-language goal that a bootstrap agent executes on first boot to finish installing toolchains. A gateway is the network boundary, an explicit allowlist of hosts and ports the sandbox may reach. A model is not a model but a named configuration — provider, model identifier, parameters, and a Kubernetes secret holding the key — so rotating a credential or pinning a version is one change cluster-wide instead of a hunt through task definitions.

The detail worth pausing on: that first-boot workspace agent is built on Google's own Antigravity SDK, reading its key from the environment. Google's agent runtime uses an agent to set up the environments its agents run in.

The architectural argument underneath is sound, and it is the same one the industry has been circling all year. An agent is neither a microservice nor a batch job. It is a stateful session that sits idle for most of its life, wakes to run model-generated code in a burst, then goes quiet — and it can burn money in a loop if nobody is watching. Kubernetes was designed around a modest number of long-lived, replicated pods, and it stays the right tool for provisioning the machines underneath. It is the wrong scheduler for a swarm of sleeping processes. AX keeps Kubernetes for infrastructure and takes it off the critical path for agent work; the execution layer is Agent Substrate, the same project that reports 30x or more oversubscription by snapshotting idle actors out to storage and resuming them in under a second. We flagged Agent Substrate in August as one to watch — Open Source Radar — August 21: memory, vectors & red teams — and it is now load-bearing for Google's own runtime.

The telling detail is the interface. AX's command line deliberately mimics the Kubernetes tooling developers already know, down to verbs for applying, describing and watching resources, and it follows whichever cluster context you have active. The guts, meanwhile, abandon the Kubernetes idioms that break at agent scale. That is a company hedging well: keep the surface familiar, replace what does not hold. The honest caveat is AX's own README, which warns that core concepts, protocols and specifications are still being refined and that major breaking changes are coming before a stable release. The repo sits at roughly 2,900 stars, and Agent Substrate carries an explicit note that it is not an officially supported Google product.

What to watch: whether the Redis-backed control plane holds up at the task counts the docs claim, and whether the other runtimes arriving this year copy the design or keep betting that the standard control plane can absorb agents. Google has now put a version of the container-orchestration playbook on the table for agents — the question is whether it consolidates the way Kubernetes did.

Should the agent layer get its own control plane, or is this one more thing platform teams have to run? Tell us in the comments.

Sources: AX (GitHub) · AX design doc (GitHub) · AX core concepts · Agent Executor project site · Agent Substrate (GitHub) · Techzine · Hacker News discussion