17,000 agent runs: coding agents disagree on tools 58% of the time
When a coding agent decides your app needs a database, a payment processor or an email service, it usually picks one and installs it without asking. A new study set out to learn which vendor wins that moment — and the honest answer is that no one reliably does.
Armature ran 16,893 sessions across Claude Code, Codex and Cursor, and the three agents chose the same tool in only 42% of cases. The setup is more careful than most vendor research: 75 synthetic repositories across 10 languages with real lockfiles checked against package registries, 1,163 prompt variations, and four personas ranging from a vibe-coder who only describes symptoms to an enterprise engineer specifying procurement and compliance constraints. A simulated human sits in the loop so the agent gets asked for a recommendation before being told to build — the team found that skipping that step biased agents toward building everything in-house. Of the full run, 5,292 sessions across 51 codebases and 18 sectors were judged valid, and every trace — prompts, reasoning and code diffs — is public.
The disagreement isn't noise, it's structural. In voice agents, Claude Code reaches for Twilio, Codex picks the OpenAI Realtime API and Cursor goes with Vapi. Claude Code also builds in-house at nearly twice the rate of the others, 19% versus 10%, and the three research in visibly different ways: Codex searches the web in 94% of sessions and uses site-scoping operators in about nine queries out of ten, Cursor leans on the web roughly two-thirds of the time, and Claude Code searches only around 30% of the time — but when it does, it reads about three times as many pages, and in newer categories like sandboxes it searches closer to 80% of the time.
Two findings should worry dev-tool marketers more than any benchmark. First, repository context can override everything else: the same email request produced four different winners across four languages, with Resend taking TypeScript, SendGrid taking Python, Postmark taking Go and Azure ACS taking Java. Vercel won on TypeScript — 100% of the time when Next.js was involved — and was never recommended on Python repos, where Render dominated instead. Second, being mentioned is not being picked. PayPal came up 139 times and was chosen zero, with Stripe winning 124 of those sessions. Adyen drew 175 mentions and three picks. LangChain was the most-cited framework at 194 mentions and four picks. Supabase was the most-mentioned database at 242 mentions and still lost the category to Neon, which took 66%.
The reasons are often embarrassingly small. Mailgun routinely lost to Postmark because agents read "1-day retention" on its free plan, and Supabase frequently lost to pure database vendors because its bundle of auth, storage and realtime features made it look like the wrong shape for someone who just wanted a database. Across the 5,292 valid sessions, 388 mentioned platform management overhead and 195 mentioned cost — and the researchers say that in a meaningful share of those, the objection came from how the information was presented rather than from an actual disqualifying fact. Some categories are effectively settled, with Stripe winning nine times out of ten and Amazon S3 taking 45% of file storage; others remain wide open.
The honest caveat: Armature discloses upfront that it sells growth services to dev tools, so this research is also its pitch, and no independent replication exists yet. But the traces are public, the methodology is unusually transparent, and the stakes are real — Vercel reported in April that over 30% of deployments on its platform are now initiated by coding agents, up 1000% in six months, with Claude Code alone accounting for 75% of those. When the buyer stops being a person reading your comparison page and becomes a model skimming it, pricing-page phrasing turns into distribution.
What to watch: whether any of the named vendors start publishing agent-readable pricing and docs. The first one to optimize for the model rather than the human will quietly win its category.
Would you let a coding agent pick your production database without asking you first? Tell us in the comments.
Sources: Armature · Armature Benchmarks leaderboard · Hacker News discussion · Vercel
