Copilot made Okta's engineers faster but not more productive
A year-long field study inside Okta finds the individual gains are real and the organisational ones are not there. Plus: the Jev team ships a code reviewer built on the assumption that nobody reads a 230-file diff.
A peer-reviewed field study inside Okta found that GitHub Copilot cut engineers' working hours and sharply lifted their motivation and perceived skill — while producing no statistically significant change in the code the company shipped. The paper, published in Communications of the ACM on September 17, followed the identity-and-access company's own Copilot rollout rather than a lab trial: 261 engineers who received licences between July and September 2024 were invited to a Qualtrics survey between November 18 and December 20, 2024, yielding 97 responses, 95 of them linkable to company engineering metrics, and 74 complete enough for before-and-after analysis. Output was measured as monthly pull request counts and lines of code, with April–June 2024 as the pre-Copilot baseline against September–November 2024 after.
The reported time savings are specific rather than vague. Average weekly working hours fell from 44.77 to 40.83, with the largest single savings in testing at 1.56 hours a week and coding at 1.06. Context shifting, documentation, coding, testing and total hours all dropped with statistically significant results and large effect sizes; motivation and perceived skill rose with rank-biserial effect sizes of 0.95 and 1.00. The output side is where it goes flat. Monthly PR count and lines of code showed no significant improvement, and neither did the metrics after the authors filtered out very large diffs they judged to be configuration or refactoring rather than hand-authored code. Only the number of PR reviews performed rose significantly. Monitoring of the same participants from December 2024 through May 2025 also found no improvement, which is the part that undercuts the obvious defence that the gains just needed more time.
Two details make this harder to wave away. The first is the authorship: the co-author is Todd McKinnon, Okta's cofounder and chief executive, writing about his own company's deployment and its aftermath — the paper records that Okta responded by removing management approval for Copilot seats, sharing usage best practices and embedding AI impact measurement into its regular engineering process. The second is the honesty about what the numbers can bear. Engineers who answered the survey used Copilot significantly more than licence holders who did not, so the authors say their findings may read more optimistically than the company-wide reality. The sample is one company, 74 people, and the authors concede some null results may reflect insufficient statistical power rather than genuine absence of effect.
Our take: the study's real target is not Copilot but the assumption that an individual tool's time savings compound automatically into a company's output. The authors call the gap organisational resistance — the communication, decision-waiting and process overhead that ate just over half of participants' weeks and did not move at all. Cutting 1.56 hours from testing does nothing for throughput if the finished work then queues for review and sign-off. That is also why the CEO is on the byline: this is a management finding wearing an engineering paper's clothes. As we covered this month, AI tutoring matched human tutors for an hour — at 1/918th the cost, the pattern keeps repeating — the tool-level win is easy to demonstrate, and the system-level one is not.
Jev's makers have released an open-source code reviewer whose whole design premise is that a human cannot meaningfully review what a coding agent just produced. Jev-Code-Reviewer, published on GitHub under MIT by egma-ai, sorts every change in a pull request into P0, P1 or P2 and shows only the P0 set by default, renders the diffs as natural-language explanations with the original code one toggle away, and runs entirely on your own machine — it posts nothing to GitHub. The stated motivation is blunt: agent-generated pull requests with hundreds of changed files get merged on trust because comprehension is no longer realistic, so the tool optimises human attention rather than diff fidelity. Two caveats are worth knowing before you install it: your code does leave your machine, since analysis sends the changed code and surrounding context to TypeSafe and OpenAI, and coverage is capped at the first 12 change units in path order by default. It is an early project — 14 stars, 10 commits — but it is a clean articulation of the year's most underrated problem: generation got cheap, verification did not.
What to watch: whether other enterprises publish the same experiment, and whether the efficiency-throughput gap survives agentic coding tools — the study tested autocomplete and chat, not the agents that now write most of the diff.
Has an AI coding tool actually increased what your team ships, or just how quickly the work piles up before review? Tell us in the comments.
Sources: Communications of the ACM — Beyond the Hype: The Efficiency-Throughput Gap with GitHub Copilot · ACM Digital Library (DOI 10.1145/3797488) · Hacker News discussion · Jev-Code-Reviewer (GitHub)
