Who taught Waymo's cars to call the police, and on what evidence

San Francisco police booked two minors into the Juvenile Justice Center this week after a Waymo robotaxi flagged a loaded AR-style ghost gun riding in its back seat. The perception was correct. The part nobody can audit is what happened next: a machine classification became a 911 call, and the rule that connected the two has never been published.

What happened, in order

The call came in just before 4 a.m. on September 3, to the 800 block of 40th Avenue in the Richmond District, according to the San Francisco Police Department's statement. Officers conducted a "high-risk vehicle stop" — the phrase police use when they approach a car expecting a gun — and removed a boy and a girl, both juveniles, whose ages and names SFPD will not release. They recovered a loaded unserialized AR-style rifle, plus suspected marijuana and mace spray. Both were booked on firearm-possession charges and the investigation remains open.

Waymo's account is one sentence long. A spokesperson confirmed to the Los Angeles Times that the vehicle was Waymo's, and that the company pulled it over after detecting a "violation of our terms of service involving a firearm." Waymo did not answer follow-up questions from KRON4 about how that detection was made or who authorized the call.

Read the sequence as a chain of decisions. An in-cabin model classified an object. The ride was terminated. A report went to emergency services. Armed officers arrived. No human sat at the top of that chain and concluded a crime was in progress — the company's own statement describes the trigger as a policy violation, not an offense, and the escalation fused the two.

What Waymo has actually disclosed

Its help page on cameras and microphones is the closest thing to a public spec. Vehicles carry up to 29 cameras, some pointed inside. Listed purposes include keeping cars clean, finding lost items, checking that "in-car rules are being followed," and "promote safety and security." Support staff may review recorded video under certain circumstances and — the load-bearing clause — "in more urgent circumstances, Support may access live video during a trip." Waymo states it does not use facial recognition or other biometric identification to identify individuals. Mics stay off unless the rider is calling Rider Support or opts in.

The rider rules are where the terms-of-service framing lands. You must be 18 to ride alone; a teen-account program for ages 14–17 exists only in the Phoenix metro. No weapons of any kind, no drug or alcohol use inside the car regardless of local law. So two unaccompanied minors carrying a loaded rifle were already three rule breaches before anyone saw the gun.

What is disclosed nowhere: what confidence turns a detection into a stop, whether an algorithm or a remote operator decided on September 3, whether these referrals are logged and reviewed for error rates, and what recourse a rider has against a classification made about them after the fact. That is the same disclosure gap we tracked in the police-pilot world — Western Australia's live facial recognition trial scanned more than 130,000 people while its match-review and error characteristics stayed private. Volume public, accuracy undisclosed.

Scale is what converts this from a good-news item into a governance question. Waymo's fleet was around 3,000 vehicles with more than 20 million trips when California's utilities commission cleared its biggest expansion on August 14, authorizing paid driverless service across 18 counties from Sonoma to San Diego — most of the state's population. We covered the commercial side of that buildout in Waymo's Ojai-only rollout and its doubling tariff clock. The policing capability scaled with it, unmentioned.

Who wins and who loses

Waymo wins the cycle. A company carrying an NHTSA document request over a collision involving a child, a public fight with Tesla over whether a safety driver counts as autonomy, and a documented problem with riders blocking its cars just proved its vehicles can be trusted as witnesses. "The world's most trusted driver" is the tagline; this incident is a datapoint for it.

The durable winners are the departments already asking for footage. Waymo cars handed Los Angeles police usable video in a hit-and-run inquiry, and demonstrators in 2025 vandalized Waymos in LA over the suspicion the vehicles were functioning as a mobile surveillance grid. A car that calls 911 unprompted is better for law enforcement than one that waits for a records request — it turns a fleet from a data source into an informant.

The losers are riders, and last month showed the shape of it. In July, San Mateo police responded to a Waymo report of two 15-year-olds drinking and firing toy guns from the windows; it was Orbeez water beads. Video of that scene shows at least five officers and a police dog approaching a stopped car with weapons raised, and the department's social post read: "Parents do you know where your teens are? @waymo does!" Ghost guns are a real and escalating problem in California — state DOJ recovery figures climbed from 11 weapons in 2014 to more than 10,000 a year in both 2021 and 2022, the reason AB 1621 passed in 2022 to regulate unserialized precursor parts. But detections go wrong, and a wrong one here means armed officers and a high-risk stop. The only party who will ever know the classification was a mistake is the company that made it.

What skeptics say

Two objections cut the other way. First, nobody is demanding Waymo install in-cabin cameras and then look away from a loaded rifle. A human chauffeur would call it in; a remote operator watching a live feed is doing the minimum. Critics who want the cameras off must also want the hit-and-run footage off and the child-safety detection off. Privacy researcher Alessandro Acquisti of MIT Sloan framed the real risk precisely in NPR's July coverage: existing law already governs a carrier's duty to report or protect, and the harm begins when a company uses that duty "as a pretext for blanket, indiscriminate accumulation of identifiable data for unspecified future purposes."

Second, this is a story about a gun in a car. Waymo's perception didn't cause it; it decided whether anyone noticed.

Both are fair. Neither closes the disclosure gap. Getting it right once, in a low-frequency setting, is not evidence about error rates in a high-frequency one.

What to watch

The asks are narrow: whether Waymo publishes escalation criteria — which detection classes trigger a police call, at what confidence, confirmed by whom. Whether the CPUC or the DMV's AV permit conditions attach any reporting duty to law-enforcement referrals, as distinct from the traffic data they already collect. Whether a rider ever learns an automated system classified them, and how to contest it. And whether the next stop ends in a wrongful-escalation claim rather than a press release. The precedent being set now is that a robotaxi may judge a passenger armed, summon officers, and be thanked for it. That is a fine outcome once. At 3,000 vehicles it is a policy.

Waymo's camera caught a real gun and two teenagers went to juvenile hall — does a car that reports you to police earn your trust or cost it? Tell us in the comments.

Sources: San Francisco Police Department statement via KRON4 · Los Angeles Times · Waymo Help — cameras, microphones and related data processing · Waymo Rider Rules · NPR · Giffords Law Center — ghost guns in California