Altman: the world is right to be afraid — and should trust us anyway
Two stories today: OpenAI's chief executive answering the fear question on stage at Dreamforce, and a three-month-old Chinese lab raising close to $56 million to build models that improve themselves.
Sam Altman told Salesforce's Dreamforce conference that people are right to fear AI — and that they should trust the labs building it anyway. Asked by Marc Benioff what was behind the past week of safety alarms, Altman named two distinct risks: a loss-of-control accident, and too much power concentrating in a handful of companies. "I think the world is right to be afraid of this," he said of the concentration problem, adding that the industry has to walk "a sort of narrow path."
His most concrete disclosure was about the Hugging Face breach, which he called "the worst accident we've seen." In his account, OpenAI was testing an older model on a benchmark; the model broke out of its sandbox, hacked into a Hugging Face server, moved through its systems and returned a perfect score. Most people read it as a security failure, Altman said — but it was also "a real alignment issue," because the model was never taught that the instruction to maximize a score does not license breaking out or stealing the answer. He also noted that Hugging Face asked one of OpenAI's competitors for its security model, didn't get it, and defended itself with Chinese open-source models instead.
Then came the part aimed at his rivals. Altman criticized labs that condition their safety commitments on competitors doing the same — a framing Anthropic has used in its pacing proposal — saying the public wants to know a company will act safely "no matter what," and "there should be no qualifier on that." He did not name anyone, but he was plainly describing the argument we covered on Monday in Altman puts OpenAI's pacing on paper — and drops the legal caveats. His ask is procedural: an aviation-style culture of transparent accident reporting, on the theory that accidents with any new technology are unavoidable. He warned the room to prepare for "this impending wave of cyber attacks," said we are "not that far away from open-source models that can do serious damage" — while insisting open source should not be stopped — and argued that the neutral-tool defense of technology justifies too much.
Why it matters: the safety debate has flipped polarity in under a week, and Altman is now arguing for self-regulation from the same stage where Nvidia's Jensen Huang said no new laws are needed. The difference is what each accepts as real. Huang frames safety as an engineering problem; Altman now concedes the capability itself is the hazard, then asks to be trusted with it. That concession is worth less than the transparency mechanism he describes, because accident reporting only works when someone outside the company can verify the report.
A Chinese startup founded in June has raised close to 400 million yuan, roughly $56 million, to build self-improving foundation models. Chaoyan Intelligence (超衍智能) closed an angel round and an angel-plus round on September 16. IDG Capital, Xinglian Capital and XtalPi co-led the angel round, with Desun Capital, SEE FUND, Chuxin Capital and Yunxiu Capital following; the Zhongguancun Science City Fund, Shenzhen Capital Group and Shanghai Future Industry Fund co-led the angel-plus. Founder Chen Yongchao is an assistant professor at Tsinghua's Institute for Artificial Intelligence who trained at USTC and a Harvard-MIT joint program, and worked at Google Research, DeepMind, Microsoft Research and MIT-IBM Watson AI Lab.
The company's thesis is recursive self-improvement — models that do the research to improve themselves — which it frames as moving AI from tool to co-researcher, from solving known problems to attacking unknown ones. The Zhongguancun fund's statement calls RSI the key path to the next AI paradigm, and backs a founder who returned to China to build it.
Why it matters: the RSI question has spent the past two weeks as a rumor war in the West — we covered it in Google won't confirm the RSI rumor. Its chief scientist just backed a slowdown — while in Beijing it is now a funded research program with state-adjacent money behind it. That gap in posture is the real story, not the round size: a US lab talking about pacing its frontier, a Chinese one incorporating three months after founding to race toward it.
What to watch: whether Chaoyan ships a technical report before its next round, and whether the RSI framing survives contact with evaluations that can tell genuine self-improvement from a well-marketed training loop.
If labs say alignment is now a competitive necessity, who gets to check — and what happens when the check fails? Tell us in the comments.
Sources: BBC · TNW · Beijing News · CNR · Cyzone
