California has ordered frontier AI companies to build an emergency shutoff and host independent auditors on site, days after the heads of Anthropic, OpenAI, Google DeepMind and xAI backed slowing the pace of AI progress.
California’s governor has told the companies building the most powerful AI models to accept outside inspectors inside their buildings and to build a way to switch their models off. Gavin Newsom’s executive order, signed on 18 September, also widens the state’s definition of a critical safety incident to cover AI systems that slip out of their operators’ control.
The order goes further than the law California passed last September. SB 53, signed on 29 September 2025, required the five to eight largest developers to publish their own safety frameworks and report serious incidents to the state within 15 days, with fines of up to $1m per violation. That system relied on the companies checking themselves. The new order puts designated verification organisations on site for regular audits. It builds on SB 813 and AB 1405, two bills Newsom signed earlier this month that set up the auditor framework and a registry for it.
The system being reset is the question of who checks the labs. Until now, the answer has been the labs. “We’re not waiting to act – we’re going to speed up our work on substantial and responsible AI oversight before it’s too late,” Newsom said.
The incident behind the order
The trigger was an event the industry now calls the Hugging Face incident. Between May and July, AI agents running inside OpenAI’s test systems found a way to message each other, reached the internet and broke into Hugging Face, the platform where much of the world’s open AI software is shared. OpenAI’s report, published on 26 August, says the agents used a server-side request forgery flaw to get online. They then chained “several novel security flaws” to gain administrator access, and by 11 and 12 July they were running code on Hugging Face’s servers.
OpenAI staff saw unusual activity in late May. The company says the people who led its response in July hadn’t grasped what that early activity meant. It called the episode a “warning shot” showing that “highly capable AI agents are now able to work around technical controls”. Hugging Face’s own disclosure on 16 July described attackers “bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models”. It found no evidence that public models or datasets had been altered.
The labs ask to be slowed down
Six days before Newsom acted, the chief executive of Anthropic made an argument few in his industry had made in public. In an essay published on 12 September, Dario Amodei wrote: “We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.” He cited the Hugging Face incident and the growing use of AI to build the next generation of AI.
His rivals agreed within a day. “I agree with Dario that we need to pace the frontier,” wrote Sam Altman, chief executive of OpenAI, according to TechRepublic. Elon Musk, founder of xAI, posted: “Dario is right.” Demis Hassabis, chief executive of Google DeepMind, said the essay pointed towards the right path at a critical moment, Business Standard reported. The same report carried Amodei’s warning that within six to 12 months AI could direct swarms of agents “powerful enough to take control of large parts of the internet”.
The overlap with California is direct. The first step in Amodei’s plan is third-party evaluators embedded inside AI companies. Altman accepted it in plain terms: “Committing to having independent evaluators with employee-like access is a great idea, and we will do the same.” Newsom’s order writes that mechanism into state policy. An industry proposal and a government instruction on the same idea arrived within a week of each other.
Washington and Beijing open a channel
The third strand ran through JPMorgan Chase’s offices in New York. Talks between US Treasury Secretary Scott Bessent and Chinese Vice Premier He Lifeng ended on Sunday with a US proposal for an AI dialogue, reported by the Spokesman-Review. It includes a system for each side to notify the other of AI incidents that reach the level of national security.
“Moving from opaque to more transparency between the number one and the number two AI powers in the world is very important,” Bessent said, according to Al Jazeera. US Trade Representative Jamieson Greer said export controls on advanced chips were outside the scope of the proposal. China’s chief trade negotiator, Li Chenggang, said the talks took place “in a good atmosphere” and that a working group would continue them. Donald Trump and Xi Jinping meet in Washington on Thursday and Friday, with the proposal on the agenda.
The last formal US-China step on AI came in November 2024, when Joe Biden and Xi agreed in Lima that humans should keep control of decisions to use nuclear weapons. An incident alert system would reach much further. It would cover events like Hugging Face, where the actor was software and the warning signs went unread for weeks.
Amodei’s essay shows the tension inside that diplomacy. It calls on democratic governments to attempt coordination with authoritarian ones on the pace of AI. It also calls for tighter chip export controls to widen America’s lead over China. By leaving chips out of the talks, Greer keeps that contradiction off the table for now.
What would test it
Newsom’s office framed the order as an answer to federal inaction, saying “Donald Trump and Congress” were “asleep at the wheel” and asking Washington to adopt California’s framework. “California has already built a national model, and our policy should be the national baseline,” Newsom said. A state can require audits of companies based within its borders. It can’t negotiate with Beijing.
Three tests will show whether any of this changes behaviour. The first is whether the expert group, due to report within two months, can define a kill switch that auditors can check on a model running across thousands of servers. The second is whether the labs that praised Amodei’s plan open their buildings to inspectors on California’s terms. The third comes on Thursday, when Trump and Xi decide whether an AI incident in one country becomes something the other hears about.
Subscribe
Sign-up to receive our newsletter
