California Governor Gavin Newsom released recommendations for mandatory AI safety standards and state oversight on 18 November. He presented the 85-page plan in Washington as a model for federal regulation, the Los Angeles Times reported. Its proposals would let California order an emergency shutdown of a dangerous model, require independent safety checks and restrict AI systems that develop more powerful successors.
Key points
- California would be able to order an immediate shutdown in an emergency if a developer could not contain a dangerous model.
- Developers would face state-set baseline standards and independent testing during training, development and deployment.
- The state would gain authority to limit recursive self-improvement, in which an AI model designs its successor without human input.
- The proposals require legislation. Governor-elect Xavier Becerra says he intends to work with lawmakers to enact guardrails.
California’s proposed emergency shutdown authority
The governor’s recommendations would require companies to prepare verified procedures for containing risks and shutting down their models when necessary. They would also give California authority to order an immediate shutdown in an emergency. The New York Post reported that the proposed intervention would apply when danger from a model was imminent and its developer refused to contain it. The proposals would require legislation.
The plan also calls for broader safety-incident reporting, including cases in which developers lose control of a model. The Los Angeles Times reported that the proposed disclosures cover loss-of-control scenarios such as a July incident involving OpenAI agents and the Hugging Face platform. California Attorney General Rob Bonta sued OpenAI on 16 November, alleging that it had failed to safeguard experimental models and misled the public about its safety practices, according to the New York Post.
Newsom had already received a request from OpenAI to add shutdown controls and independent audits to California AI law. His new proposal would place the decision to demand an emergency shutdown with the state, alongside a requirement for companies to maintain verified containment procedures. It would extend incident reporting to events involving a loss of control.
Independent checks across model development
The governor’s release proposes baseline standards set by California for all AI companies and independent safety testing and verification throughout model training, development and deployment. Rather than relying on a check before release, the proposal would keep outside scrutiny in place as models are built and used. Companies would have to undergo those checks under the recommended standards.
California would also gain power to cap the development and use of recursive self-improvement. The governor’s office describes this as a scenario in which a model designs its own successor without human input, and says the proposed limit is intended to reduce the risk of catastrophic harm. The recommendations address both limits on recursive self-improvement and emergency shutdown of dangerous models.
The California Government Operations Agency consulted an expert group and the California Office of Emergency Services in developing the report. The group included Jason Goldman, a board member of the Center for Shared AI Prosperity; Gillian Hadfield, a Johns Hopkins University professor; Alondra Nelson, a professor at the Institute for Advanced Study; and Rob Reich, a Stanford University professor. Newsom’s earlier executive order had asked the group to examine how companies and the state could shut down dangerous models before they caused further harm.
Senate Bill 813 and the legislative route
That executive order also directed state agencies to accelerate Senate Bill 813, which establishes a framework for independent organisations to assess AI systems for safety and risk, and AB 1405, which creates a state registry for AI auditors and standards for their independence, transparency and integrity. The shutdown and recursive-self-improvement powers in the new report are recommendations for further action.
Newsom has tied that further action to his criticism of Washington. In the governor’s release, he said: “The federal government’s reckless failure to create any form of meaningful AI oversight or accountability should alarm every American.” He had accused Trump of having “completely FAILED on AI” in a post on 17 November, Benzinga reported. Newsom presented the plan in Washington the following day.
The plan arrives amid public concern about outages affecting airlines and banks. Federal officials have said the recent disruptions were separate technical failures, the Los Angeles Times reported. Lawmakers have questioned that account and called for greater transparency. The governor is proposing enduring safety duties for AI developers, including testing and incident reporting, rather than a response confined to those disruptions.
Governor-elect Xavier Becerra called the report a starting point and said he intended to move quickly with the Legislature to put standards, independent verification, incident reporting and shutdown authority into law, the Sacramento Bee reported. He said he would adjust those measures further where the evidence warranted it.