California Governor Gavin Newsom at a press conference on 14 May 2025.
Anthropic has told California Governor Gavin Newsom that the state should be able to ask a court to block AI models that pose a serious risk. The AI company made the case in a letter first reported by Politico .
The letter, dated 15 October, responds to an executive order Governor Gavin Newsom signed in September. The order set up a working group to suggest, within two months, how to make California’s AI safety laws stronger. One question is whether developers must be able to switch their models off.
An Anthropic spokesperson confirmed the letter was genuine, but Newsom’s office did not immediately respond.
“When a review finds a model poses a serious risk, the government should be able to ask a court to block it,” Anthropic wrote.
Under its proposal, independent evaluators would report risks to California’s attorney general. If the attorney general saw a near-term risk of catastrophic harm, they could ask a court to pause the model, Politico reported. A court order could stop its release, its use inside the company, or further work on it.
Anthropic also wants outside safety auditors to work inside leading AI companies. Firms would pay into a state-run fund rather than paying the auditors directly. Large developers would also publish risk reports at least every six months.
On kill switches, Anthropic agreed that every leading AI company should be able to shut down its models quickly.
However, it said a law focused only on shutdowns could give false comfort. Instead, it backed narrower steps, such as turning off a single AI agent or cutting off one customer’s access.
OpenAI sent its own recommendations to Newsom on 20 October, backing shutdown controls and in-house auditors, but not court orders to block risky models, according to Politico.
Anthropic is making a similar case in Washington. A draft federal bill it has circulated would let the Commerce Department ask a federal court to block a model’s release in narrow cases.
House lawmakers have also proposed an AI Kill Switch Act that would let officials order AI firms to shut down dangerous models.
Get the most important tech news in your inbox each week.
