California Regulates AI Risks: What the New Bill Entails.

According to Vox: When it comes to artificial intelligence, California acts as a legislative leader for the world. It is the largest state in the US by population and a significant center of innovation in the AI sector, housing 32 of the 50 leading companies in this field. Its scale and influence have allowed California to set the rules for other states on issues of ecology, labor rights, and consumer protection, and recently — in the realm of artificial intelligence. After a failed attempt to pass a federal moratorium on state-level AI regulation in July, California legislators see a limited window of opportunity to draft rules that could impact the entire country.

This week, the California State Assembly plans to vote on Bill SB 53, which will require developers of powerful 'frontline' AI models to be transparent in their operations. This category includes generative systems such as OpenAI's ChatGPT, Google's Gemini, xAI's Grok, and Anthropic's Claude, which require large amounts of data and computational power. The bill, which has already passed the Senate, must be approved by the Assembly before being sent to the governor for signing or veto.

Artificial intelligence can bring significant benefits, but as highlighted in the bill, it also poses risks. Among various threats such as job loss and bias, SB 53 focuses on potential 'catastrophic risks' from AI. These could include attacks using biological weapons, where AI becomes a tool, or cyberattacks capable of seriously damaging critical infrastructure. Such risks threaten human civilization at local, national, and global levels, raising concerns.

Currently, discussions are ongoing about what constitutes a catastrophic risk, but SB 53 defines it as a 'predictable and material risk' event resulting in more than 50 casualties or causing damages exceeding 1 billion dollars, provided that the frontline model plays a significant role. How culpability will be determined in practice is a question for the courts. Identifying catastrophic risk in legislation is no easy task, but it could help prevent both immediate and long-term consequences.

Explanation of SB 53

Bill SB 53 is the third at the state level aimed at regulating catastrophic AI risks. Previously, there was California's SB 1047, which was vetoed by the governor, and New York's RAISE law, which is currently awaiting Governor Kathy Hochul's approval.

Introduced by Senator Scott Wiener in February, SB 53 requires companies dealing with frontline AI to develop safety systems that clearly outline methods for mitigating catastrophic risks. Before deploying their models, companies are obligated to publish safety and protection reports. In the event of critical safety incidents, companies will have 15 days to notify the California Office of Emergency Services. The bill also provides protection for whistleblowers who report dangerous models that may lead to catastrophic risks. SB 53 aims to hold companies publicly accountable for their commitments to AI safety, with possible financial penalties of up to $1 million for violations.

In many aspects, SB 53 is a successor to SB 1047, also introduced by Wiener. Both cover models that are created using 10^26 FLOPS — this significant computational power serves as a threshold for risks.

“The science of making AI safe is rapidly evolving, and it is currently difficult for legislators to write clear technical rules for how companies should manage safety,” said Thomas Woodside, co-founder of the Secure AI Project, an advocacy group working to reduce extreme AI risks, in an email.

The logic behind SB 53 is to adapt the framework according to progress in the AI field. The bill gives the California Attorney General the power to change the definition of a large developer after January 1, 2027, considering changes in the AI sector. Supporters of the bill hope for its positive outcome if it is passed by the legislature. On the day that Governor Gavin Newsom vetoed SB 1047, he created a task force focusing solely on frontline models.

“I would say with about 75 percent confidence that SB 53 will be signed into law by the end of September,” said Dean Ball, a former AI policy advisor in the White House.

However, some industry organizations have expressed opposition, arguing that additional requirements will be burdensome due to already existing company commitments to safeguard against catastrophic harms. OpenAI is lobbying against it, while the Chamber of Progress group argues that the bill will create unnecessary bureaucratic hurdles that hinder innovation.

“These compliance costs are just the beginning,” commented Neil Chilson, AI policy director at the Wealth Institute.

In contrast, the company Anthropic holds a favorable view of the bill in its current form.

“The issue is not whether we need AI oversight, but whether we will develop it thoughtfully today or reactively tomorrow,” the company explained in a blog post.

The debates surrounding SB 53 reflect broader discussions about whether states or the federal government should regulate AI safety. Since most companies are based in California, state legislation is crucial for the entire country.

“The federal approach to transparency is far better than the alternative with many states,” said Matviy Mittelschadt, a fellow at the Cato Institute.

The question of whether a unified federal approach is even possible remains open.

Gap in Catastrophic Risks

The law's focus on catastrophic risks raises certain controversies. The very concept of catastrophic risks emerges from philosophical musings and quantitative analysis. They pertain to existential threats that could jeopardize the survival of humanity or significantly diminish our potential.

However, what exactly falls under the definition of catastrophic risk may depend on perspective. Discussions about AI-related risks are crucial for understanding ways to prevent such risks. Without agreed terminology, we cannot engage in truly constructive discussions about risks.