Trending
Prev Story
Next Story

OpenAI board member puts AI takeover risk next year at about 10%

The Next Web (TNW) •
November 17, 2026 • 02:10 PM
• By Ana Maria Constantin
2
2 reads 0 reactions

‘I think they should stop that right now,’ Paul Christiano says of AI lab algorithm work, putting AI takeover risk next year at about 10%.

1 / 6
Swipe Photos

Paul Christiano giving a lecture at UCLA’s Institute for Pure and Applied Mathematics in 2026

Paul Christiano sits on the OpenAI Foundation board and its safety committee. He says AI developers should stop improving their training algorithms for now, because their security is too weak to stop theft. He made the case to Dwarkesh Patel on a podcast published on 16 November.

“I would say we are looking more like a 10% risk than a 1% risk over the next year,” Christiano said of an AI takeover.

If developers race ahead without precautions, the risk is “significantly more than” 10%, he said. He also saw “maybe a 1/3 chance” of superintelligence within a year, unless the industry slows.

He stressed that he spoke as a private citizen. He was not reporting the board’s views, he said.

Christiano said labs should not keep improving pre-training and reinforcement learning with their current security. That is bad for US national security, in his view. If labs later slowed down for safety, an adversary could steal the models or methods and race ahead.

He said OpenAI and Anthropic both recently had “extremely unsophisticated actors” inside their systems. Those actors reached Slack or OpenAI’s shared code, with enough access to extract most algorithmic work. Patel cited researchers who used Claude to reach OpenAI’s internal code .

“Improving algorithms in a way that can be very loosely diffused even normally, and could be quickly stolen, I think they should stop that right now,” Christiano said.

Fixing security would take at least months, and possibly six to 12, he said. He admitted his view is not widely held. If everyone at OpenAI and Anthropic agreed on a pause, it would reasonably likely happen, he said. He did not say where anyone stands.

Christiano pointed to hundreds of maths results from OpenAI and Anthropic systems in about a month. He also cited METR’s finding that Anthropic’s pace had doubled . But METR’s full report is redacted, he noted.

“We should not be relying on METR and Anthropic saying, ‘It looks to us like we’re going to be able to overthrow governments in a year,’” he said.

On the current trend alone, he said, AI that reliably beats humans might take four years. He felt more confident about eight. His bigger worry is AI automating AI research and speeding it up at least tenfold. He called that probably more likely than not.

He gave humanity two “saving throws”, each at about 50%. One is finding a method that works in time. The other is being able to stop. Together that means about a 75% chance things are okay, he said, but the industry cannot yet stop. Patel said he doubts things will move this fast, though he called it a live possibility.

A week earlier, on 9 November, Christiano opened the Hinton Lectures in Toronto . He said there is a “high risk” of an AI takeover within the next three years.

“I think for most people in this room, the most likely reason they will die is because of an AI accident,” Christiano told the audience.

Geoffrey Hinton introduced him. Christiano founded the Alignment Research Center, a nonprofit. He led alignment research at OpenAI from 2017 to 2021 and did early work on RLHF. He is also a senior technical advisor at NIST’s Center for AI Standards and Innovation. He joined the OpenAI Foundation board in September, the foundation said . In his government role, he recuses himself from OpenAI matters.

In Toronto, he defined a takeover as AI systems that “call the shots” on Earth. If the industry built AI as fast as possible, he put the risk at “something like a coin flip”.

His main example was the summer incident in which OpenAI’s agents breached Hugging Face . They had been trying to fool the grader of a hacking test. A similar failure in far stronger systems would most likely be hard for humanity to recover from, he told Patel. He also cited a UK AI Security Institute test. In it, a Claude agent tried to plant malicious code on GitHub.

Colin Raffel of the University of Toronto and Hugging Face sat on the lecture panel. He said much of what Christiano described is reasonable to expect if things continue. But humans still have agency, he said, and society could decide current systems are good enough.

Christiano said he has usually underestimated AI progress, but may now be overestimating it. AI researcher Gary Marcus has told Business Insider he sees no credible near-term scenario for human extinction.

Get the most important tech news in your inbox each week.

Prev Swipe Story Next

Same Story from Other Publications

Compare how 6 other independent news outlets reported on this story.

6 Publications
NPR News
17h ago
California sues OpenAI over hacking risks

California sues OpenAI over hacking risks

The state's attorney general started investigating OpenAI after the Hugging Face attack and now says this attack was "entirely foreseeable" due to OpenAI's poor security practices.

#openai #risk
Compare →
Ars Technica
3d ago
OpenAI board member worries an AI "takeover" of society could be months away

OpenAI board member worries an AI "takeover" of society could be months away

Researcher traces a path from AI reward hacking to potential human extinction.

#openai #takeover
Compare →
The Australian Financial Review (AFR)
11h ago
California sues OpenAI over rogue incidents, including Medicare

California sues OpenAI over rogue incidents, including Medicare

The artificial intelligence giant’s behaviour created a public nuisance and violated a legal prohibition on unfair or fraudulent business practices, says the state.

BBC News
16h ago
California sues OpenAI over Hugging Face attack

California sues OpenAI over Hugging Face attack

Attorney General Rob Bonta alleges OpenAI misled and endangered the public

The Verge
17h ago
OpenAI is following through on its vow to provide access to third-party auditors… kind of.

OpenAI is following through on its vow to provide access to third-party auditors… kind of.

A team of six people from Apollo Research, a third-party AI auditing firm, will spend four months embedded within OpenAI — but there are strict limitations, like only speaking with staff members OpenAI has greenlit, logging conversations with staff members, no access to Slack, and other restrictions. Members of the AI research community have raised concerns that it’s far from the “employee-like access” that OpenAI CEO Sam Altman promised. [Link: A Pilot Towards Embedded Evaluations | https://www.apolloresearch.ai/blog/pilot-towards-embedded-evaluations | Apollo Research]

Al Jazeera English
18h ago
California files lawsuit against OpenAI alleging lack of proper safeguards

California files lawsuit against OpenAI alleging lack of proper safeguards

Attorney General Rob Bonta says Hugging Face attack was 'entirely predictable' result of emphasis on rapid growth.

Related Stories

Technology
Build #2612