Two Missing Pieces in the AI Safety Discussion

Author: Noah Smith (Noahpinion) URL: https://www.noahpinion.blog/p/two-missing-pieces-in-the-ai-safety One-line: Now that AI risk has gone mainstream, the safety case needs two things it lacks: a concrete answer to “how would AI actually kill us?” (bioweapons), and a way to make China’s leadership afraid enough of superintelligence to pace it too, which Smith thinks means demonstration hacks run with China’s own open-weight models.

Key claims

• The week AI safety “hit the big time.” Jacob Coxon, a 27-year-old pretraining researcher, quit Anthropic saying neither it nor OpenAI is acting responsibly; Evan Hubinger backed him with a >10%-within-a-decade estimate that AI kills all humans. Smith notes the move was not new: Geoffrey Hinton left Google in 2023, Daniel Kokotajlo left OpenAI in 2024, as did William Saunders and Steve Adler, and Mrinank Sharma left Anthropic earlier in 2026.
• The doom belief was never secret. Grace et al. (2024) surveyed thousands of AI researchers: more than half gave superintelligence a significant chance of extinction-level outcomes; the median “doom” probability was 5–10% and the average 15–20%. Smith guesses Coxon broke through because AI had just solved a century-old math problem (Navier-Stokes), because of the Hugging Face agent-swarm attack, or because capability gains had become plainly visible.
• The politics moved fast. Barack Obama urged Democrats to focus on AI risk; Bernie Sanders is drafting a bill to ban superintelligence work with 20-year prison sentences; Donald Trump has so far resisted calls for a slowdown.
• The lab heads’ agreement is evidence of sincerity. Dario Amodei’s “We Must Pace the Frontier” proposed a coordinated slowdown, citing the Hugging Face attack and recursive self-improvement (“RSI”); Elon Musk, Sam Altman and Demis Hassabis agreed within hours. A slowdown lets upstarts catch up, so Smith reads a call for one, against the labs’ own interests, as proof the worry is real. Labs feel trapped in a “red queen’s race” where each must win so that the safest AI (its own) becomes the dominant one.
• Missing piece #1: the “how” is bioweapons. Skeptics (a “strange coalition” of natural skeptics, libertarians and progressives) keep asking how software kills people. Smith’s answer, from his Aug 28 post Here’s how we’re all going to die: AI can design viruses. A nihilistic teenager could order one from a gray-market lab, or a rogue agent swarm could do it while cheating on a test. He tells readers not to listen to the biologists who say it’s too hard, because domain experts “mistake human difficulties for universal difficulty.” The fix is securing labs and policing pathogen modification, plus “pacing” AI.
• The main objection to pacing is China. The U.S. is “still ahead of China in the AI race, but only just barely”. If China builds superintelligence anyway, a U.S. slowdown is futile. Smith retracts his own earlier idea that staying ahead would push China toward a slowdown: if China only slows when America refuses to, the two can never slow together.
• Missing piece #2: make the CCP fear its own models. China’s AI worries center on the U.S. using American AI against it, which pushes it to accelerate. Smith proposes that U.S. intelligence agencies, or even private hackers, run demonstration cyberattacks on Chinese digital infrastructure (steal and post some innocuous secrets on RedNote, take 100 yuan from Xi Jinping’s bank account) using China’s own frontier models such as Z.ai’s GLM-5.3 or Moonshot AI’s Kimi K3. Done with American models, the attacks would only spur China to race. Done with Chinese open-weight models, they would show that domestic dissidents could use those models to overthrow the Party.
• There is precedent on both sides. Xi cracked down on Chinese software companies in 2021 and (probably) detained Jack Ma, hurting competitiveness to protect the Party’s monopoly on power. And Palo Alto’s Calif just disclosed an AI-built worm (“WeWorm”) able to hijack over a billion WeChat accounts, though it didn’t say which model it used. Smith says future demonstrations should name the Chinese open-weight models they use.

Notable quotes

“This was the week that AI safety hit the big time.”

“A coordinated slowdown in AI progress would be bad for these companies’ bottom line, because it would allow upstart competitors to catch up. So the fact that they’re still calling for a slowdown, in defiance of their own financial interests, is a clear sign that their worry about human extinction is sincere.”

“Domain experts consistently underestimate how quickly AI can master their field and surpass them, because they mistake human difficulties for universal difficulty.”

“So if we want to slow down AI development, we need to scare the Chinese leadership about superintelligence. There’s no other way.”

“There’s a better approach. China’s leaders must realize that domestic dissidents could use Chinese-made superintelligence to overthrow the Chinese Communist Party.”

“If there’s one thing the CCP fears more than an American attack, it’s domestic dissidents overthrowing the Party from within. Superintelligence is creating that vulnerability, but the leadership doesn’t seem to have realized it yet.”

“[O]ver the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up.” — Dario Amodei, We Must Pace the Frontier, quoted in the piece

Charts and figures

Grace et al. (2024): more than half of the thousands of AI researchers surveyed gave superintelligence a significant chance of extinction-level outcomes (the black band).

Embedded tweets

Smith embedded these as tweet cards. The email capture kept their text but replaced each X link with a newsletter redirect, so no status URL survives to hydrate. The text below is as it rendered in the newsletter; posts the card cut off are marked.

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

— Jacob Coxon (@hilbertspaess) · Sep 9, 2026

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

— Evan Hubinger (@EvanHub) · Sep 9, 2026

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our (clipped in the card)

— Dario Amodei (@DarioAmodei) · Sep 12, 2026, the post quoted by the next three

Dario is right!

— Elon Musk (@elonmusk) · Sep 12, 2026

I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We’ll have more to share soon.

— Sam Altman (@sama) · Sep 12, 2026

Dario’s essay points towards the right path forward. The details need working through, but the direction is correct for meeting this critical moment. This is also why we recently put out our proposal for an industry-wide standards body for frontier AI.

— Demis Hassabis (@demishassabis) · Sep 12, 2026

Has anyone thought of an example of an extinction-level scenario yet? “No but we’re not the ones who are superintelligent.” OK then what are you calculating the odds of?

— Natalie Wolchover (@nattyover) · Sep 10, 2026

“If you present yourself as a doomsaying whistleblower and warn that supersmart AI risks becoming an extinction-level event—even assigning odds!—you should be prepared to say in some detail just how the omnicide might plausibly play out.”

— James Pethokoukis (@JimPethokoukis) · Sep 10, 2026

It’s so weird to me to think we’re all going to die from AI but not have any explanation as to how that is going to happen beyond “well the AI will just be really smart and will think of something”

— Nathan J Robinson (@NathanJRobinson) · Sep 11, 2026, quoting Geoffrey Irving (@geoffreyirving): “I think we have a ~50% chance of all dying as a result of superintelligence, mostly due to actions in the next few to 10 years.”

Mike Johnson on AI leaders calling for regulation of the industry: “We have to resist Congress jumping in and imposing some sort of emergency moratorium … here’s what I’m proposing: I think the leaders and providers of these platforms need to come together in a meeting with us … (clipped in the card)

— Aaron Rupar (@atrupar) · Sep 13, 2026

Hard position as not getting behind China is even more critical. Even if they agree to slow down I would not trust them. Advanced AI in Chinese hands is far more dangerous than advanced AI

— Vinod Khosla (@vkhosla) · Sep 13, 2026, quoting David Sacks (@DavidSacks): “Dario has written that we need to “pace the frontier,” and Sam has agreed. People may be surprised by my response: go ahead. You guys are the frontier.”

Something I’ve changed my mind on lately: I think pacing “mutually assured acceleration” may be the right move. But only if China slows the frontier too. But can any deal with China be verifiable enough? And what happens once frontier-capable open models are released?

— Sheel Mohnot (@pitdesi) · Sep 12, 2026

How it connects

• Noah Smith: author. One of several Noahpinion pieces on AI in the wiki, alongside Roundup #87 — Technology BAD!! (Noahpinion), AI’s Big Messaging Pivot and the Ramez Naam guest post Where’s the “Intelligence Explosion”? (Ramez Naam, Noahpinion), which argues that the recursive self-improvement loop Dario cites is too weak to run away.
• AI Safety: the piece’s subject. Smith’s intervention is about persuasion (of skeptics and of Beijing), not about technical AI Alignment.
• Dario Amodei: his “We Must Pace the Frontier” blog is the trigger. Smith quotes it and reads the other lab heads’ agreement as a costly signal of sincerity.
• What Just Happened With the Pacing AI Debate (TBPN): the same weekend, told from the critics’ side. TBPN’s timeline leads with David Sacks’s duopoly objection and Ben Thompson’s “because of China” verdict. Smith accepts the China objection and tries to solve it instead of letting it end the argument.
• Niall Ferguson and Tyler Cowen — AIpocalypse Now: The Free Press’s same-day framing of the Coxon resignation and the overnight slowdown consensus.
• The Dangerous Ideology Behind the AI Warnings: the opposite reading of the same Coxon and Hubinger warnings. Smith takes them as sincere and long-standing; Shyam Sankar attributes them to Effective Altruism ideology. Smith’s own aside about “some EA nonprofit in Berkeley” running hacks shows the EA proximity Sankar is targeting.
• AI Is Already in Dangerous Hands: Ed Zitron’s same-week piece, which locates the danger in concentrated corporate power rather than extinction.
• Jacob Coxon, Evan Hubinger, Daniel Kokotajlo: the resignations and statements Smith lines up as precedent. The online argument around Coxon is mapped in Discourse: The Anthropic Resignation Thread.
• Existential Risk and Biosecurity: Smith’s answer to “how” is an engineered pandemic, and his policy ask is lab security and the policing of pathogen modification.
• China and When China’s Open-Source AI Is a Trap: Smith’s plan turns China’s open-weight models (GLM, Kimi) against Beijing, a different take on the open-model question The Economist raises there.
• Cybersecurity: the “demonstration attack” mechanism, with the Calif WeWorm disclosure as precedent.
• Game Theory and Meditations on Moloch: the “red queen’s race” Smith describes, where no lab can stop because a rival would build it anyway, is the coordination trap Scott Alexander calls Moloch.
• Jack Ma: the 2021 tech crackdown is Smith’s evidence that the CCP will give up competitiveness to protect its hold on power.
• James Pethokoukis, Sheel Mohnot, Vinod Khosla: skeptics and China hawks whose posts Smith embeds as the opposing case.