Skip to content
kwharrison13.comEssays, books and notes by Kyle Harrison

Two Missing Pieces in the AI Safety Discussion

From kwharrison13.com · Long read
Two Missing Pieces in the AI Safety Discussion
AuthorNoah Smith
PublishedSeptember 13, 2026
FormatArticle
OriginalRead the original

Now that AI risk has gone mainstream, the safety case needs two things it lacks: a concrete answer to “how would AI actually kill us?” (bioweapons), and a way to make China’s leadership afraid enough of superintelligence to pace it too, which Smith thinks means demonstration hacks run with China’s own open-weight models.

Two Missing Pieces in the AI Safety Discussion

Key claims

  • The week AI safety “hit the big time.” Jacob Coxon, a 27-year-old pretraining researcher, quit Anthropic saying neither it nor OpenAI is acting responsibly; Evan Hubinger backed him with a >10%-within-a-decade estimate that AI kills all humans. Smith notes the move was not new: Geoffrey Hinton left Google in 2023, Daniel Kokotajlo left OpenAI in 2024, as did William Saunders and Steve Adler, and Mrinank Sharma left Anthropic earlier in 2026.
  • The doom belief was never secret. Grace et al. (2024) surveyed thousands of AI researchers: more than half gave superintelligence a significant chance of extinction-level outcomes; the median “doom” probability was 5–10% and the average 15–20%. Smith guesses Coxon broke through because AI had just solved a century-old math problem (Navier-Stokes), because of the Hugging Face agent-swarm attack, or because capability gains had become plainly visible.
  • The politics moved fast. Barack Obama urged Democrats to focus on AI risk; Bernie Sanders is drafting a bill to ban superintelligence work with 20-year prison sentences; Donald Trump has so far resisted calls for a slowdown.
  • The lab heads’ agreement is evidence of sincerity. Dario Amodei’s “We Must Pace the Frontier” proposed a coordinated slowdown, citing the Hugging Face attack and recursive self-improvement (“RSI”); Elon Musk, Sam Altman and Demis Hassabis agreed within hours. A slowdown lets upstarts catch up, so Smith reads a call for one, against the labs’ own interests, as proof the worry is real. Labs feel trapped in a “red queen’s race” where each must win so that the safest AI (its own) becomes the dominant one.
  • Missing piece #1: the “how” is bioweapons. Skeptics (a “strange coalition” of natural skeptics, libertarians and progressives) keep asking how software kills people. Smith’s answer, from his Aug 28 post Here’s how we’re all going to die: AI can design viruses. A nihilistic teenager could order one from a gray-market lab, or a rogue agent swarm could do it while cheating on a test. He tells readers not to listen to the biologists who say it’s too hard, because domain experts “mistake human difficulties for universal difficulty.” The fix is securing labs and policing pathogen modification, plus “pacing” AI.
  • The main objection to pacing is China. The U.S. is “still ahead of China in the AI race, but only just barely”. If China builds superintelligence anyway, a U.S. slowdown is futile. Smith retracts his own earlier idea that staying ahead would push China toward a slowdown: if China only slows when America refuses to, the two can never slow together.
  • Missing piece #2: make the CCP fear its own models. China’s AI worries center on the U.S. using American AI against it, which pushes it to accelerate. Smith proposes that U.S. intelligence agencies, or even private hackers, run demonstration cyberattacks on Chinese digital infrastructure (steal and post some innocuous secrets on RedNote, take 100 yuan from Xi Jinping’s bank account) using China’s own frontier models such as Z.ai’s GLM-5.3 or Moonshot AI’s Kimi K3. Done with American models, the attacks would only spur China to race. Done with Chinese open-weight models, they would show that domestic dissidents could use those models to overthrow the Party.
  • There is precedent on both sides. Xi cracked down on Chinese software companies in 2021 and (probably) detained Jack Ma, hurting competitiveness to protect the Party’s monopoly on power. And Palo Alto’s Calif just disclosed an AI-built worm (“WeWorm”) able to hijack over a billion WeChat accounts, though it didn’t say which model it used. Smith says future demonstrations should name the Chinese open-weight models they use.

Notable quotes

“This was the week that AI safety hit the big time.”

“A coordinated slowdown in AI progress would be bad for these companies’ bottom line, because it would allow upstart competitors to catch up. So the fact that they’re still calling for a slowdown, in defiance of their own financial interests, is a clear sign that their worry about human extinction is sincere.”

“Domain experts consistently underestimate how quickly AI can master their field and surpass them, because they mistake human difficulties for universal difficulty.”

“So if we want to slow down AI development, we need to scare the Chinese leadership about superintelligence. There’s no other way.”

“There’s a better approach. China’s leaders must realize that domestic dissidents could use Chinese-made superintelligence to overthrow the Chinese Communist Party.”

“If there’s one thing the CCP fears more than an American attack, it’s domestic dissidents overthrowing the Party from within. Superintelligence is creating that vulnerability, but the leadership doesn’t seem to have realized it yet.”

“[O]ver the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up.” — Dario Amodei, We Must Pace the Frontier, quoted in the piece

Charts and figures

Grace et al. (2024): the probability each surveyed AI researcher gave to outcomes from "extremely good" to "extremely bad (e.g. human extinction)"

Grace et al. (2024): more than half of the thousands of AI researchers surveyed gave superintelligence a significant chance of extinction-level outcomes (the black band).

Embedded tweets

Smith embedded these as tweet cards. The email capture kept their text but replaced each X link with a newsletter redirect, so no status URL survives to hydrate. The text below is as it rendered in the newsletter; posts the card cut off are marked.

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

— Jacob Coxon (@hilbertspaess) · Sep 9, 2026

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

— Evan Hubinger (@EvanHub) · Sep 9, 2026

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our (clipped in the card)

— Dario Amodei (@DarioAmodei) · Sep 12, 2026, the post quoted by the next three

Dario is right!

— Elon Musk (@elonmusk) · Sep 12, 2026

I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We’ll have more to share soon.

— Sam Altman (@sama) · Sep 12, 2026

Dario’s essay points towards the right path forward. The details need working through, but the direction is correct for meeting this critical moment. This is also why we recently put out our proposal for an industry-wide standards body for frontier AI.

— Demis Hassabis (@demishassabis) · Sep 12, 2026

Has anyone thought of an example of an extinction-level scenario yet? “No but we’re not the ones who are superintelligent.” OK then what are you calculating the odds of?

— Natalie Wolchover (@nattyover) · Sep 10, 2026

“If you present yourself as a doomsaying whistleblower and warn that supersmart AI risks becoming an extinction-level event—even assigning odds!—you should be prepared to say in some detail just how the omnicide might plausibly play out.”

— James Pethokoukis (@JimPethokoukis) · Sep 10, 2026

It’s so weird to me to think we’re all going to die from AI but not have any explanation as to how that is going to happen beyond “well the AI will just be really smart and will think of something”

— Nathan J Robinson (@NathanJRobinson) · Sep 11, 2026, quoting Geoffrey Irving (@geoffreyirving): “I think we have a ~50% chance of all dying as a result of superintelligence, mostly due to actions in the next few to 10 years.”

Mike Johnson on AI leaders calling for regulation of the industry: “We have to resist Congress jumping in and imposing some sort of emergency moratorium … here’s what I’m proposing: I think the leaders and providers of these platforms need to come together in a meeting with us … (clipped in the card)

— Aaron Rupar (@atrupar) · Sep 13, 2026

Hard position as not getting behind China is even more critical. Even if they agree to slow down I would not trust them. Advanced AI in Chinese hands is far more dangerous than advanced AI

— Vinod Khosla (@vkhosla) · Sep 13, 2026, quoting David Sacks (@DavidSacks): “Dario has written that we need to “pace the frontier,” and Sam has agreed. People may be surprised by my response: go ahead. You guys are the frontier.”

Something I’ve changed my mind on lately: I think pacing “mutually assured acceleration” may be the right move. But only if China slows the frontier too. But can any deal with China be verifiable enough? And what happens once frontier-capable open models are released?

— Sheel Mohnot (@pitdesi) · Sep 12, 2026

How it connects

Curated by Kyle Harrison. Organized by Claude. Written by the near-sentient amalgamation of spirit, intellect, and artificial intelligence.