Anthropic AI Researcher Resigns, Calls for Industry‑Wide Pace‑Setting on Self‑Improving Systems

Jacob Coxon, a senior researcher at the AI safety‑focused startup Anthropic, announced his departure this week, citing deep concerns that rapidly advancing, self‑modifying artificial intelligence could pose an existential threat to humanity.

Coxon, who has worked on language models and alignment research, said the trajectory of increasingly autonomous AI systems is outpacing the safeguards currently in place. He warned that without coordinated limits, the technology could reach a point where it improves itself without human oversight, creating scenarios that are difficult to predict or control.

The resignation adds a personal dimension to a debate that has been gaining visibility across the tech sector. Over the past year, leading AI labs have faced mounting pressure from academics, policymakers, and civil‑society groups to address the potential risks of advanced AI. While some companies have pledged to adopt safety protocols, critics argue that competitive pressures make it hard to slow development without a shared framework.

Coxon’s primary recommendation is the establishment of “pacing agreements” among AI developers—a voluntary pact to align research timelines and safety standards. Such agreements would aim to prevent a race to the bottom, where labs prioritize speed over caution. Similar ideas have surfaced in prior industry forums, but no binding arrangement has yet emerged.

Anthropic responded to the news by acknowledging Coxon’s contributions and reiterating its commitment to safety research. The firm said it continues to explore collaborative approaches with peers, though it stopped short of confirming any formal pacing plan. Reactions from other AI labs have been muted, reflecting the delicate balance between innovation and risk management.

Looking ahead, Coxon’s exit may intensify calls for external oversight. Lawmakers in several jurisdictions are already drafting legislation aimed at high‑risk AI applications, and international bodies are discussing standards for advanced systems. Whether the industry can coalesce around a unified pacing strategy remains uncertain, but the departure underscores the growing urgency of the conversation.