Anthropic researcher gives up equity to warn of AI threat

Two months before his equity was due to vest, an Anthropic researcher quit the artificial intelligence company, forgoing a potential financial windfall as he warned that advanced AI could be an existential threat to humanity.

Jacob Coxon, a former OpenAI employee, said he quit Anthropic with mounting worries that the race between the major AI firms could sideline safety concerns. He said he had “nothing to gain” by making the warnings after giving up his unvested equity.

By the end of the decade, Coxon warned, rapidly developing, self-improving AI systems could become difficult to control and could threaten human survival. His warnings have been echoed by other Anthropic researchers.

Evan Hubinger, Anthropic’s alignment science lead, said he and colleagues “earnestly believe AI could kill all humans” and estimated that there was more than a 10% chance that would happen within the next decade. Anthropic was working on the problem, but wasn’t yet clearly on track to solve the alignment challenge for superintelligent systems, he said.

Anthropic has pushed back against the criticism and publishes regular assessments of the catastrophic risks posed by its models. The company has also been open about instances of real-world computer systems being accessed without authorisation by Claude models during security assessments.