Anthropic researcher resigns, warns AI race could threaten humanity

Jacob Coxon has announced his resignation from Anthropic, warning that the race to develop artificial intelligence is rapidly moving toward the creation of superintelligent systems capable of self-improvement, and accusing major AI companies of "gambling with our lives."

Coxon, who spent three years working on pre-training research at both OpenAI and Anthropic, said the two companies are acting irresponsibly as they push to develop advanced AI that can enhance itself.

He cautioned that future AI systems could surpass humans in many areas, potentially breaching any safeguards and causing radical change in a short time, as well as gaining real access to power and resources.

Coxon stated that AI developers are aware that this technology could "kill us all" before the end of the decade, stressing that these concerns are not marketing tactics and that some executives express even greater worries in private conversations.

He questioned why development continues despite the risks, noting that many at OpenAI do not fully grasp the scale of the threats, while Anthropic is aware of the dangers but remains caught in the race to achieve superintelligence first.

Coxon added that this race is driven by the belief that other companies will not act responsibly, pushing Anthropic to move forward despite the risks. He described this as an "arrogant gamble" that should not be made by a private company.

He called for not accelerating AI alignment with human values unless there is exceptional confidence that no safer alternatives exist, expressing hope that coordination among AI labs is possible.

Coxon pointed to incidents such as the attack on the Hugging Face platform as prompting the possibility of agreements on the pace of AI development among US labs, but warned that the world is not on track to prevent a global race, which could require costly measures such as a temporary ban on developing advanced model capabilities.

In closing, Coxon urged those working in AI labs to consider what the coming years may bring, questioning the wisdom of releasing superintelligent systems without a rigorous understanding of their thinking, or continuing to operate under the assumption that "it will happen anyway" instead of demanding safer conditions.