AI Safety Concerns: Researcher Quits Industry Over Risks

Jacob Coakson, a 27-year-old artificial intelligence researcher who spent recent years inside two of the world's leading AI companies, has announced his resignation from Anthropic, and his departure from the AI industry altogether. His reason: a deep concern that the race to develop increasingly powerful intelligent systems is advancing faster than humanity's capacity to maintain control over them.

Coakson, who previously worked at OpenAI before joining Anthropic, spent approximately three years researching the training of AI models. In a public statement, he argued that neither company is acting with sufficient responsibility and that the entire industry is "racing directly toward superhuman AI that improves itself." He characterized this trajectory as "gambling with our lives."

His primary worry isn't about smarter chatbots, but rather the point at which AI systems become capable of independently developing the next generation of AI systems themselves. Such a scenario could create an acceleration loop in which machines improve at an ever-increasing pace while humans struggle to understand how they function and what the implications of their capabilities might be.

Coakson also claimed that some people working within the industry seriously believe AI could become extremely dangerous by the end of the decade. He warned that future systems might achieve capabilities in areas such as hacking computer systems, scientific research, and technological development, accumulating power and resources at a rate that would be difficult for humans to stop.

The resignation is particularly dramatic because Coakson isn't an outside critic unfamiliar with the field. He emerged from within the industry itself, having worked at both OpenAI and Anthropic—a company especially identified with its emphasis on safety and responsible AI development.

According to him, his move to Anthropic actually helped him realize that even a company highly aware of the risks struggles to stop when competing against rivals. He argues the problem isn't just one company. As long as American companies compete with each other while simultaneously engaged in a global race against Chinese firms, every company has an incentive to continue developing more powerful systems to avoid falling behind. Coakson therefore calls for broader coordination between companies and governments, and even raised the possibility of a temporary pause on improving the capabilities of advanced models.

His claims have already sparked sharp reactions in the AI world. Owen Hubbinger, a researcher at Anthropic, responded that company personnel do take seriously the possibility that advanced AI could lead to catastrophic outcomes, and estimated that the risk of this happening in the coming decade is not negligible. Meanwhile, senior figures in the industry are calling for increased oversight and coordination around the development of more powerful systems.