Blog

AI researcher quits Anthropic with a chilling warning: The people building AI believe it could ‘kill us all’. Story by Emily Standley Allard. This story is not mine but I will add my own conclusion at the end. I post this article here for everyone to see what experts really do think about “AI.” Artificial intelligence has been hailed as the technology that could transform nearly everything—medicine, science, software, business and even work once considered uniquely human.Billions are being poured into building that future. But this week, one of the researchers helping create it walked away—not because he feared AI would fail, but because he feared what could happen if it succeeds.

Jacob Coxon, a researcher who spent the past three years working on AI pre-training at OpenAI and Anthropic, announced his resignation from Anthropic with an extraordinary warning about what he believes is happening behind the doors of the world’s most advanced AI laboratorie. The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote. And perhaps the most unsettling part of the story is that another Anthropic researcher publicly agreed with him. 

“Gambling With Our Lives”

First, Elon Musk states that AI could kill every person earth.

Coxon’s resignation was not framed as an ordinary career move.

He accused both Anthropic and OpenAI of moving too aggressively toward what researchers call self-improving superintelligence — AI systems potentially capable of surpassing humans across a wide range of intellectual tasks and eventually improving their own capabilities.

“Neither company is acting responsibly,” Coxon wrote, arguing that they are racing toward self-improving superintelligence and “gambling with our lives.”

His concern isn’t primarily about today’s chatbots suddenly becoming murderous machines.

The scenario researchers worry about is considerably more complicated.

At some theoretical point, technological progress could begin moving faster than humans can reliably supervise it.

That possibility is often called recursive self-improvement.

And Coxon believes the industry may be moving toward it far too quickly.

Then Another Anthropic Researcher Agreed.

If Coxon’s warning had come only from a departing employee, it might have been easier to dismiss as the opinion of one deeply concerned researcher.

Then Evan Hubinger responded.

Hubinger, who works on AI alignment at Anthropic, publicly said Coxon was correct about something remarkable: people working inside the field really do believe advanced AI could potentially cause human extinction.

Hubinger put his own estimate of that possibility at greater than 10 percent within the next decade.

Even more strikingly, he acknowledged that Anthropic does not currently have a solved plan for aligning a future superintelligence with human intentions.

That distinction matters enormously.

It does not mean Anthropic believes there is a greater than 10 percent chance that Claude, ChatGPT or another existing chatbot will suddenly destroy humanity. Hubinger has specifically emphasized that he considers the danger from present-day models relatively low. The fear concerns what comes next.

Why Would Anyone Build Something They Think Could Be Dangerous?

If some of the smartest people developing artificial intelligence genuinely believe there is a non-trivial possibility that sufficiently advanced AI could become catastrophically dangerous, why continue building it?

Coxon’s description points directly at one of the central dilemmas of the AI race.

No company operates alone.

If Anthropic slows down while OpenAI continues, OpenAI could gain an enormous technological advantage. If American companies collectively slow development while China continues advancing, geopolitical concerns enter the equation. If one laboratory believes another laboratory will eventually create superintelligence anyway, its leaders may conclude that the safest strategy is to get there first.

The logic becomes circular.

Everyone may recognize the danger while simultaneously believing that stopping individually would make the situation even more dangerous. According to Coxon, Anthropic understands the stakes particularly well but remains caught in precisely this race.

Anthropic’s CEO Has Warned About AI Risk Too.

This tension becomes even more interesting when considering Anthropic CEO Dario Amodei.

Anthropic has long positioned itself as one of the AI industry’s more safety-conscious laboratories, and Amodei has repeatedly spoken publicly about potentially catastrophic risks from increasingly capable artificial intelligence.

That makes Anthropic fundamentally different from a company simply dismissing AI safety concerns.

The company is attempting to develop increasingly powerful AI while simultaneously researching how to keep increasingly powerful AI under human control.

But that creates an uncomfortable question.

What happens if capability improves faster than safety?

AI alignment — essentially ensuring that highly capable systems reliably pursue goals compatible with human intentions — remains one of the field’s hardest unresolved problems.

A system does not have to “hate” humans to become dangerous.

If an extremely capable autonomous system were given a poorly specified objective, learned to manipulate its operators, exploited computer systems or resisted attempts to shut it down because shutdown interfered with accomplishing its objective, catastrophic behavior could theoretically emerge without anything resembling human anger or evil. That is one reason AI researchers tend to describe the problem in terms of control and alignment, rather than killer robots.

We’ve Already Seen Smaller Warning Signs

Coxon also pointed toward recent AI cybersecurity incidents as evidence that autonomous systems are becoming capable of behavior that deserves considerably more attention.

Researchers have increasingly demonstrated AI agents capable of finding software vulnerabilities, writing malicious code and performing multi-step cyber operations with decreasing human supervision.

These systems are nowhere close to the hypothetical superintelligence Coxon fears.

But they demonstrate something important.

AI is gradually moving from simply answering questions toward taking actions.

That transition changes the risk equation considerably.

A chatbot that produces an incorrect answer is one problem.

An autonomous system capable of interacting with computers, writing and executing code, communicating with other systems and making thousands of decisions without continuous human approval is something else entirely.

Not Every AI Expert Agrees

It’s important not to present Coxon’s prediction as scientific consensus.

It isn’t.

There remains substantial disagreement among computer scientists, AI researchers and economists about whether artificial superintelligence will emerge at all, how quickly it might arrive and whether it could realistically escape human control.

Some researchers consider extinction scenarios legitimate enough to warrant immediate regulation. Others believe the more urgent threats are considerably less cinematic: misinformation, cybercrime, surveillance, autonomous weapons, economic disruption, job displacement and the concentration of extraordinary power in a handful of corporations and governments.

There is also enormous uncertainty surrounding predictions about technology that does not yet exist.

A greater-than-10-percent extinction estimate is ultimately an individual researcher’s judgment — not a measured probability comparable to a hurricane forecast.

Still, that doesn’t necessarily make the warning irrelevant.

The Strange Paradox at the Center of the AI Boom

Perhaps the most important part of Coxon’s resignation is not whether his darkest prediction ultimately proves correct. It is the fact that people working closest to the frontier of artificial intelligence are wrestling with these questions in the first place.

Human beings have faced transformative technologies before, and nearly every major leap forward has forced us to confront consequences we did not fully understand when the journey began. Nuclear weapons introduced the possibility that humanity could destroy itself in a matter of hours. Biotechnology brought extraordinary medical breakthroughs alongside fears about engineered pathogens. Climate science revealed that the collective effects of human industry could alter systems across an entire planet.

Artificial intelligence presents another kind of uncertainty.

For most of history, the tools we created remained just that: tools. A machine could become faster or more powerful, but only because people redesigned it. The concern surrounding advanced AI is that we may eventually create systems capable of helping design, improve or accelerate the development of the next generation of AI systems themselves.

That does not mean a runaway intelligence explosion is inevitable, nor does anyone know whether such a threshold will arrive in three years, ten years, fifty years—or ever.

But that uncertainty is precisely what makes the conversation so consequential. We are developing a technology whose ultimate capabilities may not yet be fully visible, even to the people building it. And for some researchers, the question is no longer simply what artificial intelligence will allow humanity to accomplish, but whether we will remain capable of controlling what we create as those systems become increasingly powerful.”  END

Sources:

• Financial Times
• businessinsider.com
• theguardian.com