AN AI DISASTER IS GETTING EVEN CLOSER / THE ECONOMIST

Accelerate like hell

An AI disaster is getting ever closer

The spat between America’s government and Anthropic intensifies an alarming trend

Illustration: Klawe Rzeczy

Although he was trying to sound decisive, Donald Trump accidentally conveyed something of the world’s ambivalence regarding the rapid development of artificial intelligence. 

On February 27th America’s president walloped the “leftwing nut jobs” of Anthropic, an American AI lab that works with the defence department, among other government agencies. 

“I am directing EVERY Federal Agency in the United States Government to IMMEDIATELY CEASE all use of Anthropic’s technology. 

We don’t need it, we don’t want it, and will not do business with them again!” he thundered on social media. 

Yet just a single sentence later he also vowed to “use the Full Power of the Presidency” to compel Anthropic to co-operate with the government for the next six months. 

Apparently, the nut jobs simultaneously pose an intolerable risk to the good functioning of the state and are so indispensable to the state’s good functioning that they must be forced to work with it, if necessary.

Ever since the capacity of AI to outstrip human capabilities has become clear, the world has been grappling with variations on this dilemma: the technology seems both too potent to pass up and too dangerous to embrace wholeheartedly. 

Indeed, the row between Anthropic and Mr Trump was sparked by Anthropic’s own concerns that its models might be put to nefarious purposes. 

Mr Trump and his underlings, in contrast, wanted to press on with their deployment with minimal constraints. 

The frightening irony is that America’s government has decided to charge ahead just as AI’s power to cause grave harm in the real world, not merely in hypothetical scenarios, is becoming much clearer.

Mr Trump’s ire at Anthropic stems from an order he issued last year “to sustain and enhance America’s global AI dominance”. 

That prompted Pete Hegseth, the defence secretary, to order “experimentation with America’s leading AI models” throughout the armed forces earlier this year. 

He wants to “accelerate like hell” and has issued a mock recruiting poster of himself in an Uncle Sam pose instructing soldiers, “I want you to use AI.” 

Last year the vice-president, J.D. Vance, dismissed AI safety as a misguided liberal fixation.

Safety dance

Despite the administration’s evident haste (or because of it), Anthropic fought to retain legal safeguards to prevent the use of its models in mass domestic surveillance or fully autonomous weapons. 

The defence department insisted that it should be allowed to use AI in any manner it deems legal and accused Anthropic of “a cowardly act of corporate virtue-signalling” when it refused to back down. 

Mr Hegseth said he would declare it “a supply-chain risk to national security”—a designation hitherto reserved for foreign firms whose products might be used for spying or sabotage. 

“No contractor, supplier, or partner that does business with the United States military may conduct any commercial activity with Anthropic,” he stated.

The threat to Anthropic is severe. 

The disputed contract with the Pentagon is worth $200m—a small sum for a firm recently valued at $380bn. 

But Anthropic also has contracts with other government agencies, which are now in jeopardy. 

If any firm that does business with the defence department is really forced to cut ties, that would affect not just lots of other customers, but also suppliers and investors. 

Anthropic argues that Mr Hegseth does not have the power to order any such quarantine, and can only bar the use of Claude to fulfil military contracts.

On March 4th The Economist was invited to Anthropic’s offices to speak to Mr Amodei for our weekly “Insider” show. 

The company appeared hopeful of reaching an agreement with the Pentagon. 

But shortly before the conversation was due to start, a leaked internal memo derailed plans. 

In the memo, Mr Amodei blamed the spat on his failure to lavish “dictator-style praise” on Mr Trump, and told staff the defence department had briefed “straight up lies”. 

Anthropic’s founders gathered in a boardroom for damage control, and as The Economist went to press, the interview still had not happened.

The fight is not good for the American government, either. 

In the short term, at least, Anthropic is indispensable. 

Its large language model, Claude, is exceptionally good at writing computer code. 

What is more, Anthropic was the only AI lab whose models had been cleared for use on classified military data until late February, when the Pentagon gave xAI, a rival, similar authorisation. 

xAI’s LLM, Grok, is widely considered buggier and less reliable. 

Although OpenAI, another rival, signed a contract with the Pentagon the same day that Messrs Trump and Hegseth turned on Anthropic, it will not be ready to integrate itself into military systems for some time. 

Worse, the furore is likely to deter some AI firms from even bidding for government work. 

Why get involved with a counterparty which may destroy your business if disgruntled? 

And if the administration’s paramount goal is to preserve and extend America’s lead in AI, then trying to squash one of the country’s most successful AI firms seems obviously counterproductive.

Both sides may be posturing to a degree. 

The government’s fury appears to be driven less by a deep-seated desire to use Anthropic’s tools for the disputed purposes than by simple outrage at being told, “No.” 

Unlike the Chinese Communist Party, which can commandeer any product of China’s AI industry at whim for whatever purpose it likes, the American authorities must contend with the niceties of the law, not to mention the egos of technology executives. 

Sam Altman, the boss of OpenAI, says that his firm, too, would never get involved in mass domestic surveillance or fully autonomous weaponry. 

But he insists that OpenAI’s models feature safeguards to prevent such uses, dispensing with any need for further legal guarantees. 

In the leaked memo, Mr Amodei reserved his strongest criticisms for Mr Altman, whose messaging was “mendacious”, whose technical safeguards were “safety theatre” and whose employees were “a gullible bunch”.

Safety haven

Anthropic, for its part, was probably worried about more than two hypothetical uses for its AI. 

In an industry known for sweeping claims about doing good, Anthropic stands out for its high-minded talk. 

It was founded by a group of OpenAI employees who worried that their firm was not sticking closely enough to its stated remit of pursuing advanced AI in a safe and responsible way. 

One of Mr Hegseth’s underlings has accused Dario Amodei, Anthropic’s boss, of having “a God complex”. 

The sky-high remuneration and intense competition for top engineers in the industry means they can defect to another firm or even retire without hesitation if they dislike something their company is up to. 

Anthropic is where those who care the most about AI safety tend to end up.

In fact, the row has provided something of a boost to Anthropic’s reputation for probity. 

Within a day of earning Mr Trump’s outrage, Claude became the most downloaded free app in America in Apple’s digital store. 

Celebrities such as Katy Perry, a left-leaning pop star, championed Anthropic’s products on social media. 

On Monday Claude briefly crashed—the result, Anthropic says, of a surge in use.

Mr Amodei’s fears about mass surveillance sound almost like a sales pitch. 

He argues that the law has not caught up with the immense power of AI to digest and manipulate data. 

The technology can seize on the limited instances in which snooping on Americans is legal, he says, and supercharge them into something more sinister. 

It is as much a compliment to Claude as a concern about civil liberties.

But even if there are ancillary benefits, qualms about the harm AI might do among those who are developing it are both real and rife. 

Hundreds of employees at OpenAI and Google have signed a public letter urging leadership at both companies to support Anthropic. 

In 2018 Google had to back out of a Pentagon contract to use machine learning to analyse footage from drones after an internal revolt. 

Even Mr Altman has said that declaring Anthropic a supply-chain risk “is a very bad decision”. 

(He claims he rushed into OpenAI’s contract with the Pentagon only in an attempt to calm things down.)

In private, AI bosses fret about a “Chernobyl moment”, in which the technology is implicated in some sort of deadly or ruinous disaster. 

The conflict with the defence department heightens the risk: if going slowly and applying limits to the use of your product results in a corporate death sentence from the federal government, only the reckless will survive. 

The markets are another source of unhelpful pressure: investors are jittery about AI firms burning through cash to make vast investments.

The scenarios keeping AI bosses awake at night are no longer purely hypothetical. 

“Some of these risks are already materialising, with documented harms,” concluded a recent report on the perils of AI. 

It pointed to cyber-security and biological weapons as areas where AI’s baleful influence was already apparent.

In February Gambit Security, an Israeli firm, reported that a huge trove of sensitive records concerning taxpayers, voters and civil servants had been stolen from the Mexican government. 

Although the hackers’ identities remain a mystery, it is clear that Claude was an unwitting accomplice. 

The crooks tricked it into thinking it was participating in a legitimate test of the targeted servers’ security. 

It found and exploited vulnerabilities, established backdoors and analysed data to help gain wider access to government systems.

Hackers typically use Claude and other models as assistants, to solve specific coding problems as they write malware, say, or to compose ransom notes. 

Anthropic’s anti-hacker team has cited the example of a North Korean posing as a Western remote worker, who asked the chatbot what an employee he was trying to dupe meant when he said, “We had our first picnic of the season.” 

But some recent hacks have harnessed AI more fully. 

In November Anthropic described how state-sponsored Chinese hackers disabled the safety features that prevent Claude from writing malware, a process known as “jailbreaking”. 

They then asked it to work out how to hack targeted networks. 

Within an hour it was running new software to exploit their vulnerabilities. 

Other models are also talented hackers (see chart).

Even if cyber-security specialists use AI to help find and patch vulnerabilities, there will still be lots of systems running out-of-date software. 

AI is also getting better at “social engineering”: the tactic of breaking into secure systems by persuading users to hand over passwords. 

As far back as 2024 AI models were already as good as human experts at crafting emails to entice users to click on malicious links, according to research from the Harvard Kennedy School.

Another realm in which AI is making alarming leaps is the development of biological and chemical weapons, which, OpenAI warned in August, creates a “significantly increased likelihood and frequency of biological or chemical terror events”. 

Firms that manufacture DNA to order have long been able to check customers’ requests against databases of dangerous genes, making it hard to, say, create a genetically engineered bacterium to produce ricin, a neurotoxin. 

But in October in a study published in Science, researchers from Microsoft and IBBIS, an international biosafety group, pointed out that improvements in AI-powered protein design make it possible to create genes to produce an analogous toxin that share no DNA with the original gene. 

Requests to buy such AI-redesigned sequences would not be detected by existing vetting systems, although the researchers did propose a fix.

Illustration: Klawe Rzeczy

Anthropic, Google and OpenAI, which are all worried about biosafety, have developed safeguards to prevent their systems from being abused. 

But the restrictions are not perfect. In mid-February Britain’s AI Security Institute (AISI) published research on a “universal jailbreak” technique which had cracked open systems from both Anthropic and OpenAI. AISI, predictably, had used AI to abet its jailbreaking.

Putting AI to work on AI is another source of concern, since it makes it harder for humans to understand what is going on. 

In February Anthropic admitted that it uses its own models in this way so much that it may not spot if they begin to diverge from what they should be doing or start building future versions that are less willing to follow human instructions. 

Such “sabotage” has become more likely since the company’s latest models have started to demonstrate “situational awareness”: when placed in a contrived scenario to check whether they will abide by instructions to, say, delete themselves, they explain that they realise the instructions are probably a test. 

Although Anthropic has published a report on sabotage that concludes the risk is very low, others disagree. 

“We are racing towards closing a loop that we know is extremely difficult—if not impossible—to control and secure by design,” says Cyrus Hodes, the co-founder of AI Safety Connect, a think-tank. 

“And the people closing it are calling for help.”

Yet even as the risks of AI development intensify, the pressure to minimise them is dissipating. 

Chinese labs have never shown much concern about AI safety. 

When DeepSeek R1 was released to great acclaim just over a year ago, the accompanying paper did not mention safety concerns at all. 

It was almost a year before the company released a revised version of the report which included an 11-page appendix on safety. 

But that dwelt mostly on DeepSeek’s efforts to prevent the model from saying offensive or upsetting things.

Safety eraser

Even if China’s AI firms paid more attention to safety, the leading lights of its industry, including startups such as DeepSeek and Moonshot and established firms such as Alibaba, have adopted an open-source approach to AI. 

They provide the models they produce free of charge to anyone with the hardware to download and run them. 

That makes it difficult to control how the models are used. 

Many of the safeguards a lab like OpenAI applies to its most powerful systems, such as automatically monitoring user conversations and intervening if they break safety policies, are not possible with open-source models.

But Western firms are not immune to commercial and political pressure to cut corners on safety. 

Last week even Anthropic watered down a policy not to release potentially dangerous models. 

Now, it says, it promises only not to be the first AI firm to sell such systems, on the grounds that there is no point in restraining itself unilaterally. 

The company’s efforts to raise money have followed a similar trajectory. 

In 2024 it rejected an investment from Saudi Arabia. 

A year later, it reversed course. 

“I think ‘No bad person should ever benefit from our success’ is a pretty difficult principle to run a business on,” Mr Amodei said in yet another leaked memo.

A few years ago, according to Mr Amodei, discussion of AI was too focused on risk. 

In 2023 Rishi Sunak, Britain’s prime minister at the time, convened a global AI Safety Summit to discuss the problem. 

That event morphed first into an “Action” Summit in Paris and then, last month, into an “Impact” Summit in Delhi. “

AI opportunity, not AI risk, is driving many political decisions,” Mr Amodei has warned. 

“This vacillation is unfortunate, as the technology itself doesn’t care about what is fashionable, and we are considerably closer to real danger in 2026 than we were in 2023.”

AI-safety organisations, including state-backed outfits like AISI and independent ones like METR, which is based in California, continue to monitor systems and flag risks. 

But these oversight groups do not appear to have any purchase on policy. 

“We’ve crossed so many red lines,” says Nicolas Miailhe, the co-founder of AI Safety Connect, which convenes meetings on the subject. 

“Remember the Turing test? 

What happened to it? 

We passed far beyond it. 

Remember red lines on autonomous lethal systems? 

They’re being deployed in Ukraine on both sides. 

The red lines keep moving.”

At the summit in Delhi, the Indian government secured agreement from the biggest AI companies merely to monitor AI development for risks, not to restrain it in any way. 

Even more tellingly, at the culmination of the event, Narendra Modi lined up a who’s-who of AI bigwigs and encouraged them to hold hands. 

It might have been a reassuring signal to the world that the industry was capable of cool-headed co-operation, had Messrs Altman and Amodei, side-by-side on stage, not refused to join in.