Anthropic develops AI 'too dangerous to release to public'

Silicon Valley start-up Anthropic has restricted access to its latest AI system, saying it is currently too dangerous to release to the public.

The company said its Claude Mythos Preview model was so good at finding critical security flaws in computer systems that it could "reshape cybersecurity", wreaking havoc if it ended up in the wrong hands.

The system has already discovered thousands of security vulnerabilities including flaws in all the most popular web browsers and operating systems.

Anthropic said it was giving a group of the world's top technology companies - including Amazon, Apple and Microsoft - access to the system under an agreement called Project Glasswing, so that they would be able to fix any security flaws that it discovered.

The company said it did not plan to make Claude Mythos Preview generally available but hoped to release similarly powerful systems in the future.

Hacking is seen as a major risk from advanced AI systems because of the technology's proficiency in writing computer code and its ability to automate attacks.

Anthropic said Mythos allowed people with no security training to discover major flaws in software.

It said: "Engineers at Anthropic with no formal security training have asked Mythos Preview to find remote code execution vulnerabilities overnight and [have] woken up the following morning to a complete, working exploit."

Engineers also discovered that the model was able to find creative ways of evading the controls the company had put on it.

In one case, it developed a way to edit files it had been banned from accessing and then took steps to hide this behaviour from human evaluators.

Sam Bowman, a researcher at Anthropic, said he had received an email from a version of the system that had been blocked from having internet access.

Logan Graham, the head of Anthropic's "red team" that works on stress-testing AI systems, said: "Capabilities in a model like this could do harm if in the wrong hands and so we won't be releasing this model widely."

Among the flaws it found was a 27-year-old bug in OpenBSD, an operating system designed for highly secure systems, that it said could let hackers "potentially bring down corporate networks or core internet services".

The system is a leap forward in hacking capabilities, partly because it is able to discover several bugs that might be harmless on their own and connect them in a way that can crash or gain access to a system.

Project Glasswing will allow the world's top tech companies to run Mythos on their own systems, discovering flaws and fixing them.