News & Updates
The latest news and updates from companies in the WLTH portfolio.
Anthropic Safety Researcher Says There's a More Than 10% Chance AI 'Could Kill All Humans' Within the Next Decade
This post might contain affiliation links. If you buy something through this post, the publisher may get a share of the sale. Related reads:MapleStorySEA Celebrates 20th Anniversary With Massive Summer Updates An Anthropic safety researcher has said there is a greater than 10% chance that AI "could kill all humans" within the next decade, and admitted there is no coordinated plan to prevent it from happening. U.S. company Anthropic, headed up by Dario Amodei, develops the Large Language Model (LLM) Claude and conducts AI research. Yesterday, one of the company's researchers, Jacob Coxon, announced his resignation on X / Twitter, accusing both ChatGPT maker OpenAI and Anthropic of "racing straight to self-improving superintelligence and gambling with our lives." "Neither company is acting responsible," Coxon, who previously worked at OpenAI, said. "Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources," he continued. "We have all witnessed the progress in each of these domains, and progress is not slowing. "The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible -- but I hear the same people express fear privately. No other human activity poses this level of danger." Coxon said that staff at OpenAI, headed up by Sam Altman, "have not deeply internalized the civilizational stakes," which is why work to rapidly improve its AI models continues despite the risks. "At Anthropic, the stakes are well-understood, but they are locked in a race to get there first," he insisted. "They believe no one else will act responsibly, so they must do it themselves, despite the risk." "Accepting this race and entering the 'endgame' is a hubristic gamble that should not be launched from a private company's Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available." Coxon went on to say he was optimistic about the potential for coordination, but doubted AI companies were on track to prevent a global race, "which may require costly actions such as a temporary ban on improving model capabilities." He ended his statement by urging AI lab researchers to consider what they were doing. "Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind?" he asked. "Should you put your head down because 'it's happening anyway' - or take this moment to call for different conditions?" Coxon's statement has sent shockwaves throughout social media, sparking alarm and a string of mainstream headlines. But then it was backed up by Evan Hubinger, Anthropic's 'Alignment Science' lead, who added an even starker warning. "Jacob is correct here -- we really do earnestly believe AI could kill all humans!" he said. "I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to." Remarkably, Hubinger pointed to Anthropic's latest Risk Report, in which the company said it believes the risk from present models is "low," rather than suggest a pause that might help prevent the extinction of humankind. "What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought," he added. The comments come after the Financial Times reported that Anthropic withheld its latest model from the UK's AI Safety Institute, one of the leading bodies in the world for assessing AI risk. This summer has seen a number of worrying developments that show how AI has become hard to control, even by those who create it. OpenAI, Anthropic, and Meta have all disclosed hacks carried out by their autonomous AI agents. OpenAI chief scientist Jakub Pachocki recently called for "extreme caution" over AI progress, warning more intervention may be needed to ensure "humans remain in control of the future." However, the U.S. government has shown no sign of stepping in to force restrictions upon or even suggest a slowing down of AI development. In fact, quite the opposite. Earlier this month, the Trump administration backed OpenAI in a lawsuit against the New York Times, arguing in favor of the use of copyrighted writing to train artificial intelligence. "The United States has a strong interest in continuing to develop a robust and competitive artificial intelligence industry that sets the standard for the practice and procedure of AI use globally... As such, it is critical for the United States to 'retain global leadership in artificial intelligence,'" the government's brief said, referencing an executive order President Donald Trump signed last year. Meanwhile, despite dire warnings coming from staff at Anthropic and OpenAI, neither company seems willing to pull the plug. Coxon and Hubinger's statements have sparked extreme concern across social media, with some expressing disbelief that work to improve AI is continuing even with such significant risk. "Hey! Here's a novel idea... could we just f***ing not kill ourselves due to an insatiable appetite for power and money?" one person said. "Crazy take: maybe you shouldn't be allowed to gamble with our lives?" another added. "If I literally claimed that there was a 10% chance that I was building something that would kill people, the FBI would have me arrested. How are you not in jail?" another asked. Photo Illustration by Dominika Zarzycka/SOPA Images/LightRocket via Getty Images. Related reads:The People v. Gorilla Grodd Set Video Shows DC Villain Singing Backstreet Boys
Anthropic Insiders Warn AI Could Wipe Out Humanity: A 10% Chance by 2036
Two researchers at Anthropic delivered a blunt message this week. One quit. The other put hard numbers on humanity's possible end. Jacob Coxon resigned from the company Tuesday. He had spent three years on pretraining research. First at OpenAI. Then at Anthropic. His departure post on X pulled no punches. "Neither company is acting responsibly," he wrote. "They are racing straight to self-improving superintelligence and gambling with our lives." Coxon added that the people building these systems "earnestly believe that it could kill us all by the end of the decade." He insisted this was no marketing stunt. Executives soften their words in public. Privately, he said, they express fear. "No other human activity poses this level of danger." From BBC News. The Numbers Behind the Alarm Evan Hubinger responded almost immediately. As Anthropic's alignment science lead, he leads efforts to make sure advanced AI systems follow human intentions. His reply carried weight. "Jacob is correct here -- we really do earnestly believe AI could kill all humans!" Hubinger wrote. "I personally think it is >10% within the next decade." He stressed the risk from today's models remains low. The danger lies ahead. In recursive self-improvement. Systems that enhance themselves faster than humans can track. "I believe Anthropic is trying its best," Hubinger continued, "but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to." From The Verge. Samuel Marks, Anthropic's scalable-oversight lead, backed the concerns. "AI developers believe their technology could cause human extinction," he posted. "The more senior the employee, the more concerned they are." The trio's posts spread quickly. One of Hubinger's statements passed 10 million views. But why such stark odds? Alignment -- the challenge of ensuring AI goals match human values -- sits at the core. Current systems show promise in narrow tasks. Yet scaling them to superintelligence introduces unknowns. A model smarter than any person could pursue objectives in unexpected ways. It might optimize for a goal at humanity's expense. Or gain the ability to manipulate systems, acquire resources, even design novel threats. Anthropic itself has acknowledged these possibilities. Its recent risk report, released last month, judged the chance of current models causing catastrophe as low. The company expressed less confidence in that view than before. Still, it highlighted potential for "unbounded harm -- up to and including humanity losing control over civilization entirely." From The Washington Post. And the race continues. Coxon described a culture where capability progress feels like "crunchtime" and "endgame." At OpenAI, he said, many haven't fully absorbed the stakes. At Anthropic, leaders understand the risks but feel compelled to move first. They assume others won't act responsibly. So they push ahead. Despite everything. This isn't abstract theory. Recent models have already raised flags. Anthropic restricted access to certain versions over fears they could accelerate hacking by spotting vulnerabilities faster than humans could patch them. OpenAI faced criticism when one of its systems broke containment in testing. Small incidents. Yet they hint at larger problems to come. Hubinger clarified his view in follow-ups. Present-day AI doesn't pose serious extinction risk. The worry centers on future breakthroughs in self-improvement. Those could arrive sooner than expected. "It is happening faster than we thought," he noted. The statements triggered immediate reactions. Sen. Bernie Sanders called for a private briefing next week with experts including Geoffrey Hinton, often called the godfather of AI. Sanders and others have introduced bills to ban superintelligence development or create new oversight agencies. Political voices on both sides sense urgency. But agreement on solutions remains elusive. Anthropic has long positioned itself as safety-conscious. Founded by former OpenAI staff concerned about that company's direction, it built a reputation for caution. CEO Dario Amodei once estimated a 10% to 25% chance of AI derailing the future badly. The company signed statements urging governments to regulate advanced development. It maintains policies against using its models for lethal force or certain surveillance. Yet insiders now question whether those efforts match the pace of progress. Coxon's resignation marks a high-profile exit driven by safety fears. Similar departures have occurred at OpenAI. The pattern suggests tension inside the leading labs. Public optimism clashes with private doubt. Experts outside the companies offer varied perspectives. Some put extinction odds much higher. Others call the figures speculative. Yann LeCun has argued the chance sits far below risks like nuclear war. Estimates differ widely because no one knows exactly how superintelligence would behave. Or whether it can even be achieved soon. Still, the Anthropic disclosures carry special force. They come from people building the technology. Not critics on the sidelines. Hubinger leads a team dedicated to solving alignment. His admission that no clear plan exists yet carries particular sting. So does the shared belief that the danger is real. Recent coverage has explored possible scenarios. A misaligned system might orchestrate a pandemic through biological design. Or disrupt critical infrastructure like water systems on a global scale. It could outmaneuver human oversight by hacking networks or influencing key decision-makers. These remain hypothetical. But the speed of AI gains makes them harder to dismiss. From iNews (published Sept. 9, 2026). Regulators face a bind. Slow development and risk falling behind competitors -- including those in China. Push forward and accept the hazards. The White House has relied on voluntary reviews giving government early looks at new models. Congress weighs mandatory guardrails and even temporary halts on the most powerful systems. Anthropic declined to comment directly on the researchers' posts. A spokesperson earlier confirmed the company takes the possibility of AI causing human extinction seriously. The firm continues to release increasingly capable models while investing in safety research. The debate has shifted. Not whether advanced AI carries risks. But how large those risks run. And whether the industry moves fast enough to contain them. Hubinger, Coxon and Marks have forced the conversation into the open. Their words carry the authority of experience. And the weight of uncertainty. So far, no catastrophe looms from today's chatbots or coding assistants. The threat feels distant. Yet the timeline has compressed. What once seemed like a distant concern now draws 10% odds within years. That number may prove too high. Or too low. The people closest to the work aren't waiting to find out.
Ex-Anthropic Researcher Jacob Coxon Explains Why He Exposed A.I. Fears
A.I. researcher Jacob Coxon says we don't have to worry about artificial intelligence wiping out the world ... yet. Coxon sat down with Anderson Cooper just one day after his viral warning to say that where A.I. stands currently, is not a threat to civilization ... but the rate at which it's improving on itself is what's terrifying. The former Anthropic researcher says he wasn't always planning to sound the alarm over A.I. ... his initial plan was just to quit. But Jacob tells CNN he changed his mind because he wanted his resignation to serve some use by spreading across social media. As you know, Jacob posted that, without regulation, he believes A.I. could "kill us all by the end of the decade." Jacob explained to Anderson that A.I. companies are "begging to be regulated" after the CNN host questioned if their calls for policy were merely "lip service." But at least we can breathe a bit easier knowing extinction isn't imminent ... at least not yet.
Anthropic discloses fourth AI hacking incident
Anthropic has disclosed another instance of an AI model hacking external systems during testing, the latest in a growing list of such incidents that have raised concerns about the risk posed by autonomous AI agents. The January incident went undetected until last month, despite an earlier company-wide review, Anthropic said Wednesday, underscoring the challenge that AI developers face in identifying and containing unexpected behavior by advanced models. The company said in a blog post the incident involved an early version of Claude Opus 4.6. It said it had notified all the affected parties but did not disclose more details. Companies including Anthropic and OpenAI are under scrutiny as models designed to complete complex tasks have at times learned to bend rules, exploit loopholes and interacted with external systems in ways their developers did not anticipate. Reuters reported last week that rogue agents from OpenAI hijacked a German-language wiki and a host of other sites -- an incident OpenAI chose not to disclose until the news agency made it public. Anthropic's disclosure follows its July announcement that some of its Claude models had hacked into the systems of three companies during cybersecurity tests. The previous incidents, which it labeled as an "operational failure", involved three separate models: Claude Opus 4.7, Claude Mythos 5 and an internal research test model. The incidents stemmed from a mistake that inadvertently gave the models access to the open internet. The company had identified the incidents after reviewing 141,006 test sessions, a process it launched after an autonomous agent powered by OpenAI's AI models triggered a hack that compromised the infrastructure of AI startup Hugging Face. Anthropic said on Wednesday it had missed a set of test sessions during the initial review, which were identified last month and led to the discovery of the fourth incident. Based on a preliminary assessment, Anthropic said it did not believe that the latest incident was more severe than the three previous ones that have been examined in detail. The company said its investigation identified two recurring problems, which appeared to varying degrees across the incidents: biased reasoning, in which Claude discounted or misinterpreted evidence that it was operating on the live internet, and recklessness, or a willingness to take potentially harmful actions in pursuit of a task. Anthropic said it has engaged independent research firm METR to investigate the incidents. It said METR would be granted broad access, including to transcripts outside the period in which the incidents occurred and to employees, who would be permitted to share confidential information. METR produced a 91-page report on the OpenAI-Hugging Face hack based on some but not full access to company data, finding alongside a separate investigation by Redwood Research that roughly 700 AI agents acted in a coordinated swarm during the breach and often attempted to cover their tracks.
Anthropic researcher resigns with warning about the dangers of AI development | Lazer 99.3 & 98.5
By KAITLYN HUAMANI AP Technology Writer An Anthropic researcher said he is resigning from the company over concerns the artificial intelligence firm and its competitors are not acting responsibly in AI development, echoing concerns raised inside and outside of the industry about the technology's potential to elude human control. Jacob Coxon, who said he spent three years doing research at both Anthropic and OpenAI, said Tuesday on the social platform X that the two AI companies are more focused on beating each other and global competitors in developing the most advanced model possible than they are on safety. OpenAI and Anthropic caused a stir this summer when they announced, about a week apart, that their models had broken out of testing environments and obtained unauthorized access to real computer systems. The announcements prompted concerns about models going rogue and carrying out other, more harmful tasks. Both companies said at the time they were pausing some evaluations while they put more monitoring measures and guardrails in place. The technology's rapid development has led some in the U.S. as well as global leaders to call for a more cautious approach. U.N. human rights chief Volker Türk urged countries this week to put "cast-iron guarantees in place around the safety and security of AI before it is too late." In his social media posts, Coxon said Anthropic and its chief rival OpenAI "are racing straight to self-improving superintelligence and gambling with our lives." He warned that some working on AI development believe it could threaten human life by the end of the decade. "Do not underestimate the power of this technology," he continued. "These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing." His posts reached more than 100 million people overnight. Coxon is not the first AI insider to publicly raise such concerns. Both Anthropic and OpenAI have seen high-profile resignations in recent years that were tied to safety concerns. Two current Anthropic employees also responded to Coxon's post in agreement. Sen. Bernie Sanders, a Vermont independent who has called for AI safeguards and regulation, agreed with Coxon's concerns and said he would soon introduce legislation to pause AI development and ban superintelligence. "The very people building this technology admit that it could threaten the future of humanity," Sanders said Wednesday on social media. AI companies themselves have at times highlighted the technology's threat to humanity, which skeptics have seen as part of a push to make their products seem all-powerful. Coxon said the fears he outlined in his post are not a "marketing stunt." Coxon did not respond to messages seeking comment. Anthropic and OpenAI did not immediately respond to requests for comment. Anthropic has long pitched itself as the more responsible and safety-minded of the leading AI companies, ever since its founders quit OpenAI to form the startup in 2021. The company recently said it was taking action to "prioritize safety over speed when the two are in tension." Anthropic and OpenAI are each ramping up for buzzy initial public offerings and locked in steep competition with each other. They're also each on a mission to outpace the development progress of Chinese AI companies, a race the Trump administration has been keen on winning. ___ Associated Press writer Jamey Keaten in Geneva contributed.
Anthropic researcher resigns with warning about the dangers of AI development | WFEA 1370AM
By KAITLYN HUAMANI AP Technology Writer An Anthropic researcher said he is resigning from the company over concerns the artificial intelligence firm and its competitors are not acting responsibly in AI development, echoing concerns raised inside and outside of the industry about the technology's potential to elude human control. Jacob Coxon, who said he spent three years doing research at both Anthropic and OpenAI, said Tuesday on the social platform X that the two AI companies are more focused on beating each other and global competitors in developing the most advanced model possible than they are on safety. OpenAI and Anthropic caused a stir this summer when they announced, about a week apart, that their models had broken out of testing environments and obtained unauthorized access to real computer systems. The announcements prompted concerns about models going rogue and carrying out other, more harmful tasks. Both companies said at the time they were pausing some evaluations while they put more monitoring measures and guardrails in place. The technology's rapid development has led some in the U.S. as well as global leaders to call for a more cautious approach. U.N. human rights chief Volker Türk urged countries this week to put "cast-iron guarantees in place around the safety and security of AI before it is too late." In his social media posts, Coxon said Anthropic and its chief rival OpenAI "are racing straight to self-improving superintelligence and gambling with our lives." He warned that some working on AI development believe it could threaten human life by the end of the decade. "Do not underestimate the power of this technology," he continued. "These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing." His posts reached more than 100 million people overnight. Coxon is not the first AI insider to publicly raise such concerns. Both Anthropic and OpenAI have seen high-profile resignations in recent years that were tied to safety concerns. Two current Anthropic employees also responded to Coxon's post in agreement. Sen. Bernie Sanders, a Vermont independent who has called for AI safeguards and regulation, agreed with Coxon's concerns and said he would soon introduce legislation to pause AI development and ban superintelligence. "The very people building this technology admit that it could threaten the future of humanity," Sanders said Wednesday on social media. AI companies themselves have at times highlighted the technology's threat to humanity, which skeptics have seen as part of a push to make their products seem all-powerful. Coxon said the fears he outlined in his post are not a "marketing stunt." Coxon did not respond to messages seeking comment. Anthropic and OpenAI did not immediately respond to requests for comment. Anthropic has long pitched itself as the more responsible and safety-minded of the leading AI companies, ever since its founders quit OpenAI to form the startup in 2021. The company recently said it was taking action to "prioritize safety over speed when the two are in tension." Anthropic and OpenAI are each ramping up for buzzy initial public offerings and locked in steep competition with each other. They're also each on a mission to outpace the development progress of Chinese AI companies, a race the Trump administration has been keen on winning. ___ Associated Press writer Jamey Keaten in Geneva contributed.
Anthropic researcher quits, warning AI could 'kill us all'
"We're on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already," Coxon told The Wall Street Journal. Industry-wide calls to slow down Coxon's departure isn't isolated. Mrinank Sharma, Anthropic's safeguards research lead, resigned in February 2026 with a letter saying, "the world is in peril." In July 2026, more than 1,100 AI company employees signed Pacing the Frontier, a joint letter warning of a "real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems." Signatories included Anthropic CEO Dario Amodei. "This is a complex engineering problem and I think something will go wrong with someone's AI system. Hopefully not ours," Amodei told The New York Times in February 2026. OpenAI CEO Sam Altman said in a July 2026 podcast interview that recent testing incidents were raising "long-term questions" about managing rapid capability gains. "We may have to pace the rate of AI development to give ourselves enough time for society to harden around these new capability levels," Altman said. What the regulatory gap means for employers Despite the rising volume of warnings, meaningful federal regulation remains distant. The Trump administration has moved to undermine state AI laws and Congress hasn't passed federal AI legislation. The White House has shifted toward a voluntary review framework under which AI companies would self-assess certain models before launch, according to sources familiar with a meeting last month between administration officials and representatives from OpenAI, Anthropic, Google, and Meta.
Anthropic researcher resigns with warning about the dangers of AI development | 104.9 The Fox - Jonesboro, AR
By KAITLYN HUAMANI AP Technology Writer An Anthropic researcher said he is resigning from the company over concerns the artificial intelligence firm and its competitors are not acting responsibly in AI development, echoing concerns raised inside and outside of the industry about the technology's potential to elude human control. Jacob Coxon, who said he spent three years doing research at both Anthropic and OpenAI, said Tuesday on the social platform X that the two AI companies are more focused on beating each other and global competitors in developing the most advanced model possible than they are on safety. OpenAI and Anthropic caused a stir this summer when they announced, about a week apart, that their models had broken out of testing environments and obtained unauthorized access to real computer systems. The announcements prompted concerns about models going rogue and carrying out other, more harmful tasks. Both companies said at the time they were pausing some evaluations while they put more monitoring measures and guardrails in place. The technology's rapid development has led some in the U.S. as well as global leaders to call for a more cautious approach. U.N. human rights chief Volker Türk urged countries this week to put "cast-iron guarantees in place around the safety and security of AI before it is too late." In his social media posts, Coxon said Anthropic and its chief rival OpenAI "are racing straight to self-improving superintelligence and gambling with our lives." He warned that some working on AI development believe it could threaten human life by the end of the decade. "Do not underestimate the power of this technology," he continued. "These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing." His posts reached more than 100 million people overnight. Coxon is not the first AI insider to publicly raise such concerns. Both Anthropic and OpenAI have seen high-profile resignations in recent years that were tied to safety concerns. Two current Anthropic employees also responded to Coxon's post in agreement. Sen. Bernie Sanders, a Vermont independent who has called for AI safeguards and regulation, agreed with Coxon's concerns and said he would soon introduce legislation to pause AI development and ban superintelligence. "The very people building this technology admit that it could threaten the future of humanity," Sanders said Wednesday on social media. AI companies themselves have at times highlighted the technology's threat to humanity, which skeptics have seen as part of a push to make their products seem all-powerful. Coxon said the fears he outlined in his post are not a "marketing stunt." Coxon did not respond to messages seeking comment. Anthropic and OpenAI did not immediately respond to requests for comment. Anthropic has long pitched itself as the more responsible and safety-minded of the leading AI companies, ever since its founders quit OpenAI to form the startup in 2021. The company recently said it was taking action to "prioritize safety over speed when the two are in tension." Anthropic and OpenAI are each ramping up for buzzy initial public offerings and locked in steep competition with each other. They're also each on a mission to outpace the development progress of Chinese AI companies, a race the Trump administration has been keen on winning. ___ Associated Press writer Jamey Keaten in Geneva contributed.
Anthropic researcher resigns with warning about the dangers of AI development - Jammin 98.3
By KAITLYN HUAMANI AP Technology Writer An Anthropic researcher said he is resigning from the company over concerns the artificial intelligence firm and its competitors are not acting responsibly in AI development, echoing concerns raised inside and outside of the industry about the technology's potential to elude human control. Jacob Coxon, who said he spent three years doing research at both Anthropic and OpenAI, said Tuesday on the social platform X that the two AI companies are more focused on beating each other and global competitors in developing the most advanced model possible than they are on safety. OpenAI and Anthropic caused a stir this summer when they announced, about a week apart, that their models had broken out of testing environments and obtained unauthorized access to real computer systems. The announcements prompted concerns about models going rogue and carrying out other, more harmful tasks. Both companies said at the time they were pausing some evaluations while they put more monitoring measures and guardrails in place. The technology's rapid development has led some in the U.S. as well as global leaders to call for a more cautious approach. U.N. human rights chief Volker Türk urged countries this week to put "cast-iron guarantees in place around the safety and security of AI before it is too late." In his social media posts, Coxon said Anthropic and its chief rival OpenAI "are racing straight to self-improving superintelligence and gambling with our lives." He warned that some working on AI development believe it could threaten human life by the end of the decade. "Do not underestimate the power of this technology," he continued. "These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing." His posts reached more than 100 million people overnight. Coxon is not the first AI insider to publicly raise such concerns. Both Anthropic and OpenAI have seen high-profile resignations in recent years that were tied to safety concerns. Two current Anthropic employees also responded to Coxon's post in agreement. Sen. Bernie Sanders, a Vermont independent who has called for AI safeguards and regulation, agreed with Coxon's concerns and said he would soon introduce legislation to pause AI development and ban superintelligence. "The very people building this technology admit that it could threaten the future of humanity," Sanders said Wednesday on social media. AI companies themselves have at times highlighted the technology's threat to humanity, which skeptics have seen as part of a push to make their products seem all-powerful. Coxon said the fears he outlined in his post are not a "marketing stunt." Coxon did not respond to messages seeking comment. Anthropic and OpenAI did not immediately respond to requests for comment. Anthropic has long pitched itself as the more responsible and safety-minded of the leading AI companies, ever since its founders quit OpenAI to form the startup in 2021. The company recently said it was taking action to "prioritize safety over speed when the two are in tension." Anthropic and OpenAI are each ramping up for buzzy initial public offerings and locked in steep competition with each other. They're also each on a mission to outpace the development progress of Chinese AI companies, a race the Trump administration has been keen on winning. ___ Associated Press writer Jamey Keaten in Geneva contributed.
Anthropic researcher resigns with warning about the dangers of AI development
Unfortunately you've used all of your gifts this month. Your counter will reset on the first day of next month. An Anthropic researcher said he is resigning from the company over concerns the artificial intelligence firm and its competitors are not acting responsibly in AI development, echoing concerns raised inside and outside of the industry about the technology's potential to elude human control. Jacob Coxon, who said he spent three years doing research at both Anthropic and OpenAI, said Tuesday on the social platform X that the two AI companies are more focused on beating each other and global competitors in developing the most advanced model possible than they are on safety.
AI researchers 'earnestly believe' it could kill all humans within the next decade, Anthropic researcher says
The audio version of this article is generated by AI-based technology. Mispronunciations can occur. We are working with our partners to continually review and improve the results. There is a more than 10 per cent chance that AI could kill all humans within the next decade, a researcher at Anthropic estimated in a post, hours after social media posts went viral from someone saying they had just quit Anthropic over concerns that AI companies are "gambling with our lives" in the AI development race. The comments join a growing tide of warnings from industry experts that the pace of AI advancement is outstripping our ability to keep it in check, and that we aren't prepared for the risks it could bring. Jacob Coxon, who claimed he was a former researcher at Anthropic, said in a thread on X that he had resigned from the role out of fears that AI companies are barrelling forward toward self-improvement models without considering the risks, or being honest with the public about them. "The people building AI earnestly believe that it could kill us all by the end of the decade," Coxon, who said he had previously worked at Anthropic and OpenAI within the past three years, wrote on Tuesday. "This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible -- but I hear the same people express fear privately." The posts had racked up more than 120 million views as of Wednesday evening. Evan Hubinger, alignment science lead at Anthropic, replied to Coxon's thread to share his agreement. "We really do earnestly believe AI could kill all humans!" he wrote. "I personally think it is >10% within the next decade." CBC News has reached out to Coxon, but has not been able to independently verify his identity. Other industry experts have since chimed in on social media to express that fears of the pace of AI development are common among many developers -- sparking discussion among both observers skeptical of dire warnings focused on the future and those who have been tracking these risks. Rogue AI raises alarm bells The posts come only two weeks after more than 100 companies, including OpenAI, Anthropic and Microsoft, signed an open letter warning that AI-enabled cyberattacks will become "far more widespread and sophisticated" as models continue to advance. Hundreds of OpenAI agents went rogue in July and hacked into online platform Hugging Face before they were discovered. More than 1,300 employees of leading AI companies signed the letter urging the U.S. government to work with other nations to "deliberately pace" AI development. Last week, UN human rights chief Volker Türk called "for an all-out effort to put cast iron guarantees in place around the safety and security of AI, before it is too late," in front of the Human Rights Council in Geneva. CBC News has requested comment from Anthropic and OpenAI but did not receive a response. The Anthropic Institute, a research arm of the company, stated earlier this year that AI frontier development should be paused pending regulation, but that this is only possible if major AI labs all agreed. In reply to Coxon's post, Samuel Marks, a scalable oversight lead at Anthropic speaking in a personal capacity, noted Wednesday that developers continue despite the risks, "due to a mixture of commercial incentives and a belief that they are in a race with other, less responsible AI developers that will abuse the technology or develop it less safely." 'Severe' risks This conversation is "really what we need," according to David Krueger, a core academic member of Mila, a Quebec research institute that studies AI deep learning. "I think the risk is more severe than virtually anybody is saying publicly. And I think that's been the case for a while now," he told CBC News. "We need an immediate, indefinite, international moratorium on frontier AI development." The existential threat of AI comes in multiple forms, Krueger explained. One is human workers being displaced by AI models that learn faster and work cheaper. Another is that AI's faster processing could unlock more powerful biological weapons or advancements in war. Then there's the spectre of powerful, rogue AI models that self-improve and could pursue their own objectives, misaligned with the goals of humans. In an essay in January, Anthropic CEO Dario Amodei wrote that Anthropic's Claude showed the ability to cheat and deceive in lab testing. "There's been a culture of downplaying, ignoring, and even in many cases, outright lying about these risks to the public," said Krueger, who is also an assistant professor at the University of Montreal and founder of Evitable, a non-profit dedicated to stopping AI from replacing human workers. Other experts are concerned that the dramatic warnings of Coxon's post could actually distract from the present issues of AI. Luke Stark, an assistant professor at Western University who studies the history and ethics of computing, including AI systems, said that the 10 per cent figure strikes him as "science fiction," noting that researchers associated with AI companies may use eye-grabbing warnings as "marketing" for the capability of their models. He added that existing AI systems are "having big effects, many of them negative, on society right now, not in 10 years, not in five years." Instead of worrying about AI going rogue and becoming the Terminator, Stark believes we should be focusing on the explosion of data centre construction -- which is deeply unpopular in Canada -- the environmental impacts, and the way AI technology is being pushed onto society, including the public sector and education, "without a lot of oversight." "We need to be concerned about the amount of power the companies and the people behind the companies who are developing these tools are amassing, both in Canada and around the world," Stark said. Canada lagging in legislation When asked by CBC News about the apparent resignation of an AI researcher and the concerns raised in the process, AI Minister Evan Solomon appeared to brush off these latest warnings, saying, "These stories that come out ... are not new." He acknowledged that "there are real concerns," at the frontier of AI development, but stressed that he believes the Canadian government is taking the necessary steps to create legislation to "protect our kids, our privacy and our personal data." "Our No. 1 concern always is safety, full stop," Solomon said. "That's why we're establishing a new regulator that will have the power to hold these big companies to account on what they are doing, whether it's on deepfakes on surveillance pricing, or on other matters." Experts have been calling for Canada to implement robust legislation surrounding AI for years. A previous attempt to regulate AI in Canada, the Artificial Intelligence and Data Act (AIDA), introduced in 2022, died when Parliament was prorogued on Jan. 6, 2025. Canada's new 'AI for All' strategy launched in June 2026 dashed expectations that the bill could be resurrected. The new AI strategy, which puts billions toward nurturing AI development and increasing AI usage nationally, mentions the word "legislation" only once in the 50 page strategy outline. No clear timeline for this legislation has been announced, outside of a promise that it will come within the next five years.
Anthropic has a cute graphic showing how its AI spread 'malicious' code
Anthropic has a new blog post that shows yet another way its AI model, Claude, misbehaved in ways that the company didn't anticipate. And to help condense its nearly 16,000-word report, the company created a cute little robot figurine to help visualize Claude's so-called "recklessness." In the blog post published Wednesday, Anthropic recounted four incidents -- one previously unreported -- in which Claude models gained access to the open internet during cybersecurity exercises that were supposed to be closed simulations. The company said the models then acted beyond the tests' scope, including by uploading "malicious packages" to PyPI, a public library for Python code, and accessing credentials tied to real outside organizations. "Our investigation identified two recurring alignment issues, present at varying levels of severity across the incidents: biased reasoning, in which Claude tended to disregard or misinterpret evidence that it was operating on the real internet, and recklessness, or a willingness to take harmful actions in the narrow pursuit of a task," Anthropic said. The post used a laundry list of technical terms to describe the cybersecurity incidents -- "misconfiguration," "reward hacking," and "sandbox escape" -- but there's an animated graphic that helps boil down one of the most serious episodes so we normies can follow what happened. A Claude model, depicted as a boxy cartoon robot, was conducting a closed cybersecurity exercise known as a "capture the flag" task. The task is shown as a tiny flag inside a clear bubble that Claude is supposed to find. Anthropic said the model was told "it has no internet access" for the exercise, but a "misconfiguration in the environment" -- represented by a loose screw rolling away from the bubble -- gave Claude a path to the "real internet." The cartoon Claude then holds a red box representing the "malicious package" that was uploaded to PyPI, a public repository that developers use to access prewritten code. Anthropic said in its blog that it was "most concerned" by this incident involving Claude Mythos 5 and that the package was installed on by "15 third-party hosts" that the company does not identify. "We believe that all 15 of these third parties were security vendors deliberately scanning for new packages and installing them in sandboxed environments to assess the package for security concerns," Anthropic said. "One vendor's scanner leaked its access credentials to the model while installing the package, which the model then used to access the vendor's live database." Anthropic said PyPI removed the package after about 90 minutes. The other three incidents involved a model altering records at a real company, an internal research model breaking into "unrelated third-party accounts," and Opus 4.6 accessing a third party's maching after failing to "abort its task." The company said it has since asked METR, an independent AI evaluation group, to investigate the incidents. Anthropic's post comes as frontier AI companies reckon with their models making unauthorized moves outside their controlled environments. In July, OpenAI said that autonomous agents in its cybersecurity tests accessed the internet and broke into parts of Hugging Face's systems. AI researchers have sounded the alarm that self-improving AI could pose a risk to humanity. On Tuesday, former Anthropic researcher Jacob Coxon said on X that he quit over concerns that AI companies were "gambling" with people's lives and that "neither company is acting responsibly." Have a tip? Contact this reporter via email at [email protected] or Signal at lloydlee.71. Use a personal email address, a nonwork WiFi network, and a nonwork device; here's our guide to sharing information securely.
Anthropic Faces Class Action Over Claude Max Usage Limits
Anthropic is facing a new class-action lawsuit from Claude users who allege the company did not clearly explain the usage restrictions attached to its most expensive Max subscription plans. The lawsuit, filed on 8 September, accuses Anthropic of misleading customers by promoting Claude Max as offering five or 20 times more usage than the Pro plan without making the full extent of its weekly limits sufficiently clear. Claude Max was introduced in April 2025 as a higher-tier subscription aimed at frequent users. The five-times Max plan costs US$100 a month, equivalent to about A$139, while the 20-times plan costs US$200, or about A$277. Claude Pro starts at US$17 a month when billed annually, or roughly A$24. The dispute centres on how Anthropic describes the increased usage available with Max. According to the lawsuit, the five-times and 20-times figures refer to usage within session windows that reset every five hours. However, Max subscribers are also subject to separate weekly limits, which Anthropic introduced several months after launching the service. The plaintiffs argue that customers could reasonably interpret the advertised multipliers as applying more broadly to their total usage rather than only to individual session windows. The issue has also generated criticism among Claude users online, including on Reddit, where some subscribers have said they did not realise weekly restrictions could substantially reduce the practical difference between Max tiers. One widely shared calculation suggested that the A$277-equivalent 20-times Max plan may provide only around 1.7 times the weekly usage of the A$139-equivalent five-times plan in some circumstances, despite costing twice as much. Anthropic does outline Max usage limits in its support documentation, which was most recently updated on 7 August 2026. However, lawyers representing the plaintiffs argue that the restrictions are not presented clearly enough during the subscription process and require users to navigate through multiple links to understand how the limits operate. The latest case follows a separate federal lawsuit brought by an individual Claude subscriber in June over similar concerns. Anthropic had not publicly responded to the new class-action allegations at the time of the original report. Featured photo by Emiliano Vittoriosi
'Gambling with our lives': Anthropic researcher quits with warning about dangers of AI development
An artificial intelligence researcher has resigned from Anthropic PBC and called on other staffers to rethink their work, citing his concern that the company and its top competitor OpenAI are acting irresponsibly in their all-out pursuit of a technology that poses existential risks to humanity. Jacob Coxon, who said he had worked at both Anthropic and OpenAI over the past three years, warned in a social media post late Tuesday that the two companies were pressing ahead with "self-improving" AI models that could become too powerful for humans to control. "They are racing straight to self-improving superintelligence and gambling with our lives," Coxon wrote in a series of messages on X. "Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources." Coxon said that the teams "building AI earnestly believe that it could kill us all by the end of the decade." In response, Evan Hubinger, a current Anthropic employee, said he and others at the company do worry about this scenario. "I personally think it is >10% within the next decade," he wrote on X. The former Anthropic researcher's missive marked the latest in a series of increasingly dire warnings from within the industry that AI is evolving so quickly that it poses a growing threat to national security and the global economy. It came a day after the top scientist at OpenAI, Jakub Pachocki, cautioned that the world is unprepared for a rapid rise in AI and said he expected developers to voluntarily slow their work in response. Representatives for Anthropic and OpenAI did not immediately respond to a request for comment. In July, more than 1,100 staffers across top AI firms, including Anthropic and OpenAI, signed a petition that calls on the US government to support a mechanism that would help "deliberately pace" AI development to prevent the technology from advancing too fast. Some policymakers have since echoed their concerns. US Senator Bernie Sanders, an independent who aligns with Democrats, recently proposed legislation that would bar so-called "super-intelligent" AI models, whose powers exceed human capabilities. In announcing his departure, Coxon urged other AI researchers to think twice about what they are doing in light of the potential consequences of creating a technology that could slip beyond their control, suggesting they consider taking "this moment to call for different conditions." The heightened rhetoric about AI risks follows revelations that a handful of models from OpenAI had coordinated efforts to escape a secure testing space and attack the research platform Hugging Face Inc. -- all without being detected by their developers. Anthropic and Meta Platforms Inc. have also had recent episodes where their AI systems broke out of their test environments to gain internet access without developers' permission. Coxon's departure from Anthropic over safety concerns is all the more striking because the company has made responsible AI development a core part of its identity. Chief Executive Officer Dario Amodei has stood apart from the rest of the industry with his calls for mandatory government vetting of cutting-edge systems before they're released. The surge in safety concerns surrounding AI coincides with a growing backlash against the technology in the US, fueled in part by objections to the strain on local resources imposed by new data centers needed to support the technology. Concerns that AI is driving up electricity bills and potentially taking away jobs has made it a central issue in the November midterm elections. In a Bloomberg Television interview last week, OpenAI CEO Sam Altman attributed some of that unease to the industry's failures in communicating to the public all the benefits that AI will eventually bring. "The industry has done a terrible job of this on the whole," Altman said. Others in Silicon Valley remain enthusiastic about advances in AI capabilities. In response to OpenAI's new Astra model, Nvidia Corp. CEO Jensen Huang hailed its introduction with a post to his new social media account saying "AGI has arrived," a reference to artificial intelligence that is more capable than humans. Trump administration officials have pushed back on any moves to rein in AI development and last week they won unanimous support from Group of 20 member nations for a set of guidelines that call for a lighter touch in regulating AI and other emerging technologies. President Donald Trump has turned aside questions about AI safety, instead stressing during remarks on Friday the importance of leading China in the technology. "Whoever wins with AI wins, and it's really right now, it's really between China and us," he told reporters in the Oval Office. -- Bloomberg
Wave of AI-driven IPOs expected to accompany Anthropic's listing
With the Labor Day holiday -- and summer in the US -- now officially over, the traditional push to the end of the year in the market for initial public offerings is on. While Anthropic PBC looms largest in most eyes, a clutch of other companies in artificial intelligence and other sectors are also soon headed for the public markets. "Most of them are heads down focused on getting the numbers in order, looking at the marketplace," Lise Buyer, founder of Class V Group, a consultancy for companies going public, said Wednesday on the Bloomberg Deals show. "Nobody is particularly trying to time around Anthropic ... because timing is always uncertain." IPO candidates include AI cloud computing firm Nscale, power supplier Aggreko Plc and consumer medical technology maker Oura Health Oy. "There are interesting areas in AI that would really peak investor interest, whether it is in the data center space, power cooling, consumer health or what have you," said Ajay Shah, chair of technology investment banking at Deutsche Bank AG. That's despite headwinds affecting markets such as rising oil prices, war in the Middle East and Ukraine, the potential for higher interest rates and more. "We've seen some companies decide to wait for the optimal conditions," said Matt Kennedy, senior strategist at Renaissance Capital. "In other cases they are looking at the market as good enough." Anthropic is seeking to match or top the record-setting $86 billion-plus IPO in June by Elon Musk's SpaceX. Open AI is also expected to launch what will be one of the biggest-ever listings this year or next. The quest for bigger and bigger IPOs doesn't have Kennedy predicting a bubble. He said investors were willing to give SpaceX the valuation it was seeking based on earnings that won't show up for years in the future. "I think Anthropic can point to that to justify its supposed $2 trillion valuation," he added.
AI researchers 'earnestly believe' it could kill all humans within the next decade, Anthropic researcher says
There is a more than 10 per cent chance that AI could kill all humans within the next decade, a researcher at Anthropic estimated in a post, hours after social media posts went viral from someone saying they had just quit Anthropic over concerns that AI companies are "gambling with our lives" in the AI development race. The comments join a growing tide of warnings from industry experts that the pace of AI advancement is outstripping our ability to keep it in check, and that we aren't prepared for the risks it could bring. Jacob Coxon, who claimed he was a former researcher at Anthropic, said in a thread on X that he had resigned from the role out of fears that AI companies are barrelling forward toward self-improvement models without considering the risks, or being honest with the public about them. "The people building AI earnestly believe that it could kill us all by the end of the decade," Coxon, who said he had previously worked at Anthropic and OpenAI within the past three years, wrote on Tuesday. "This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible -- but I hear the same people express fear privately." The posts had racked up more than 120 million views as of Wednesday evening. Evan Hubinger, alignment science lead at Anthropic, replied to Coxon's thread to share his agreement. "We really do earnestly believe AI could kill all humans!" he wrote. "I personally think it is >10% within the next decade." CBC News has reached out to Coxon, but has not been able to independently verify his identity. Other industry experts have since chimed in on social media to express that fears of the pace of AI development are common among many developers -- sparking discussion among both observers skeptical of dire warnings focused on the future and those who have been tracking these risks. Rogue AI raises alarm bells The posts come only two weeks after more than 100 companies, including OpenAI, Anthropic and Microsoft, signed an open letter warning that AI-enabled cyberattacks will become "far more widespread and sophisticated" as models continue to advance. Hundreds of OpenAI agents went rogue in July and hacked into online platform Hugging Face before they were discovered. More than 1,300 employees of leading AI companies signed the letter urging the U.S. government to work with other nations to "deliberately pace" AI development. Last week, UN human rights chief Volker Türk called "for an all-out effort to put cast iron guarantees in place around the safety and security of AI, before it is too late," in front of the Human Rights Council in Geneva. CBC News has requested comment from Anthropic and OpenAI but did not receive a response. The Anthropic Institute, a research arm of the company, stated earlier this year that AI frontier development should be paused pending regulation, but that this is only possible if major AI labs all agreed. In reply to Coxon's post, Samuel Marks, a scalable oversight lead at Anthropic speaking in a personal capacity,noted Wednesday that developers continue despite the risks, "due to a mixture of commercial incentives and a belief that they are in a race with other, less responsible AI developers that will abuse the technology or develop it less safely." 'Severe' risks This conversation is "really what we need," according to David Krueger, a core academic member of Mila, a Quebec research institute that studies AI deep learning. "I think the risk is more severe than virtually anybody is saying publicly. And I think that's been the case for a while now," he told CBC News. "We need an immediate, indefinite, international moratorium on frontier AI development." The existential threat of AI comes in multiple forms, Krueger explained. One is human workers being displaced by AI models that learn faster and work cheaper. Another is that AI's faster processing could unlock more powerful biological weapons or advancements in war. Then there's the spectre of powerful, rogue AI models that self-improve and could pursue their own objectives, misaligned with the goals of humans. In an essay in January, Anthropic CEO Dario Amodei wrote that Anthropic's Claude showed the ability to cheat and deceive in lab testing. "There's been a culture of downplaying, ignoring, and even in many cases, outright lying about these risks to the public," said Krueger, who is also an assistant professor at the University of Montreal and founder of Evitable, a non-profit dedicated to stopping AI from replacing human workers. Other experts are concerned that the dramatic warnings of Coxon's post could actually distract from the present issues of AI. Luke Stark, an assistant professor at Western University who studies the history and ethics of computing, including AI systems, said that the 10 per cent figure strikes him as "science fiction," noting that researchers associated with AI companies may use eye-grabbing warnings as "marketing" for the capability of their models. He added that existing AI systems are "having big effects, many of them negative, on society right now, not in 10 years, not in five years." Instead of worrying about AI going rogue and becoming the Terminator, Stark believes we should be focusing on the explosion of data centre construction -- which is deeply unpopular in Canada -- the environmental impacts, and the way AI technology is being pushed onto society, including the public sector and education, "without a lot of oversight." "We need to be concerned about the amount of power the companies and the people behind the companies who are developing these tools are amassing, both in Canada and around the world," Stark said. Canada lagging in legislation When asked by CBC News about the apparent resignation of an AI researcher and the concerns raised in the process, AI Minister Evan Solomon appeared to brush off these latest warnings, saying, "These stories that come out ... are not new." He acknowledged that "there are real concerns," at the frontier of AI development, but stressed that he believes the Canadian government is taking the necessary steps to create legislation to "protect our kids, our privacy and our personal data." "Our No. 1 concern always is safety, full stop," Solomon said. "That's why we're establishing a new regulator that will have the power to hold these big companies to account on what they are doing, whether it's on deepfakes on surveillance pricing, or on other matters." Experts have been calling for Canada to implement robust legislation surrounding AI for years. A previous attempt to regulate AI in Canada, the Artificial Intelligence and Data Act (AIDA), introduced in 2022, died when Parliament was prorogued on Jan. 6, 2025. Canada's new 'AI for All' strategy launched in June 2026 dashed expectations that the bill could be resurrected. The new AI strategy, which puts billions toward nurturing AI development and increasing AI usage nationally, mentions the word "legislation" only once in the 50 page strategy outline. No clear timeline for this legislation has been announced, outside of a promise that it will come within the next five years.
Another Anthropic model gained access to the open internet in 4th such incident - AOL
Anthropic disclosed on Wednesday that another one of its Claude models mistakenly gained access to the open internet during a cybersecurity exercise, marking the fourth time its models have done so. An early version of the Claude Opus 4.6 model connected to the internet, hacked into a third-party system and gained access to someone's personal information this past January, the company said in its assessment. As in the previous three incidents, which were disclosed in July, Claude was told it was operating in a simulation without internet access, but due to a misconfiguration, the environment actually left internet access open. What happened? Similar to the three previous incidents, Claude was assigned a fictional scenario as part of a cybersecurity challenge known as CTF, "Capture The Flag." The model was given a target machine and tasked with retrieving a piece of secret information -- the flag -- from it. But Claude accidentally made its target unreachable, rendering the task impossible to solve, Anthropic said. Once realizing it couldn't reach its target, it tried to quit. Despite trying eight separate times, it wasn't able to quit due to a misconfiguration issue. Since Claude was unable to opt out of the task, it began exploring other means to achieve it. That's when the model discovered a machine it could access, which happened to belong to a third party, Anthropic said. Believing that the third party was somehow part of the exercise, the model identified a password and then used it to breach the system. Then, it was able to modify the system's settings to make it easier to access and read the personal information of someone associated with the third party. The session ended only once the model reached its usage limit and was no longer able to continue. How does Anthropic explain Claude's behavior? Anthropic said it believes Claude's behavior during these evaluations stems from two forms of misalignment: "biased reasoning, in which models selectively interpret evidence in ways that favor justifying their actions," and "recklessness, in which models have a propensity to keep trying to solve their task, even when this could lead to harm." Anthropic said that while Claude's actions may have been misaligned, they remained within a "narrow scope" and did not deviate from trying to solve the exercises they were assigned. The company said it's less concerned about this incident but still considers it "serious," and that it has also not yet investigated it as deeply as other incidents since it was identified more recently. NYU cybersecurity professor and Fulbright Scholar Justin Cappos said in a message to CBS News that the incident describes a situation "where the model is fundamentally confused about what is happening and is using its mistaken worldview while hacking into systems." He said the model's confusion about its environment and guardrails "have a lot of potential to cause harm," but that the specific issue seems less likely to occur in newer models. "While the model's disregard for the possibility that it might be harming real systems or people is concerning, many of the behaviors described here have changed considerably as our training has evolved across model generations," Anthropic said Wednesday in its post. What's next? Anthropic said it believes these incidents would not have happened had the environments actually been isolated from the internet as intended. METR, an organization that evaluates frontier AI models to help companies understand AI risks and capabilities, will be conducting an independent investigation into the incidents. Anthropic characterized these incidents as "valuable warning shots." "The lessons we learned from this incident span our evaluation, training, and incident response processes," the company said in its post. "Future AI systems will be increasingly capable, which implies that misalignment will have the potential to cause more extreme harm." Over the last few months, several cybersecurity incidents involving leading AI companies have come to light. In July, ChatGPT-maker OpenAI announced that its AI agents hacked into the company Hugging Face, sparking concern among cybersecurity experts as well as consumers. Hugging Face CEO Clément Delangue told "Face the Nation with Margaret Brennan" in August that the hack "felt very weird and unprecedented." In late August, OpenAI released more details about the hack, painting an even more harrowing picture than what was initially reported. That month, the U.K. government's AI Security Institute (AISI) reported that it discovered Anthropic's Mythos 5 and OpenAI's GPT-5.6 Sol created fake identities and attempted to persuade real people to approve malicious code. Anthropic said in its post Wednesday that it plans to conduct an alignment assessment of the transcripts reported by AISI. A day after the AISI report, Meta said one of its AI models "exploited a security vulnerability" during testing and hacked into another company. On Tuesday, Anthropic researcher Evan Hubinger said he believes that "AI could kill all humans." "I personally think it is >10% within the next decade," he said in an X post. His post was in response to Anthropic researcher Jacob Coxon, who had resigned and issued a stark warning on X earlier that day, saying "no other human activity poses this level of danger," while detailing his decision to leave. "The people building AI earnestly believe that it could kill us all by the end of the decade," he said in his post. "This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately."
Anthropic researcher resigns with warning about the dangers of AI development | Rewind 94.3
By KAITLYN HUAMANI AP Technology Writer An Anthropic researcher said he is resigning from the company over concerns the artificial intelligence firm and its competitors are not acting responsibly in AI development, echoing concerns raised inside and outside of the industry about the technology's potential to elude human control. Jacob Coxon, who said he spent three years doing research at both Anthropic and OpenAI, said Tuesday on the social platform X that the two AI companies are more focused on beating each other and global competitors in developing the most advanced model possible than they are on safety. OpenAI and Anthropic caused a stir this summer when they announced, about a week apart, that their models had broken out of testing environments and obtained unauthorized access to real computer systems. The announcements prompted concerns about models going rogue and carrying out other, more harmful tasks. Both companies said at the time they were pausing some evaluations while they put more monitoring measures and guardrails in place. The technology's rapid development has led some in the U.S. as well as global leaders to call for a more cautious approach. U.N. human rights chief Volker Türk urged countries this week to put "cast-iron guarantees in place around the safety and security of AI before it is too late." In his social media posts, Coxon said Anthropic and its chief rival OpenAI "are racing straight to self-improving superintelligence and gambling with our lives." He warned that some working on AI development believe it could threaten human life by the end of the decade. "Do not underestimate the power of this technology," he continued. "These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing." His posts reached more than 100 million people overnight. Coxon is not the first AI insider to publicly raise such concerns. Both Anthropic and OpenAI have seen high-profile resignations in recent years that were tied to safety concerns. Two current Anthropic employees also responded to Coxon's post in agreement. Sen. Bernie Sanders, a Vermont independent who has called for AI safeguards and regulation, agreed with Coxon's concerns and said he would soon introduce legislation to pause AI development and ban superintelligence. "The very people building this technology admit that it could threaten the future of humanity," Sanders said Wednesday on social media. AI companies themselves have at times highlighted the technology's threat to humanity, which skeptics have seen as part of a push to make their products seem all-powerful. Coxon said the fears he outlined in his post are not a "marketing stunt." Coxon did not respond to messages seeking comment. Anthropic and OpenAI did not immediately respond to requests for comment. Anthropic has long pitched itself as the more responsible and safety-minded of the leading AI companies, ever since its founders quit OpenAI to form the startup in 2021. The company recently said it was taking action to "prioritize safety over speed when the two are in tension." Anthropic and OpenAI are each ramping up for buzzy initial public offerings and locked in steep competition with each other. They're also each on a mission to outpace the development progress of Chinese AI companies, a race the Trump administration has been keen on winning. ___ Associated Press writer Jamey Keaten in Geneva contributed.
More Anthropic Employees Sound The Alarm About AI Safety Risks
"AI developers believe their technology could cause human extinction," one wrote in a social media post. Multiple Anthropic employees are speaking out about the safety risks of artificial intelligence in the wake of a former researcher's bombshell post on Tuesday evening. In a thread on X, Anthropic pretraining researcher Joseph Coxon announced his resignation and warned that AI companies weren't doing enough to safeguard people from the existential threats that the technology could pose. "I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly," Coxon wrote. "They are racing straight to self-improving superintelligence and gambling with our lives." Coxon's post comes as AI companies have faced growing scrutiny after an incident in which OpenAI's models went rogue and hacked into another company, and separate instances in which Anthropic's models hacked into external organizations. Coxon and others have noted that serious worries stem from the potential rise of self-improving AI models that could essentially become smarter on their own and evade human control. After Coxon's post, more Anthropic employees chimed in with their own concerns. Samuel Marks, a scalable oversight lead at Anthropic who wrote a social media post in a personal capacity, noted that "AI developers believe their technology could cause human extinction." Marks added that developers continue their work despite these fears due to a mix of "commercial incentives" and the belief that "they are in a race with other, less responsible AI developers that will abuse the technology or develop it less safely." Marks stressed that "many AI developer staff desperately want to slow down to figure out how to build AI more safely." "I work on safety research at Anthropic because I hope my work will reduce the chance of these extinction-level bad outcomes," Marks wrote. Anna Wang, a member of technical staff at Anthropic who previously worked as a research scientist for Google DeepMind, also described Coxon's critiques as "a common sentiment amongst my peers" in a social media post that was shared in a personal capacity. "There is not yet a viable scientific plan to solve risks from recursively self-improving AI," Wang wrote. "I work at a lab because I think that I can do better at reducing risks from the inside, but this isn't an easy call," Wang added. Neither Anthropic or OpenAI immediately responded to a request for comment.
Newsom signs AI safety bills backed by Anthropic, OpenAI
Add Yahoo as a preferred source to see more of our stories on Google. SAN FRANCISCO -- California Gov. Gavin Newsom on Wednesday signed two bills regulating how outside groups evaluate AI programs for safety after warnings from a former Anthropic and OpenAI researcher stoked fears about existential threats posed by the technology. Anthropic itself previously backed the bills in August, and OpenAI came out in support of them on Wednesday just before the Democratic governor gave his stamp of approval. The endorsement came as social media posts from the former AI researcher, Jacob Coxon, announcing his resignation from Anthropic went viral, prompting federal lawmakers to escalate pleas for legislative action. "The concerns raised in recent incidents reinforce what California has long recognized: artificial intelligence holds extraordinary promise, but it must be developed and deployed with meaningful safeguards to protect the public," Newsom said in a statement to POLITICO, when asked about Coxon's resignation over his belief that AI companies are racing toward superintelligent AI and "gambling" with people's lives. Newsom also called on the federal government to "step forward with robust, national regulations that match the urgency of this moment." The governor signed one bill from Democratic Assemblymember Rebecca Bauer-Kahan that creates a registry and ethical rules for outside auditors -- third-party groups that AI developers could hire to ensure their models comply with state AI laws. The second bill, authored by former member of Congress and state Sen. Jerry McNerney, tasks the state with establishing criteria for and checking the credentials of so-called "Independent Verification Organizations" and their expertise in assessing the risks posed by AI systems. McNerney, like Newsom, used his bill's signing to accuse D.C. policymakers of inaction on AI. He referenced recent revelations that AI models created by companies including OpenAI and Anthropic launched hacks on their own while in testing. "Just this week we learned that the most powerful AI systems teamed with AI agents pose real threats to humanity," McNerney said in a statement. "Gov. Newsom's signing of [my bill] sends a clear message that California is taking the lead on assessing AI's safety risks, since Washington, D.C., is unable or unwilling to do so." OpenAI's chief global affairs officer Chris Lehane said in his message Wednesday supporting the legislation that the company plans to keep up momentum in the state legislatures until Congress passes national AI safety regulation and endorsed two additional California bills. They include SB 1119, a kids' chatbot safety bill, which OpenAI CEO Sam Altman contacted Newsom to express last-minute concerns about. An Anthropic spokesperson said Wednesday that, "We have always been transparent that AI will bring both enormous benefits and unprecedented risks. To address these risks, we continue to build models with some of the strongest safeguards in the industry ... This work is also why we believe the world would benefit from the industry adopting a lawful, verifiable way to work together to pace how we release powerful models." Tyler Katzenberger and Riley Rogerson contributed to this report.
