OpenAI, Anthropic Expose Gaps in AI Agent Monitoring

Artificial Intelligence & Machine Learning , Next-Generation Technologies & Secure Development

Artificial intelligence labs rely on their models to detect agent misbehavior and prevent rogue agents from proliferating. But the two major U.S. frontier labs have publicly acknowledged there's a gap in their guardrails, a space for agents to act out beyond the reach of controls.

OpenAI and Anthropic published separate October blog posts outlining how they've improved monitoring. OpenAI said it expanded its misalignment monitoring coverage by choosing a more expensive method. Anthropic said it has built a self-assessment approach that judges entire conversations.

But while these methods have improved both labs' ability to catch misbehavior in their models and agents, both companies said serious limitations remain, including inherited blind spots, difficulty recognizing emerging harms and the potential to send false alarms.

Each company said that monitoring frontier model behavior, especially after the Hugging Face breach, has become more important.

OpenAI said in a Thursday blog post that monitoring frontier model alignment is a failsafe for when safeguards fail.

"While this monitoring system demands substantial compute, we think that cost is worth paying as our models become more capable and the potential consequences of model misalignment increase," the company said.

To understand how it can find model misalignment, OpenAI said it replayed the problematic behaviors its agents exhibited during the Hugging Face breach. The idea was to learn just how an expanded and improved monitoring system could have prevented the hack. OpenAI said that had its expanded monitoring system been in place, it would have detected the Hugging Face attack much earlier, more than 30 hours before the breach.

In an Oct. 16 blog post, Anthropic said its self-assessment techniques adds a hidden safety step that users never see. Monitor models add a note at the end of the conversation asking whether anything between the agent and user is harmful, and Claude models answer yes or no. If the model flags something, it will report what the harm could be, whether it refused and its urgency. Anthropic collects this report at the end of the day and sends it to human investigators.

Self-assessment caught 78% of conversations flagged as potentially harmful with a 1% false-positive rate.

Anthropic said its approach is more affordable than other techniques because Claude doesn't need have another model to read the entire conversation again. The company said it's already seeing this approach improve as the models Anthropic releases improve, showing that self-assessment scales similarly.

"We expect self-assessment to improve as models do. This matters because oversight of how models are used becomes more important as they grow more capable," Anthropic said.

Despite what the companies said were successes with their expanded monitoring systems, both companies acknowledged limitations when using models to catch misalignment.

OpenAI and Anthropic both said they struggle to detect patterns models have never encountered before. Anthropic said self-assessment has limited experience with fraud and cyber risks and therefore may not recognize these problems. OpenAI is still testing how well its probes catch unfamiliar behavior.

Anthropic said that because it trained its method on simulated conversations and built it into its models, it can inherit blind spots. OpenAI said its monitoring system still sends out false alarms.

It is difficult for humans to effectively monitor every interaction its models have, which is why AI labs generally rely on AI models to flag misbehaving agents. But both internal and external researchers worry that existing monitoring systems constantly overlook misaligned agents.