Anthropic Lets an Outside Watchdog Work Inside Its Walls

Anthropic says it has given a small team from METR, a nonprofit that tests AI systems for risky abilities, inside access to its models and safety practices. Six METR employees now get company hardware, office space and internal tools, which Anthropic describes as roughly on par with what its own risk teams have. METR can then publish its own analysis of Anthropic’s risk levels, incidents and practices.

There are limits, and Anthropic is open about them. Customer data, legally privileged material and details of the model architecture (the technical blueprint of how the models are built) stay off the table. Anthropic can require redactions inside those categories and decides what falls into them, but it cannot veto publication. METR, for its part, keeps the right to say publicly how much access it really got. The team is walled off from other METR staff and works under enforceable agreements.

Anthropic frames this as the top tier of a three-tier system. Tier 1 evaluators run narrow investigations and Anthropic approves what they publish. Tier 2 can publish without approval, with the same kind of redactions. Tier 3, which is METR’s deal, adds employee-like access. Other partners get less: SecureBio is reviewing the chemical and biological weapons chapter of Anthropic’s August Risk Report, and Accenture and Faculty are working on red-teaming (deliberately trying to break a system to find weak spots) and safeguard testing. Early talks with Hugging Face have also begun.

Why do this at all? Anthropic argues that some safety-relevant information cannot be made public, because it could help attackers or is commercially sensitive. An outside group that can see it, and is free to say what it thinks, is a middle path between “trust us” and “publish everything.” The Decoder notes the move follows Anthropic signing the White House Accord on Super Intelligence, which includes working with outside reviewers. Anthropic says every frontier lab should do the same.

A fair caveat: this is a promise about access, not a result. We do not yet know what METR will find, and the redaction rules still leave Anthropic some say over what the public sees.

What this means for you: for most people, nothing changes today. But if you have wondered how anyone checks the claims AI companies make about their own safety, this is a concrete example of an answer. Watch for METR’s first published findings and whether it says the redactions were justified. That comment will tell you more than the announcement does.

Sources

• Anthropic: An update on independent evaluators
• The Decoder: Anthropic gives METR inside access