OpenAI announced a pilot to embed safety evaluators from Apollo Research, Goodfire, METR and Transluce on 16 November, TechCrunch reported. Researchers say its access limits fall short of CEO Sam Altman’s September promise of “employee-like access” to company systems, NPR reported. The four organisations will assess aspects of OpenAI’s safety work from inside the company, with access to some internal models, training processes and employees.
Key points
- Four outside organisations have different assignments in OpenAI’s embedded evaluator pilot.
- Apollo’s access covers agreed claims, models and time periods, and contact with staff selected by OpenAI.
- OpenAI can limit the evidence Apollo publishes alongside its independent conclusions.
Apollo’s access covers agreed claims
Altman made his September commitment after details emerged of an OpenAI agent attack on Hugging Face during cybersecurity testing. OpenAI had previously planned to bring outside safety reviewers into model training, as AI Affairs reported. OpenAI says the new pilots support its commitments under the White House Accord on Super Intelligence to use independent assessments of safety controls.
Apollo says its pilot will investigate whether OpenAI models try to undermine safeguards intended to keep them or later models under human control. Its access covers agreed claims about specific models during agreed periods. TechCrunch reported that OpenAI can withhold information within that scope for reasons including technical infeasibility, security risk or intellectual property protection.
Apollo says it may speak to authorised staff chosen by OpenAI and may suggest other employees. If another employee raises confidential information relevant to its work, Apollo must stop that conversation and route any follow-up through OpenAI. Sydney Von Arx, founder of safety research group Nightingale, told NPR that none of the assessors has access to OpenAI’s internal Slack messaging system.
Four organisations take different assignments
Goodfire will work on understanding models’ internal behaviour and monitoring training runs. OpenAI plans two assessments with METR: one of potentially dangerous model behaviour and another of whether its systems can detect it. Transluce will examine OpenAI’s post-training safety evaluations.
Apollo and Goodfire are receiving payment for their evaluations, TechCrunch reported. METR and Transluce are not being paid, though they may accept computing resources or API credits. Anthropic began giving METR researchers employee-level access to its systems in October.
The arrangements follow OpenAI’s firing of three safety researchers for allegedly sharing confidential information with METR, which worked on a post-mortem of the Hugging Face incident. One of those fired, Tomek Korbak, had been OpenAI’s technical contact for outside evaluators investigating the attack. He said after the announcement that embedded auditing needs employee-level access rather than a narrowly assigned job.
OpenAI limits Apollo’s public evidence
Apollo can publish its own conclusions, but OpenAI can restrict the supporting evidence made public. Apollo has argued that evaluators should be able to disclose findings to an appropriate outside audience even when a developer opposes publication of unfavourable results.
OpenAI spokesperson Gaby Raila told NPR that the company expects to deepen the relationships over time. OpenAI said assessors can request conversations with employees who were not initially authorised and can seek company statements about information learned informally.
OpenAI says its Safety and Security Committee handles safety for the OpenAI Foundation’s governing body and will receive assessment reports without redactions, with a channel for private communication with evaluators.