Latest
AI Affairs, home

Monday 23 November 2026

Technology

OpenAI publishes model prediction scores and proposes shared pacing limits

Chief scientist Jakub Pachocki wants frontier labs to agree on thresholds they would not cross until tools for monitoring more capable models catch up.

OpenAI's headquarters at 1515 Third Street, with palm trees and overhead wires
Photo: Coolcaesar, CC BY 4.0, via Wikimedia Commons (cropped)

OpenAI chief scientist Jakub Pachocki published an essay titled Perplexity and Intelligence on 22 November 2026, sharing the company’s internal model-prediction scores and proposing shared limits on how far frontier labs develop their models before monitoring tools improve, Explainx reported.

Key points

  • OpenAI shared a log-perplexity curve spanning GPT-3.5 through more recent models, measured against a fixed set of internal code.
  • Pachocki proposes limits tied to prediction scores and other measurements, with monitoring milestones required before labs move past them.

OpenAI’s fixed code snapshot

Perplexity scores a model’s success at guessing the next small unit of text, as though it were filling in a page one piece at a time without having seen it before. OpenAI’s curve uses log-perplexity measured on a fixed snapshot of its internal code, so each model faces the same prediction material. A lower score means its predictions are sharper.

On the published curve, OpenAI associates GPT-3.5 with a log-perplexity of about 0.99 and GPT-4 with about 0.91. Pachocki associates those stages with fluent conversation and productive chain-of-thought reasoning, respectively, and says the capabilities were not predictable from the thresholds in advance.

Pachocki’s 0.76 mathematics account

An internal model that reached about 0.76 log-perplexity began resolving longstanding open mathematics problems, according to Pachocki’s account reported by OfficeChai. Pachocki warns that research into pretraining algorithms could see an avalanche of advances as soon as 2027, without a matching acceleration in alignment work.

Pachocki’s proposed perplexity limits

Pachocki proposes that frontier labs publish comparable measures of pretraining progress where possible, agree on evaluations and safety testing, and set initial limits tied to perplexity and companion measurements. Sam Altman, OpenAI’s chief executive, said he hoped the field could agree to “common limits”.

The proposal faces measurement problems Pachocki acknowledges: scores from different labs are difficult to compare without an agreed dataset, perplexity is not a score for tasks people perform, and reasoning techniques can add capability without lowering a base model’s loss. He proposes that labs hold to an agreed limit until monitoring reaches a concrete milestone, giving a successor to chain-of-thought monitors as an example.

Sources

Topics: Foundation models, Safety

What should we cover? Tell us

The weekly briefing

The AI Affairs weekly briefing: the week's 10 most important AI stories in business, government and markets, every Monday. We'll email you a link to confirm. Every email has a one-click unsubscribe. Privacy policy