OpenAI publishes internal perplexity scores

On Nov. 22, OpenAI chief scientist Jakub Pachocki published log-perplexity figures the company tracks on a fixed snapshot of its internal codebase: about 0.99 for GPT-3.5, about 0.91 for GPT-4, losses below 0.8, and about 0.76 for a model he said had begun producing mathematics results he called groundbreaking. He proposed that frontier labs agree not to cross shared perplexity limits until a concrete monitoring step, such as a successor to chain-of-thought monitors, and he said further capabilities should be expected around 0.68 to 0.7. The scores are OpenAI's and have not been independently measured, and the post does not name the model at about 0.76.

/ Sources

/ About this story

Compiled by Venture Atlas from the sources above, using automated AI-assisted research. This is a summary of reporting published elsewhere, not original reporting - follow the source links for the full account. See our editorial standards.

Something wrong here? Email apostructura@gmail.com and we will fix it.

/ Related