{"id":73859,"date":"2025-04-21T03:30:57","date_gmt":"2025-04-21T03:30:57","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2025\/04\/21\/openais-o3-ai-model-scores-lower-on-a-benchmark-than-the-company-initially-implied\/"},"modified":"2025-04-21T03:32:02","modified_gmt":"2025-04-21T03:32:02","slug":"openais-o3-ai-model-scores-lower-on-a-benchmark-than-the-company-initially-implied","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2025\/04\/21\/openais-o3-ai-model-scores-lower-on-a-benchmark-than-the-company-initially-implied\/","title":{"rendered":"OpenAI’s o3 AI model scores lower on a benchmark than the company initially implied"},"content":{"rendered":"

<\/a>\n
<\/p>\n

\n

A discrepancy between first- and third-party benchmark outcomes for OpenAI\u2019s o3 AI mannequin is raising questions about the company\u2019s transparency<\/a> and mannequin testing practices.<\/p>\n

When OpenAI unveiled o3 in December<\/a>, the corporate claimed the mannequin may reply simply over a fourth of questions on FrontierMath, a difficult set of math issues. That rating blew the competitors away \u2014 the next-best mannequin managed to reply solely round 2% of FrontierMath issues accurately.<\/p>\n

\u201cAs we speak, all choices on the market have lower than 2% [on FrontierMath],\u201d Mark Chen, chief analysis officer at OpenAI, said during a livestream<\/a>. \u201cWe\u2019re seeing [internally], with o3 in aggressive test-time compute settings, we\u2019re in a position to recover from 25%.\u201d<\/p>\n

Because it seems, that determine was possible an higher certain, achieved by a model of o3 with extra computing behind it than the mannequin OpenAI publicly launched final week.<\/p>\n

Epoch AI, the analysis institute behind FrontierMath, launched outcomes of its impartial benchmark assessments of o3 on Friday. Epoch discovered that o3 scored round 10%, effectively beneath OpenAI\u2019s highest claimed rating.<\/p>\n

\n

OpenAI has launched o3, their extremely anticipated reasoning mannequin, together with o4-mini, a smaller and cheaper mannequin that succeeds o3-mini.<\/p>\n

We evaluated the brand new fashions on our suite of math and science benchmarks. Ends in thread! pic.twitter.com\/5gbtzkEy1B<\/a><\/p>\n

\u2014 Epoch AI (@EpochAIResearch) April 18, 2025<\/a><\/p>\n<\/blockquote>\n

That doesn\u2019t imply OpenAI lied, per se. The benchmark outcomes the corporate revealed in December present a lower-bound rating that matches the rating Epoch noticed. Epoch additionally famous its testing setup possible differs from OpenAI\u2019s, and that it used an up to date launch of FrontierMath for its evaluations.<\/p>\n

\u201cThe distinction between our outcomes and OpenAI\u2019s is perhaps as a result of OpenAI evaluating with a extra highly effective inside scaffold, utilizing extra test-time [computing], or as a result of these outcomes had been run on a unique subset of FrontierMath (the 180 issues in frontiermath-2024-11-26 vs the 290 issues in frontiermath-2025-02-28-private),\u201d wrote<\/a> Epoch.<\/p>\n

According to a post on X<\/a> from the ARC Prize Basis, a company that examined a pre-release model of o3, the general public o3 mannequin \u201cis a unique mannequin [\u2026] tuned for chat\/product use,\u201d corroborating Epoch\u2019s report.<\/p>\n

\u201cAll launched o3 compute tiers are smaller than the model we [benchmarked],\u201d wrote ARC Prize. Typically talking, greater compute tiers might be anticipated to attain higher benchmark scores.<\/p>\n

\n

Re-testing launched o3 on ARC-AGI-1 will take a day or two. As a result of right this moment\u2019s launch is a materially completely different system, we’re re-labeling our previous reported outcomes as \u201cpreview\u201d:<\/p>\n

o3-preview (low): 75.7%, $200\/job
o3-preview (excessive): 87.5%, $34.4k\/job<\/p>\n

Above makes use of o1 professional pricing\u2026<\/p>\n

\u2014 Mike Knoop (@mikeknoop) April 16, 2025<\/a><\/p>\n<\/blockquote>\n

OpenAI\u2019s personal Wenda Zhou, a member of the technical employees, said during a livestream last week<\/a> that the o3 in manufacturing is \u201cextra optimized for real-world use circumstances\u201d and pace versus the model of o3 demoed in December. Consequently, it could exhibit benchmark \u201cdisparities,\u201d he added.<\/p>\n

\u201c[W]e\u2019ve finished [optimizations] to make the [model] extra value environment friendly [and] extra helpful on the whole,\u201d Zhou mentioned. \u201cWe nonetheless hope that \u2014 we nonetheless suppose that \u2014 it is a a lot better mannequin [\u2026] You received\u2019t have to attend as lengthy whenever you\u2019re asking for a solution, which is an actual factor with these [types of] fashions.\u201d<\/p>\n

Granted, the truth that the general public launch of o3 falls wanting OpenAI\u2019s testing guarantees is a little bit of a moot level, because the firm\u2019s o3-mini-high and o4-mini fashions outperform o3 on FrontierMath, and OpenAI plans to debut a extra highly effective o3 variant, o3-pro, within the coming weeks.<\/p>\n

It’s, nevertheless, one other reminder that AI benchmarks are finest not taken at face worth \u2014 notably when the supply is an organization with providers to promote.<\/p>\n

Benchmarking \u201ccontroversies\u201d have gotten a typical incidence within the AI business as distributors race to seize headlines and mindshare with new fashions.<\/p>\n

In January, Epoch was criticized<\/a> for ready to reveal funding from OpenAI till after the corporate introduced o3. Many teachers who contributed to FrontierMath weren\u2019t knowledgeable of OpenAI\u2019s involvement till it was made public.<\/p>\n

Extra lately, Elon Musk\u2019s xAI was accused<\/a> of publishing deceptive benchmark charts for its newest AI mannequin, Grok 3. Simply this month, Meta admitted to touting benchmark scores for a model of a model that differed from the one the company made available to developers<\/a>.<\/p>\n

Up to date 4:21 p.m. Pacific: Added feedback from Wenda Zhou, a member of the OpenAI technical employees, from a livestream final week.<\/em><\/p>\n<\/div>\n