The public version of o3 scored 10 percent on the same benchmark
It is speculated that the tested o3 model had more processing power
The incident raises concerns about the validity of benchmark scores
OpenAI told ARC Prize that the released o3 model is different from the one tested by the organisation
Photo Credit: Unsplash/Levart_Photographer
OpenAI's o3 artificial intelligence (AI) model, which was released last week, is underperforming on a specific benchmark. Epoch AI, the company behind the FrontierMath benchmark, highlighted that the publicly available version of the o3 AI model scored 10 percent on the test, a much lower value than the company's claim at launch. The San Francisco-based AI firm's chief research officer, Mark Chen, had said that the model scored 25 percent on the test, creating a new record. However, the discrepancy does not mean that OpenAI lied about the metric.
OpenAI's o3 AI Model Scores 10 Percent on FrontierMath
In December 2024, OpenAI held a livestream on YouTube and other social media platforms, announcing the o3 AI model. At the time, the company highlighted the improved set of capabilities in the large language model (LLM), in particular, its improved performance in reasoning-based queries.
One of the ways the company exemplified the claim was by sharing the model's benchmark scores across different popular tests. One of these tests was FrontierMath, created by Epoch AI. The mathematical test is known for being challenging and tamper-proof, as more than 70 mathematicians developed the test, and the problems are all new and unpublished. Notably, till December, no AI model has solved more than nine percent of the questions in a single attempt.
However, at the time of launch, Chen claimed that o3 was able to set a new record by scoring 25 percent on the test. External verification of the performance was not possible at the time, as the model was not available in the public domain. After o3 and o4-mini were launched last week, Epoch AI made a post on X (formerly known as Twitter), claiming that the o3 model, in fact, scored 10 percent on the test.
While a score of 10 percent also makes the AI model the highest ranking on the test, the number is less than half of what the company claimed. The post has led to several AI enthusiasts talking about the validity of the benchmark scores.
The discrepancy does not mean that OpenAI lied about the performance of its AI model. Instead, the AI firm's unreleased model likely used higher compute to get that score. However, the commercial version of the model was likely fine-tuned to be more power efficient, and in that process, some of its performance was toned down.
Separately, ARC Prize, an organisation behind the ARC-AGI benchmark test, which tests an AI model's general intelligence, also posted on X about the discrepancy. The post confirmed, “The released o3 is a different model from what we tested in December 2024.” The company claimed that the released o3 model's compute tiers are smaller than the version it tested. However, it did confirm that o3 was not trained on ARC-AGI data, even at the pre-training stage.
ARC Prize said that it will re-test the released o3 AI model and publish the updated results. The company will also re-test the o4-mini model, and label the prior scores as “preview”. It is not certain that the released version of o3 will underperform on this test as well.
OpenAI’s Sam Altman Reportedly Hints at New AI Device Being Developed With Jony Ive22 May 2025
Search
