Benchmaxing: Why AI Benchmark Winners Can Disappoint

A new AI model beats Claude or ChatGPT on a launch chart. You try it on your own work. It misses an instruction, breaks the code, or needs more help than the model you already use.

How can both things be true?

A benchmark win can be real and still tell you too little about the work you need done. Benchmaxing is an informal name for chasing high benchmark scores, sometimes without matching gains on tasks outside the test.

The details matter. A lab might promote a special model version, focus on a narrow skill, or use a more generous test setup. Here are examples that show how those gaps arise.

Meta's Llama 4 score came with a catch

When Meta announced Llama 4 in April 2025, it said Maverick beat GPT-4o and Gemini 2.0 Flash across several benchmarks. It also highlighted a score of 1417 on LMArena, where people compare answers from different models.

But Meta's announcement attached that score to an experimental chat version. A reader choosing the released model needed to notice that distinction. The promoted score was not a result for the standard released version. Meta's Llama 4 announcement

Separate research found another reason to examine Arena results closely. The authors of The Leaderboard Illusion identified 27 private Meta model variants being tested before the Llama 4 release. Their experiments showed how testing several versions and reporting the best result can raise a leaderboard score. That finding describes a weakness in the process; it does not tell us exactly how much of Maverick's score came from it. The Leaderboard Illusion

For someone choosing a model, the first question is simple: Can I use the exact version that earned this score?

A small DeepSeek model beat bigger names—until the test changed

Researchers behind LiveCodeBench found a striking example in their 2024 study. DeepSeek-Coder's small 1.3B instruction model beat Gemini Pro and Claude Instant on HumanEval, a coding benchmark. But it performed much worse than those models on the easy part of LiveCodeBench, which uses problems drawn from programming contests. LiveCodeBench research

The researchers described this as a sign of possible overfitting: getting good at a particular test without improving as much on other problems. It raises a real concern about how broadly the result applies. It does not, by itself, prove that anyone deliberately copied test answers.

These were older models, not today's Gemini or Claude. Still, the example shows why “beats a leading lab” needs a second question: On which test?

Even an honest results table can tell different stories. DeepSeek's original V3 report put it ahead of Claude 3.5 Sonnet on LiveCodeBench, but behind on SWE-bench Verified, which tests fixes to existing software.

Test in DeepSeek's V3 report	DeepSeek V3	Claude 3.5 Sonnet
LiveCodeBench, with reasoning before the answer	40.5%	36.3%
SWE-bench Verified, software issues resolved	42.0%	50.8%

These are DeepSeek-reported results for the original V3 and the October 2024 Sonnet version. Compare the models within each row; the two rows measure different tasks. DeepSeek's published results

DeepSeek published both wins and losses. This table is evidence of different strengths, not evidence of cheating. A claim that V3 was simply “better at coding” would leave out the part a reader fixing software might care about most.

Claude's score rose when the computer limits changed

The setup can also change the result without changing the model.

In February 2026, Anthropic reported an experiment on Terminal-Bench 2.0, a test of computer-based tasks. It kept the Claude model, tasks, and testing software the same while changing the computer resources available.

The gap between the strictest and most generous setups was six percentage points. Extra resources reduced crashes and, at higher levels, let the model complete tasks it could not solve under tighter limits. Anthropic's experiment

That is a meaningful gap on a leaderboard. For scale, a move from 50% to 56% is six percentage points; those are example numbers, not Anthropic's reported endpoints.

Anthropic disclosed this problem. Its finding shows why a comparison needs the test conditions alongside the score. If one model gets more memory, time, or attempts, the chart may partly measure that advantage.

Does a strong score mean the model saves you time?

This is the question a launch chart often leaves open.

In a study published in July 2025, METR followed 16 experienced developers working on 246 tasks in software projects they knew well. Tasks were assigned to allow or disallow AI help. The developers mainly used Cursor with Claude 3.5 or 3.7 Sonnet.

With AI allowed, they took 19% longer on average. Even afterward, they believed AI had made them faster. METR's 2025 study

That result applies to those developers, tasks, and early-2025 tools. In February 2026, METR said newer tools likely helped more, but problems with who joined its follow-up study made the size of the benefit hard to measure. METR's follow-up

The practical lesson is to count the whole job: asking, waiting, checking, and fixing. A good answer that takes several rounds to reach may cost more time than its benchmark score suggests.

What to check before trusting the next launch chart

Start with the model version. Then look at what the test actually asks it to do. A chat vote, a coding puzzle, and a fix to an existing app answer different questions.

Look for the setup too: available tools, time limits, and number of attempts. Independent results are useful when they test the same version under clear conditions.

Finally, look for evidence close to your own needs. If you need help fixing a large app, a result on short coding problems gives you only part of the picture.

A launch chart can earn a model a closer look. Before switching, look for proof that it can finish the kind of work you need, with an amount of checking you can accept.