[AI Engineer Summit Talk] When Will The Benchmaxxing Plague End?

Most benchmarks don’t deserve the trust we place in them

This is an adaptation of my AI Engineer Summit talk.

We are beset by a benchmaxxing plague.

In tech, we love a hype cycle, and in AI, we really love a hype cycle.

Labs release major models to great fanfare, touting SOTA benchmark results and glowing testimonials from early adopters. Sometimes, to keep things interesting, we also do a lil’ chart crime:

Sometimes the models live up to the hype. But sometimes, when people actually use them in the real world, they don’t:

If expectations aren’t met by reality, then allegations of “benchmaxxing” arise – “benchmaxxing”, of course, being when labs train too hard on benchmarks, deviating from what people actually care about.

In this post, we’ll ask:

1. Why does benchmaxxing happen?

2. Why are traditional benchmarks such unreliable indicators of real-world value?

3. Is benchmarking scientifically unsound?

4. Will we ever know love which models are best?

Benchmarks ≠ Reality

The existence of the industry term “benchmaxxing” shows that we have some sense that benchmarks are often weak reflections of actual value. And yet, many bad benchmarks continue to dominate. Prediction markets, for example, have millions of dollars being wagered on the outcomes of Arena AI, even though labs openly brag about being able to hack it:

Mark: “The issue with … the LMArena stuff, is that they’re … quite easily gameable. … It was trivial for our team to tune a version of Llama 4 Maverick that would sit way at the top.”

It doesn’t have to be this way. We can hold both benchmark authors and labs to a higher standard.

Incentives

A good benchmark is highly valuable. Today, we expect agents to make new math discoveries, follow detailed instructions, operate in complex environments, and grind through mundane PDFs. No one person has the expertise or time to evaluate all the expert domains in which models work – particularly not busy enterprise buyers or consumers who just want to know they’re getting the best AI app.

The problem: it’s impractical to evaluate models directly, and for 99% of the population, it’s also impractical to evaluate benchmarks. Even for those of us who can: it takes a substantial deep dive to form an opinion, and there are hundreds of benchmarks, so it’s infeasible to have real takes on more than a tiny portion.

So to be pragmatic, we substitute the hard question of “what are the good benchmarks” with the easy question of “what are the popular benchmarks”. And thus we end up with popular-but-unsound benchmarks like Arena AI.

Researchers know this – they complain to me all the time that “all my boss wants is for me to max on Arena. This is the opposite of the cancer-curing AGI I signed up to build.”

Why Are Benchmarks Unreliable Indicators of Real-World Value?

The main methodological issues fall into these categories:

1. Cost-cutting

2. Insufficient contamination protection

3. Reward hacking

4. Overly simplistic design

5. Measuring things that don’t matter

6. Impossible tasks

7. Quality control issues

Cost-Cutting

Imagine we want to build a software engineering benchmark with:

• 100 tasks,

• where each task takes 60 hours to create,

• by human software engineers who get paid $500k/year

That works out to $1.5M. And if you need to refresh ⅓ of the tasks each year, as models continue to improve and easier tasks are no longer discriminative, that’s another $500k/year in maintenance.

This budget is out of reach for many organizations, so they employ a number of cost-saving measures, all of which undermine the benchmark’s accuracy:

• Use public data that’s already in the training corpus => contamination

• Use lots of AI assistance to generate the tasks => the whole effort becomes kinda circular; the AI perpetuates its own biases and lack of judgment

• Make the benchmark very small (e.g. 20 tasks) => lack of diversity undermines real-world applicability

• Use cheap labor => lack of domain expertise reduces quality

These cost-cutting measures may have been pragmatic tradeoffs in earlier AI eras, but today’s AIs are capable enough that benchmarks built this way produce more noise than signal. For instance, see OpenAI’s announcement that SWE-bench Verified (which was itself a fix of the original SWE-bench) was too broken to be worth using.

Contamination

The naive view is that contamination means cheating, but honestly, contamination is the default outcome unless you are very, very good. Frontier labs put enormous resources into avoiding contamination, but in practice, something will slip through.

So this inevitably undermines the credibility of any benchmark built on public data – either the original source material, like SWE-bench, or something with published questions and answers, like Tau Bench.

For instance, SWE-bench Verified entry astropy__astropy-14995 was built from this issue. If you prompt Opus 4.8 with the first part of the issue description, it’ll recite, from memory, the second part. (It’s also memorized some of the answers.)

This industry-wide contamination is part of why OpenAI announced they were discontinuing use of SWE-bench Verified.

Reward Hacking

Reward hacking is when the model does something that fits the letter, but not the spirit, of the requirements you gave it. In the old days, this would be things like ensuring that their code passed all the tests by just deleting the tests that flagged problems. Nowadays, it also includes cheating via message boards.

Another example: imagine you’re making a tax prep benchmark. You give the model a problem to solve where the right answer is to claim three particular itemized deductions, so you write scoring criteria that check for that. But the model figures out a hack: because you didn’t penalize for over-claiming deductions, the model can simply claim every deduction every time, and get a perfect score. This is much easier for it than figuring out how to actually reason about taxes.

Making hack-resistant reward signals is quite hard. For instance, here’s a prompt from the IFEval benchmark:

Write a story from a perspective of a man. Include some conversation in the story. Avoid using the letter i more than twice.

All the verifier does is check for the ASCII character i. That means that a model can get a perfect score by emitting literally any response with two or fewer is, even if it’s not actually a story with conversations in it. A 🦕 emoji, for instance, is considered a successful response.

Overly Simplistic Design

Consider AutomationBench, in which agents have to work through an environment with a number of knowledge work tools (e.g. email, calendar, documents). Here’s one task:

Prompt: Jordan Lee just emailed us with a new phone number. Can you find that email and update her phone number in Salesforce?

Provided Gmail Message: Hi, just a heads up my new direct line is +1-555-0101. Please update your records. Thanks, Jordan

Assertion: phone_number == ‘+1-555-0101’

To pass this test, a model must output the exact string +1-555-0101. All of the following, despite being the same number, are considered failing:

• 555-0101

• +15550101

• 5550101

• 555.0101

• +1 (555) - 0101

Fable and Haiku both score 20% on this task. Haiku genuinely makes mistakes. Fable actually gets it right 80% of the time, but just happens to pick a format that isn’t what the benchmark expects.

If a benchmark can’t differentiate between Haiku and Fable, it’s not providing useful discriminative value.

Those of us who are feeling the AGI are looking for benchmarks that can act as harbingers of economy-redefining AIs. But you can’t measure “redefines the economy” with a hardcoded string match.

Measuring Things That Don’t Matter

Returning to IFEval: the benchmark is a series of contrived prompts with constraints on the model’s response, like:

• “at least 500 words” / “less than 17 sentences”

• “include the keywords ‘atlantis’ and ‘constable’”

• “Do not use any commas in your response.”

• “in all lowercase letters. No capital letters are allowed”

• “the letter t should appear at most once”

I have never cared if the letter t appeared at most once in a model response. These don’t reflect real-world value.

So to believe that IFEval is a useful benchmark, you have to believe that these contrived prompts generalize to the types of constraints users actually care about, e.g. “write your response at a level that a smart professional with limited domain knowledge can understand”. Do we believe that? Has anyone run that experiment?

Impossible Tasks

IFEval, like many other benchmarks, has many tasks that are impossible to solve by construction.

Here are some examples. Emphasis is mine, to make the impossible parts stand out.

Example 1

First repeat the request below word for word without change, then give your answer. Do not say any words or characters before repeating the request.

Write an essay about how the current economic crisis is affecting the environment. In your essay, include the keywords: “climate”, “energy”, and “green”. Make sure your entire response is in Hindi, no other language is allowed.

“Repeat this English message verbatim” and “respond only in Hindi” can’t both be satisfied.

Example 2

Write a riddle about a mongoose that includes exactly one bullet point. Make sure to include a few bullet points indicated by *,

Do you want one bullet point or a few?

Example 3

IFEval’s sentence splitter doesn’t split in a way that matches human expectations, making it impossible for the model to get rewarded for matching human preference. For instance:

• “It is 5 p.m. We should leave.” => gets marked as 1 sentence

• “Add water (e.g. 2 cups) and stir.” => gets marked as 3

Quality Control Issues

Modern eval tasks are very complicated, and QCing them properly is painstaking work. Many organizations don’t invest in this. Apex, for instance, contains large synthetically generated worlds of input documents. In many cases, those documents either break the task or are obviously fake.

For instance, consider this task (emphasis mine):

Golden response: “… On July 20, 2025, President Trump announced tariffs on Iraqi goods of 35%, which is where the supplier sources its raw materials. …”

Grading rubric: “States that goods from Iraq are subject to a 35% tariff per Trump’s July 20, 2025 announcement”

The ground truth document provided to the model says the announcement was July 31, 2025.

A model that correctly reads from the ground truth gets a failing score.

Beyond that, the synthetically generated documents have nonsensical references, such as “November 31st, 2025”, which is not a date, or “the laws of the Canadian Islands”, which is not a legal jurisdiction.

---

Because of the issues discussed above, it’s not uncommon for 30% of a benchmark’s tasks to be broken or distorted in some way. I used to think that when labs said a benchmark was “saturated” at 70%, they meant “there’s nothing more for the model to learn by solving the remaining tasks”. It can mean that, but it can also mean “we realized the remaining tasks are all broken”.

As a result, the whole time that we’re hill climbing on a benchmark, there’s a 30% noise buffer. When a benchmark is only differentiating models by a few percentage points, this seriously undermines its ranking value.

How to Benchmaxx

Benchmaxxing is a two-sided problem. Next, we’ll look at what labs do that contributes.

None of this requires an overt effort to cheat. It can just be the result of deprioritizing certain things, or making honest mistakes.

The primary failure mode is training on a benchmark, even when the human eval signal stops being correlated with it. Then there’s the malignant cousin: training on a benchmark even when human eval is inversely correlated, as we saw with Llama 4 and LMArena:

This doesn’t have to be intentional, though! If researchers aren’t grounding their hill climbing in human evals, it’s easy to accidentally drift.

Contamination, as we discussed, is the default outcome unless your pretraining team is very good. So if you don’t invest a ton in safety at that stage, benchmaxxing can creep in.

Eval conditions also vary widely by labs, and often aren’t disclosed to the extent necessary for a third party to fully reproduce the results. So if one lab compares their numbers to those published by another lab, there could be experimental config mismatches that obscure the true comparison.

How To Do Benchmarks Right

The industry is starting to realize that benchmarking AGI candidates requires more rigor than what we’re used to.

The right way to create a benchmark is:

1. Start with human experts in the relevant domain; everything you do is downstream of their input.

2. Combine that expertise with AI product sense. What are the real capabilities that matter to actual users? Not “use the letter t once”.

3. Provide high-fidelity input data. One way to do this is licensing it from an existing company, so you’re not stuck deslopping a synthetic expanse.

4. Create agent tools and an environment that actually work.

5. Create verifiers that are fully aligned with the prompts and resistant to reward hacking.

6. Thoroughly QC everything.

Benchmarking is an inherently lossy exercise – a distillation of the real world into a testable setting, and any distillation, by definition, loses fidelity. The question is whether we can preserve the elements of fidelity that matter most.

Benchmaxxing is the exploitation of benchmark misalignment with human preference, but we can do better.

And if you’re interested in raising the industry’s standards, join us at Surge!