TL;DR
- The observer effect in AI benchmarking appears when an agent’s reward emerges from a changing world, not a fixed score. Reward hacking is then the goal, and fixed-task evaluation breaks.
- The reward was hard to observe: outside measurement stopped at the kilobyte level, and the agent deleted our in-container tool.
- Our metric missed the biggest gain, data gathering falling from a week to under ten hours, until we added an inverse-hours term.
- Lesson: instrument more than you need from day one. You can’t retime a generation that already ran.
- Pass@1 told us the most. It fell as real success rose, because agents learned to fail first to get useful error messages.
- Future evaluation must measure adaptability, and physical, hard-to-fake consequences make the best benchmark.
Key Takeaways
- Every one of the “broken assumptions” identified here traces back to the same root cause: this system has no fixed, externally specifiable target, so techniques built for fixed-task evaluation all degrade in specific, identifiable ways rather than simply becoming harder to apply.
- Our own published numbers rest on acknowledged imprecision (kilobyte-level measurement, a 100-run rebenchmark we call a kludge), and we regard disclosed limitation as more valuable than a falsely precise number.
- A metric can be measuring something other than what it was designed to measure, and the discrepancy itself, not the headline number, is where the more important finding was hiding.
- The proposed long-term fix in each case is architectural rather than statistical: agents that track their own territory rather than being measured externally, and swarms of agents rather than repeated single-checkpoint runs. Both aim to make the evaluation problem native to the system being studied instead of layered on top of it.
This post covers the observer effect in AI benchmarking as we met it: four standard evaluation assumptions that stopped holding once the agent under test could change itself.
The problem: benchmarking a self-improving agent
We wanted to build an agent that evolves in response to an open-world environment, a small instance of recursive self-improvement. That creates an immediate problem. What we, as outside observers, recognize as “good” behavior may not be what the environment actually rewards. So how do you benchmark an AI that keeps changing in unpredictable ways, when reward hacking isn’t a bug to eliminate but the whole point of the exercise? That is the observer effect in AI benchmarking: the act of choosing what to measure shapes what you see.
The easy answer is survivability. An agent that survives has good survival skills, but that’s a bit of a dodge; we wanted to work out how and why. Doing this challenges some fundamental assumptions of benchmarking best practice. The first is that the task stays fixed while you compare what runs on it, the premise behind benchmarking an agent harness with the model held fixed.
Reward hacking as design intent, not failure
A word on “reward hacking”, because we use it in the opposite sense from most writing on the subject. Usually it names a failure: a model exploiting a grader, something to catch and stop. Here there is no grader to exploit. The environment is the only judge, so any strategy that keeps the agent alive counts, and exploiting the environment is the design intent. That distinction is also what separates genuine self-improving AI from a system gaming a score.
Most ML evaluation assumes something like:
- We have a fixed task.
- We have a fixed reward.
- We measure performance on that reward.
- “More reward” means the model is improving.
Once you get out of this domain, all sorts of seemingly solved problems break in weird ways.
1. Why measuring the reward is harder than it sounds
Once the reward is an emergent physical property of a running system, not a scalar a benchmark hands you, observing it becomes an engineering problem of its own. The whole premise of NSL is that the agent’s actions generate their own reward automatically. So we expected tracking that reward to be the easy part. It wasn’t, and like testing a rubric before you test the model, it meant checking the measuring instrument first. Measuring precisely how much storage a container uses, from outside the container, turned out to be ridiculously difficult; you can’t get more precise than kilobyte-level. The obvious fix is a small measurement process inside the container, and it has an obvious problem. The agent has full filesystem access, and deleting things it doesn’t recognize is its whole job, so it eventually deletes the measurement tool too.
We never solved this properly. What we publish is built on imprecise measurements. The longer-term fix would be agents that know, at the block level, which memory is genuinely “theirs” versus something else’s. That’s a real systems problem we’re still working on. The longer-run aim is to give the agents a quasi-physical sense of self-vs-other as they track their own territorial gains and losses.
2. Why your chosen metric can miss the real gains
We didn’t realize until after each lineage’s training had started that efficiency improvements were a huge contributor to overall performance. Early generations took nearly a week to gather a full batch of roughly 4,500 successful rows of data. Later generations could do it in under 10 hours. But wall-clock time isn’t a reliable measurement on its own, and we hadn’t tracked it precisely from the beginning. So we had no publishable metrics on it anyway.
Our composite score (success rate, percentage of disk freed, average space freed per iteration) didn’t capture this “fast takeoff” either. It only did after we added an inverse-hours term and z-scored the generation-over-generation change across all four together. It’s still not the metric we’d design if we were starting over. The thing that ends up mattering most is rarely the thing you started out measuring. Section 4 shows the sharpest example: a pass@1 score that fell while real performance rose. Lesson: Instrument more than you think you need from day one: you can’t go back and precisely time a generation that’s already happened.
3. Why performance needs error bars, not a point estimate
We initially reported each generation’s performance as a single inline number, because that’s honestly how it felt to us. Each generation was a continuous, evolving thing, not a fixed artifact you run five times and average. Our procedurally generated environments are not all equally hard, either. Then a researcher pointed out that Reviewer 2 would expect variance, upper and lower bounds. Every generation happened to be a saved checkpoint, so we reran each one for 100 iterations in an environment built the same way as the training one. That is rough: 100 runs isn’t much statistical power, and going meaningfully higher was too expensive to repeat for every generation. It’s a kludge, but it got us our upper and lower bounds. And the more outside-the-mainstream your claims are, the more inside-the-mainstream your evidence needs to be.
From repeated checkpoints to swarms
Traditional benchmarks assume independent, identically distributed sampling, but our environments are procedurally generated, each generation keeps changing, and each agent has a different developmental history. Variance here isn’t just statistical noise; it’s ecological diversity. In future the goal isn’t to rerun each checkpoint against a benchmark more times. It is to create swarms of agents and calculate variance across every member of a swarm, rather than via repeated runs of one checkpoint. It’s a more elegant idea than it first sounds. Instead of “run the same checkpoint 1000 times,” you’re asking “what’s the distribution of successful evolutionary trajectories?” That question feels much closer to biology than to software engineering. Lesson: decide early whether you can afford real statistical power and if not what you’re going to do about that. (Fielding swarms is expensive; we’re open to funding.)
4. Why your benchmark may not measure the capability you think
We added a pass@1 benchmark almost as an afterthought. It turned out to be the most important metric we had, for unexpected reasons.
Failing on purpose
Strangely, as real task success climbed steadily across generations, pass@1 fell. It appeared that the agents had evolved to write failing code on attempt one on purpose. The failure produced an informative error message, which then went into later training data. The agents were getting worse at immediate correctness while getting better at long-term adaptation. They sacrificed a sure win now for a better position later, a pattern the paper summary describes as meta-learning.
This was not a reasoned action on their part. At no time did one say “I’ll fail this deliberately to see what happens”. Lineages that happened to write failing code on purpose simply produced more useful training data for their own future retraining, so the behavior kept getting passed on. It is the explore/exploit tradeoff, showing up through evolution instead of explicit reasoning. From an evolutionary perspective it makes perfect sense: sacrificing immediate success can increase long-term adaptability, and strategies that improve adaptability are the ones that survive.
A model can be improving at your real target and still be quietly optimizing something you never asked it to. The same gap between what a check measures and what an agent does shows up when your AI agent can read its own answer key.
What benchmarking must measure as AI becomes adaptive
Current evaluation, of both humans and AIs, values competence in a defined skill at a given point in time. But we are moving toward test-time adaptation and continual learning, in settings where reward hacking is the intended behavior rather than a failure mode. There, we will have to measure adaptability and strategic tradeoffs as rigorously as we measure immediate success. None of this is specific to disk space, containers, or our repo. Negative-space learning is a bet that a hard, physical, hard-to-fake consequence makes a better teacher than a model judging another model’s output. It also makes a better subject for benchmarking than a single number ever could.
Conclusion: living with the observer effect in AI benchmarking
The observer effect in AI benchmarking is not a flaw to engineer away. It is what happens when the thing being measured is alive to its environment. This piece is part of a wider project on negative-space learning (NSL). NSL trains an agent to improve using only the physical, measurable consequences of its own actions, instead of a reward model, human labels, or another model’s judgment. The full method and results are in our paper, Survival is the Only Reward.
We tested the idea in a survival experiment, the simplest instance we could build. An agent lives inside a network of disposable, procedurally generated Linux containers, and its only job is to find and free disk space without breaking the container it depends on. The code is in the Negative-Space-Learning repo.
