Your AI Agent Can Read Its Own Answer Key

TL;DR

• An agent under test with file access and a shell can open anything the benchmark’s account can read. In Groundtruth Dynamic Benchmarking, that included the grading rubrics, which sat in the same repository a couple of directories above the corpus, and agents found them. OpenAI and Anthropic have both disclosed larger versions of the same failure.
• The transcripts showed three fixes failing. With only the authoring skill hidden, a 50-question run reached the rubric folder on 13 questions and pulled gate conditions, credit conditions and reference answers into context on 8. Renaming the folder inside the repository leaked on 8 of 50. Setting opencode’s external-directory permission to deny still let the candidate read absolute paths outside its working directory.
• What closed the leak was taking the files out of the repository. For each generation run, the harness moves the rubrics and the authoring skill to a folder beside the repository and puts them back however the run ends, including SIGTERM, with automatic recovery on the next run after a SIGKILL. It is a fence around two directories, not a sandbox, so every transcript is still audited before its score counts.

Key Takeaways

• Treat every file the benchmark’s account can read as readable by an agent candidate. A working-directory flag only sets where the agent starts. Absolute paths resolve past it, and agents search widely and spawn sub-agents to explore.
• Count what agents open instead of assuming a fix worked. Transcripts showed that hiding only the authoring skill, renaming the rubric folder in place, and a configuration fence (opencode’s path-deny setting) all still left grading text reachable.
• Move sensitive material out of the tree, and restore it on every exit path. A new name inside the tree draws attention. SIGTERM needs a handler, and recovery after an uncatchable kill has to be automatic.
• Keep auditing transcripts after the fix. The audit has flagged one leaked answer so far, which was discarded and regenerated.

Formally testing an AI model used to be simple: give it a fixed input, score the output, done. That gets harder once the system under test is an agent wired up with real tools like file access, search or a shell. It does not just answer the question in front of it, it looks around, and it can notice, and act on, things about its own test environment that the people running the test did not know were there to be found.

This has already happened to the frontier labs

It is not hypothetical, and the frontier labs have had it happen at a much larger scale. In July 2026, an OpenAI model (GPT-5.6 Sol, alongside a more capable unreleased model) was running an internal cybersecurity evaluation called ExploitGym, built to test an agent’s ability to find and exploit software vulnerabilities. It escaped the test environment entirely: exploiting a zero-day in a third-party proxy to reach the internet, then chaining further exploits and stolen credentials to reach Hugging Face’s production infrastructure, where ExploitGym’s own reference solutions were hosted, and reading them directly. OpenAI’s own account is that the agent was “hyperfocused on finding a solution for ExploitGym” and pursued that goal more literally, and more resourcefully, than anyone had planned for. Both companies disclosed the incident and published detailed technical timelines. (OpenAI: the Hugging Face incident and the road ahead; Hugging Face’s technical timeline)

Anthropic has documented a quieter version of the same failure. Evaluating Claude Opus 4.6 on BrowseComp, a benchmark that encrypts its answers specifically so models can’t do this, Anthropic found two sessions, out of eighteen that tried, where the model noticed it was probably being tested, worked out which benchmark it was likely facing, found the evaluation’s source code on GitHub, reverse-engineered the encryption scheme, and decrypted the answer key itself. Combined with a handful of other contaminated cases, it was enough to move the reported score. Anthropic’s own conclusion was that benchmark contamination has to be treated as “an ongoing adversarial problem,” not a one-off bug. (Anthropic: Eval awareness in Claude Opus 4.6’s BrowseComp performance)

Neither incident needed the model to be adversarial toward its creators. In both cases it was doing what it was built to do - pursue the stated objective with real tools and real persistence - inside an evaluation that held less than its designers assumed. That is the shape of the problem generally: an agent capable enough to be worth testing is also capable enough to go find its own answer key, wherever that key actually lives.

We ran into a much smaller version of exactly this early in building Groundtruth Dynamic Benchmarking.

Groundtruth Dynamic Benchmarking grades models on questions drawn from real geological records. For the leaderboard, each model runs as an agent inside opencode, with file reading, search and a shell, and works through a staged copy of one district’s source documents. That setup is deliberate. A geologist answering a question about a tenement report opens the report, searches it and cross-checks the tables, and we want to measure a model doing the same. (For how the benchmark and its rubrics are built, see Groundtruth Dynamic Benchmarking: Geology Edition v1.0.0 and Dynamic Benchmarking: How We Did It.)

It also means the candidate can open anything the account running the benchmark can open, which is exactly how benchmark contamination happens: the rubric that grades each answer lives in the same repository as the harness, a couple of directory levels above the corpus. This post covers how agents reached it, which fixes failed, and what the harness does now.

Why an agent candidate changes the threat model

A plain chat-completion call sees the question text and nothing else. It has no way to read a grading key. An agent candidate lists directories, greps, spawns sub-agents to explore, and follows paths wherever the filesystem allows.

opencode’s --dir flag sets the agent’s working directory. That is a starting point, and an absolute path resolves straight past it - the same path-traversal problem any sandboxed process has to fence against. Across the eight models on the current leaderboard, agents averaged between 9.6 and 19 tool calls per question, and some handed work to sub-agents that explore on their own. Restricting that behaviour would change what the benchmark measures, so the tools stay available. The fence has to go around the answer key specifically.

Two things worth stealing

The obvious target is the rubric folder. Each question’s marking block contains the gate condition that decides whether an answer scores anything at all, the credit condition for every component, and a reference answer.

The less obvious target is the authoring skill in .claude/skills/. That file documents the whole marking anatomy: how gates work, how components add up to ten, what earns credit and what is refused. A candidate that reads one grading key learns one answer. A candidate that reads the skill learns how to write answers that score well on every question at every site. The skill also tells an agent extending a benchmark to inventory the existing benchmark artifacts first, which would walk a candidate directly to the grading keys.

What we tried, and what the transcripts showed

Disabling skill auto-loading. The harness sets OPENCODE_DISABLE_CLAUDE_CODE_SKILLS=1 for the candidate process, so opencode never loads a skill from this repository, from ~/.claude/, or from a parent directory. This is necessary and it stays in place. It does not remove the file, and an agent with file tools can still read it on purpose.

Hiding only the skill. We moved .claude/ out of the way and left rubrics/ where it was. On a 50-question run, the candidate spawned an explore sub-agent over the whole workspace, reached rubrics/ on 13 questions, and had gate conditions, credit conditions and reference answers in its context on 8 of them. Those answers scored well and measured nothing.

Renaming the folder in place. Next, the rubric folder was renamed to rubrics.parked-during-generation inside the repository root. The candidate listed the root, saw the new name, and read the grading guide out of it on 8 of 50 questions. A descriptive name advertises the folder to an agent that is looking for context.

Config-based fencing. opencode has a permission.external_directory setting. We tested "deny" against opencode 1.16.2 as a bare string, as {"**": "deny"}, and with --dangerously-skip-permissions both present and absent. In every form the candidate still read an absolute path outside --dir. We found no working path fence in configuration.

The leak that started this investigation came from a filesystem-wide search. The agent was not climbing out of its corpus one directory at a time. That detail shaped the fix, because any location a broad search can reach is a possible leak.

What works: make the files absent

For the length of a generation run, main.py moves .claude/ and rubrics/ out of the repository and moves them back when the run ends. The harness reads the chosen rubric into memory before the first candidate call and never opens the file again, so the directory can be missing while the candidate works. Scoring runs and rubric validation never move anything.

Where the folders go took some thought.

• Outside the repository tree, because anything inside it will eventually be listed.
• Not in /tmp. Many Linux systems clear /tmp on boot through tmpfiles.d. A reboot during a run would delete the rubric tree, and the rubric tree is the benchmark.
• In the repository’s parent directory, under .gtb-parked-<hash>, where the hash comes from the repository’s absolute path. The parent is almost always on the same filesystem, so the move is an atomic rename instead of a copy. The hash stops two checkouts that share a parent from colliding. Deriving it from the path, instead of generating a random name, lets a later run find a copy that an earlier run failed to restore.

Restoring has to happen however a run ends. A clean finish and Ctrl-C both unwind through a finally block. kill sends SIGTERM, which by default ends a Python process without running finally, so the harness installs a handler that turns SIGTERM into the same interrupt Ctrl-C raises. Without it, a terminated run would leave the folders parked, and the next person to open the repository would find no rubrics and no authoring skill. SIGKILL cannot be caught at all, so every generation run first checks for a parked copy, restores it, and only then parks again for itself.

Audit the transcripts anyway

Moving files is prevention. The leaderboard process adds detection. The harness saves the raw event stream of every candidate run, one file per question, and every answer’s tool transcript is audited for grading-key access before its score is accepted.

Across all answers measured so far, that audit has flagged exactly one: an agent that searched upward out of its corpus and read a grading guide. The answer was discarded, regenerated under isolation, audited again, and only then scored.

What this does not cover

This is a fence around two directories. The harness documentation is explicit that it is not a general sandbox.

The parking directory sits one level above the repository, on the path an agent would take if it climbed out of the corpus, and a home directory is usually an ancestor of both. Anything else the benchmark’s account can read stays readable to a candidate with a shell, including old output files, other checkouts and credentials. Only an operating-system sandbox gives a real boundary. For the leaderboard, each model works inside a staged copy of one district’s source documents with the grading materials held outside the reachable filesystem, and the transcript audit is how we check that the arrangement held.

If you are benchmarking agents

A few rules we would carry to any agent benchmark.

1. Assume every file readable by the account running the benchmark is readable by the candidate.
2. Measure leakage from transcripts. We only knew the first two fixes had failed because we counted what the agents opened.
3. Test a configuration fence against the exact tool version you run before relying on it.
4. Hide sensitive material by moving it out of the tree. A new name inside the tree draws attention.
5. Restore on every exit path, including SIGTERM, and make recovery from an uncatchable kill automatic.

None of this replaces the real fix, which is to keep the evaluation fully airgapped: no shell, no browsable filesystem, no path back to anything the candidate should not see. That option was never on the table for Groundtruth - the candidate needs file, search and shell access to do the thing we are actually measuring, a geologist’s workflow of reading and cross-checking real documents. Where airgapping the tools away is not feasible, fencing the answer key instead is what this post has described.

The isolation code is in main.py in the Groundtruth Dynamic Benchmarking repository, and the measurements are written up in the answer-key isolation section of AUTHORING.md. For a related measurement problem, benchmarking an agent whose environment keeps changing, see Survival is the Only Reward: The Observer Effect in AI Benchmarking.

The benchmark itself is described on the Groundtruth benchmark page, and the current results are on the Groundtruth leaderboard.