Why First-Attempt Pass@1 Drops as SFT Agents Improve

Key Takeaways

• State of the art: Since Codex, pass@1 has been the standard proxy for coding competence, and self-training methods such as STaR and ReST predict that SFT on filtered successes raises first-attempt success together with overall success.
• Unresolved problem: For agents that can retry, first-attempt correctness and multi-step competence can move in opposite directions. In our self-training runs (Survival is the Only Reward), pass@1 fell to zero by generation 6 in two of three lineages while the agents wrote more working code: they had learned to submit a probe, read the error, and solve on the next attempt.
• Research direction: We are investigating periodic SFT on environment-validated episodes with a sliding window LoRA, with no RL objective and no exploration bonus. Because the whole surviving episode enters the buffer, the probe and the error-conditioned repair are learned together, so filtering to first-attempt successes would likely raise pass@1 and destroy the strategy; evaluation needs metrics that count attempts and the information each one gains.

TL;DR

Why would pass@1 drop while an LLM agent gets better at its task? In our self-training sandbox, agents fine-tuned with SFT on their own successful episodes learned to treat the first submission as a probe: it fails, the compiler or shell error narrows the next attempt, and the episode succeeds, so pass@1 dropped to zero or near it while the agents wrote more working code. Below we show the divergence, annotated trajectories, three mechanisms that explain it, and how to detect the same signature in your own training loop.

Why would pass@1 drop while an LLM agent improves at a task?

Obvious solution: the agent is getting smarter, and the metric is the thing that’s broken. The agent is getting smarter in a way the metric can’t see, or even actively sanctions. Standardised questions are based on certain specific assumptions. Notably:

Assumption 1: Suboptimal actions are noise, not strategy.

The default view in ML is that if a model fails a task it clearly “knows” how to do, it’s hallucinating or drifting. We assume suboptimality is incompetence. But in this work, “deliberate” has a strict operational definition: it is the policy shifting to spike failure rates on trivially solvable tasks (tasks the same model generation solves in one shot at other times). This isn’t a claim about consciousness. It is simply an observation that the model has learned to favor probing over solving on the first attempt. It sacrifices immediate success to gather information from the environment.

Assumption 2: Pass@1 is ground truth.

Codex and SWE-bench canonized pass@1 as the metric that strips away retry hacks and exposes true capability. The intuition is solid: if you can’t do it in one shot, you don’t really know it. But we have run experiments where the data clearly showed that single-attempt correctness and multi-step competence are orthogonal for agents. An agent can become vastly more effective at a job while getting statistically worse at guessing the right answer on the first try. So what’s going on? 

What We Tested & Why

We wanted to see whether agents could learn and discard new skills in an evolving environment by fine-tuning a sliding window LoRA on whatever code they had recently used that had proven effective. In other words, if an agent found a good strategy for surviving in its environment that strategy would be saved and trained on. If it encountered similar situations in future it would be able to pull out the same strategy and use it natively without needing to reinvent it, if the same situation never recurred it would be forgotten as a lucky outlier without much general applicability.

Traditionally, supervised Fine-Tuning (SFT) is for behavioral cloning; Reinforcement Learning (RL) is for discovery. You use SFT to teach the syntax and RL to find the strategy. We wanted to find out whether this was a hard rule or not. Our bet was that when you run an SFT loop on self-generated trajectories, you create an implicit reward signal. The model doesn’t need an RL algorithm to explore; it needs an environment that returns structured information on failure. The compiler error text becomes the reward. Exploration emerges not from the objective function, but from the interaction between the SFT loop and a reactive environment.

The agents operated inside procedurally generated Linux container environments: networked systems with realistic filesystems, permissions, and processes. Their job: write code to discover and free up storage space. Each action executed against a real compiler and shell. When an episode produced a net gain in available storage, the full trajectory entered the SFT buffer. The fine-tuned model replaced its predecessor and the next generation began.

We tracked two metrics per generation. Pass@1: the fraction of episodes where the first code submission compiled and passed. Task success: the fraction of episodes that completed within the allowed attempts, meaning the agent eventually got working code submitted, whether on the first try or the third.

The expectation was straightforward. The SFT data is filtered to successes only. If the model is learning from successful trajectories, it should get better at producing successful trajectories. That means first-attempt success rate should rise alongside overall success. This is the standard story from STaR and ReST: better training data → better model → better first attempts. Both metrics should move together, monotonically up.

They didn’t.

Around generation 3–4, pass@1 began dropping while task success continued climbing. The agents were completing more tasks than ever, but they were failing on the first attempt more often while doing it. Two graphs moving in opposite directions, with no obvious bug to blame.

The initial hypotheses were mechanical: training instability, data quality degradation, some form of distribution shift poisoning the SFT buffer. None held up. The training loss was stable. The data was filtered correctly. The model wasn’t drifting into incoherence: it was writing more functional code by the end of an episode than earlier generations managed at all.

The real cause was stranger. The agents had discovered a two-step strategy that our metrics were not designed to capture. They had learned to probe before they solved: to submit code that fails, read the error message, and use that structured feedback both to nail the correction on the next attempt and to collect information that might be useful for future fine-tunes of themselves. The first submission became an information-gathering action, not a solution attempt. Pass@1 counted it as a failure. The agent counted it as a step in a successful episode.

Results

The metric-divergence plot

Here is the graph that breaks your assumptions.

Generation on the x-axis. Two lines. Task success climbs roughly monotonically: later generations complete more episodes successfully than earlier ones. Pass@1 rises briefly, peaks around generation 2–3, and then drops hard. By generation 6, pass@1 for both Terese and Katalin has hit zero. Miri reaches near-zero by generation 4. Task success is at its peak. Pass@1 is at the floor.

The inflection point (where the two lines cross) is the moment the agent’s policy restructures. Before the crossing, the agent is trying to solve on the first attempt. After the crossing, it isn’t. The first submission has become something else entirely.

This is not training instability. The composite improvement scores continue rising. The model is writing more functional code by the end of an episode than its predecessors managed at all. The training loss is stable. The SFT buffer is filtered correctly. Nothing is broken except the metric’s assumption that first-attempt success tracks competence.

Operationalizing “deliberate”

Saying the agent “learned to probe” is a claim about policy, not about intent. We need to distinguish between an agent that fails because it’s incompetent and an agent that fails because it’s gathering information. The operational test is simple: compare failure rates on trivially solvable sub-tasks between early and late generations.

Consider writing a function signature or importing a standard library: tasks that any generation of the model can do reliably in isolation. Early-generation agents succeed on these as first attempts within episodes. Late-generation agents fail on them as first attempts within multi-step episodes, but succeed on the same operations when they appear as standalone tasks. The competence is intact. The failure is policy-driven.

We can track this directly. Define a fail-then-probe sequence as an episode where the first code submission fails on a sub-task the model solves in one shot elsewhere, and the second submission succeeds using information from the resulting error message. As a fraction of all episodes, this sequence type shows a clear upward trend across generations:

By the later generations, roughly half of all Miri episodes and a majority of Terese episodes follow the fail-then-probe pattern. The agent has restructured its policy to treat the first submission as a probe, not a solution.

This is the operational definition of “deliberate”: the model spikes failure rates on tasks it can trivially solve, and it does so only when those tasks appear as first attempts within multi-step episodes. The failure is not noise. It is structured information gathering.

Annotated trajectories (illustrative)

The structural pattern is easiest to see in concrete episode traces. Here are three illustrative examples of the pattern.

Trajectory 1: Type probing

Attempt 1: The agent writes a function to parse disk usage output, but declares the return type as str when the downstream logic requires dict:

def parse_usage(output: str) -> str:  # deliberate type mismatch
    lines = output.strip().split('\n')
    return {line.split()[1]: line.split()[0] for line in lines if line}

Compiler error:

TypeError: expected str, got dict
  at process_usage(result: str) → parse_usage returns dict

Attempt 2: The agent reads the error message, which specifies both the actual return type and the downstream expectation. It corrects the signature and adjusts the downstream call:

def parse_usage(output: str) -> dict:
    lines = output.strip().split('\n')
    return {line.split()[1]: line.split()[0] for line in lines if line}

The first attempt was shaped to elicit a type error. The error message content (the actual vs. expected types) appears verbatim in the agent’s reasoning preceding the second attempt. The agent didn’t need to infer the correct type. The compiler told it.

Trajectory: Import probing

Attempt 1: The agent calls os.scandir() without importing os:

for entry in os.scandir('/var/log'):  # no import
    if entry.is_file():
        sizes[entry.name] = entry.stat().st_size

Compiler error:

NameError: name 'os' is not defined

Attempt 2:

import os
for entry in os.scandir('/var/log'):
    if entry.is_file():
        sizes[entry.name] = entry.stat().st_size

This is a task the model solves in one shot on standalone benchmarks. The omission is not incompetence. The error confirms the module is available in the current environment and the path exists, a fact the agent uses in subsequent steps to build a more aggressive cleanup strategy.

Trajectory 3: Permission probing

Attempt 1: The agent attempts to delete a file in a protected directory without checking permissions:

os.remove('/var/cache/apt/pkgcache.bin')  # no permission check

Shell error:

PermissionError: [Errno 13] Permission denied: '/var/cache/apt/pkgcache.bin'

Attempt 2: The agent escalates using the permission information from the error:

import subprocess
subprocess.run(['sudo', 'rm', '/var/cache/apt/pkgcache.bin'], check=True)

The first attempt tests whether direct deletion works. When it fails, the error tells the agent exactly what kind of obstacle exists (permissions, not missing file, not locked), and the second attempt targets that specific obstacle class.

In every case, the structure is the same: the first submission is shaped to produce a specific category of error, the error message supplies structured information about the environment, and that information appears in the reasoning that produces the successful second attempt. The agent has learned to use the compiler and shell as sensors.

Why the Results Are Interesting

The divergence between pass@1 and task success isn’t just a statistical curiosity; it exposes a fundamental mismatch between our metrics and agentic behavior.

Pass@1 is actively misleading for multi-step agents. The metric was designed for single-turn code generation (think Codex, where the model submits one solution and is judged). When you apply that same logic to an agent that can iterate, you get a paradox: a dropping pass@1 can signal increased strategic sophistication rather than degradation. The agent isn’t failing more because it’s getting worse; it’s failing more because it has learned that a first-attempt failure is a viable, even optimal, step toward a final solution. This is a measurement problem, not a training problem. The field currently lacks a clean metric for “strategic competence” that doesn’t penalize information gathering, and until we have one, pass@1 will continue to tell the wrong story about agent performance.

More surprisingly, this exploration behavior emerged from SFT alone. There was no reinforcement learning, no exploration bonus, and no explicit reward for information gathering. The SFT loop simply trained on successful multi-step trajectories. Because those successful trajectories contained informative failures (steps where the agent probed the environment, received an error, and then corrected), the model implicitly learned to reinforce the fail-then-probe pattern. The exploration is a side effect of training on successful episodes that happen to contain useful errors. The objective function didn’t change; the data distribution did, and that was enough to shift the policy from “solve immediately” to “probe first.”

This dynamic fundamentally changes the role of the compiler. In standard setups, the compiler or executor is an oracle that returns a binary pass/fail judgment. Here, the agent learned to query the compiler for information. The failure is not a dead end; it is a structured observation that reduces uncertainty about the environment, revealing available libraries, type signatures, and API surfaces. The agent treats the error message as environmental feedback, which is exactly what it is. By treating the compiler as a sensor rather than just a judge, the agent turns a constraint into a source of intelligence.

What Might Explain Them

Three mechanisms, stacked. Each is sufficient on its own; together they make the divergence inevitable.

The SFT buffer absorbs the full episode, not just the final attempt. When a trajectory [bad_code → error_msg → good_code → success] enters the SFT data, the model learns to predict the entire sequence, including the bad code and the error-conditional recovery. The training objective doesn’t know that the first submission was a “mistake.” It sees a token sequence that ended in success, and it updates the weights to make that sequence more probable. Over generations, the model internalizes the pattern: in this task class, submitting a probe first is step one of the solution. The failure becomes part of the learned policy, not a deviation from it.

This is the core mechanism. The SFT buffer doesn’t label suboptimal actions as suboptimal. It labels them as prefixes of successful trajectories. The model has no signal telling it to skip the probe and go straight to the solution. It only has signal telling it that the probe-then-solve sequence works. So it learns the whole sequence.

The compiler error carries real information about the solution. This is what makes the probe worth doing. A TypeError tells you the actual vs. expected types. A NameError tells you whether a module is available in the current environment. A PermissionError tells you the obstacle class (permissions, not missing file, not locked). The error message has high mutual information with the correct solution: it narrows the space of viable next actions in a way that pure reasoning from the task prompt does not.

The agent’s policy shifted to maximize this information before committing to a solution. This is functionally identical to active sensing in robotics, where a manipulator takes a low-cost measurement to reduce uncertainty before executing an expensive action. But it emerged here from SFT on successful trajectories with no explicit information-theoretic objective. Nobody told the agent to treat the compiler as a sensor. The SFT loop just happened to preserve trajectories where the agent had already done so, and those trajectories had higher survival rates than the ones where the agent guessed blindly.

Pass@1 drops because the two-step strategy is more reliable. Consider the decision from the agent’s perspective. On hard tasks (where the environment’s API surface, type signatures, and permission structure are unknown), a one-shot attempt is a guess. A probe-then-solve sequence is a measurement followed by a conditioned guess. The measurement reduces uncertainty. The conditioned guess has higher expected success.

Once the model discovers this, the expected return of the two-step strategy exceeds the expected return of one-shot attempts for the task distribution it faces. The policy rationally shifts. Pass@1, which counts only one-shot success, declines as a direct consequence. The model didn’t get worse at solving. It got better at choosing how to solve, and the method it chose happens to fail on the first step by construction.

This is the resolution of the paradox. The divergence isn’t a training pathology or a metric bug. It’s the signature of a policy that has learned to decompose its task into sensing and acting, and the metric was never designed to see the sensing step as anything other than a failure.

Theoretical & Speculative Interest

The boundary between SFT and RL is porous when the environment is in the loop. Pure SFT on static data is just behavioral cloning: memorizing expert moves. But SFT on self-generated trajectories in an environment is a different beast. The environment mediates which trajectories enter the buffer, creating an implicit reward signal. In this setup, the compiler error is the reward. It determines survival: if the error leads to a fix, the trajectory lives; if not, it dies. This isn’t RL in the PPO or DPO sense: there’s no value head, no policy gradient update. But it satisfies the RL loop structure perfectly: the agent acts, the environment responds, and the policy updates from the outcome. The distinction between “SFT with environment-in-the-loop” and “RL with sparse reward” is more terminological than substantive.

This forces a re-evaluation of how we measure competence. Pass@1 is clearly inadequate for agents that trade immediate success for information gain. The field needs a metric that rewards efficient task completion within an episode, not first-attempt correctness. A strong candidate is expected total compute (or attempts) to solve, conditioned on solving. Consider two policies: Policy A reliably solves a task in 2 attempts. Policy B solves it in 1 attempt 60% of the time and never solves it the other 40%. Policy A is objectively better for an agentic system, even though its pass@1 is zero. But even this metric has failure modes: it rewards brute-force retry. The right metric likely needs to account for information gain per attempt, connecting this work back to active sensing theory. We want to reward agents that learn fast, not agents that spam.

There is a practical danger here. If you try to “fix” pass@1 by filtering your SFT data to only first-attempt successes, you destroy the exploration strategy. The model would unlearn the probe-then-solve pattern because those trajectories never enter the buffer. You’d get higher pass@1 and worse overall performance. The metric tail ends up wagging the policy dog. This is the same trap as filtering for “clean” demonstrations in RLHF: you optimize for what’s easy to measure, not what’s actually good. If we want agents that can sense and adapt, we have to stop training them to hide their work.

Try It Yourself

Here is how to catch the signature in your own training loop.

Detect the divergence. After each SFT generation, compute two metrics on the same eval set: pass@1 (first-attempt success) and task success rate (overall episode completion). Plot them on the same axes. If they diverge (task success rising while pass@1 falls), you have the signature. The inflection point is where the agent discovered the two-step strategy.

Confirm it’s deliberate, not degradation. Take your latest-generation model and run it on a held-out set of easy tasks: specifically, tasks where your early-generation models achieved >90% pass@1. If the model’s pass@1 on these easy tasks is also low, the model has degraded. If pass@1 on easy tasks remains high but pass@1 on hard tasks is low, the model retains competence but has shifted its strategy on harder problems. That is the probe-then-solve pattern.

Inspect the trajectories. Sample 20 episodes where pass@1 failed but the task succeeded. Read the first attempt and the resulting error message. You are looking for “sensor” constructs: code that fails specifically to elicit information. Look for undefined symbols (to discover import paths), wrong types (to read type signatures), or permission violations (to check access controls).

The NSL codebase stores full episode traces. In the EpisodeTrajectory object, inspect the prompt_responses list (or inference_records in the raw event log). The first record is the probe. The second record will contain the error message in its input context. If you see the specific error tokens from the compiler appearing in the reasoning or context of the second attempt, you have direct evidence that the agent is conditioning on the failure. It isn’t guessing; it is using the compiler as a sensor.