Why First-Attempt Pass@1 Drops as SFT Agents Improve

TL;DR: For multi-step coding agents that can retry against a real shell or compiler, first-attempt correctness (pass@1) and overall task competence can move in opposite directions. Across 13 generations of modified SFT with fresh sliding window LoRAs on environment-validated episodes, pass@1 fell to zero by generation 6 in two of three lineages even as overall task completion climbed: the policies learned to submit an initial diagnostic probe, read the error output, and solve on the second attempt. Because the training buffer retains complete surviving episodes rather than single-shot answers, filtering data for higher pass@1 would strip out the probe-and-repair strategy that drives multi-step gains.

Why Pass@1 Drops While Multi-Step Agents Improve

Why does pass@1 drop toward zero while an LLM agent improves at its job inside a recursive self-improvement loop?

The short answer is that the agent is getting smarter in a way single-shot metrics actively penalise. Standardised code evaluations rest on two hidden assumptions that break as soon as an agent can interact with an execution environment across multiple turns.

Assumption 1: Suboptimal First Actions Are Noise Rather Than Strategy

The default view in machine learning is that if a model fails a sub-task it already knows how to solve, it is hallucinating or drifting. Here, “deliberate probing” has a strict operational definition: the policy shifts to spike first-attempt failure rates on trivially solvable sub-tasks (operations the same checkpoint solves in one shot on standalone prompts). Without any claim about intent or consciousness, the policy has learned to favour probing over solving on turn one, trading immediate completion for structured environmental feedback.

Assumption 2: Single-Shot Pass@1 Measures True Agent Capability

The Codex pass@1 benchmark canonised single-attempt accuracy as the metric that strips away retry luck on standalone function synthesis. On single-turn prompts, the intuition holds: if a model cannot solve a self-contained problem in one shot, its reliability is weak.

Later repository benchmarks such as SWE-bench issue resolution brought single-run headline scoring into interactive environments, yet our multi-generation data shows that first-attempt correctness and multi-step competence are orthogonal when execution feedback is free. An agent can become far more effective at finishing complex jobs while getting statistically worse at guessing the right code on the first submission.

Experiment Setup: Modified SFT on Environment-Validated Episodes

We tested whether coding agents could acquire and discard strategies in a changing environment by training a fresh LoRA adapter on a sliding window of recent, environment-validated episodes. When an agent found an effective strategy for freeing disk space, the full episode entered the buffer for the next generation; if environmental conditions moved on and the tactic stopped working, it aged out of the window automatically.

Classic supervised fine-tuning on static datasets performs behavioural cloning, whereas reinforcement learning is typically used for exploration; we examine that trade-off directly in our sparse-feedback SFT vs. RL comparison. Here we ran a modified SFT recipe: periodic fine-tuning with fresh LoRA adapters on self-generated episodes filtered by real container outcomes. Rather than relying on a policy-gradient exploration bonus, exploration emerged from coupling iterative trajectory selection with a reactive shell and compiler whose error messages narrow the next step.

The agents operated inside procedurally generated Linux container environments: networked systems with realistic filesystems, permissions, and running processes. Their task was to write Python or shell code to discover and free storage space. Every submission ran against a live interpreter and shell. Whenever an episode produced a net increase in available storage, the full multi-turn trajectory entered the training buffer for the next fresh LoRA checkpoint.

We tracked two metrics per generation:

• Pass@1: the fraction of episodes where the very first code submission compiled and succeeded without errors.
• Task success: the fraction of episodes completed within the allowed attempt budget, whether working code landed on try one or try three.

Under standard self-training intuition from STaR rationale bootstrapping, training exclusively on successful trajectories should raise first-attempt accuracy alongside overall task completion.

Iterative algorithms like ReST self-training make the same monotonic prediction: cleaner positive examples should yield stronger policies and higher first-try accuracy. Both curves should climb together.

Instead, around generation 3 to 4, pass@1 began falling while task success kept rising. The agents completed more container cleanups than earlier generations while failing on their first submission far more often.

Initial mechanical suspects (loss spikes, corrupted exports, or catastrophic weight drift) all failed inspection. Training loss stayed smooth, rehearsal anchors held general coding competence steady, and late-generation checkpoints executed substantially more working cleanup logic by episode end than v1 ever achieved.

The actual cause was behavioural: the agents discovered a two-step probe-then-solve routine. Submitting a diagnostic first script that triggers an informative compiler or shell error gave the second attempt exact type, module, or permission constraints, turning turn one into a sensor read rather than a blind guess.

Pass@1 vs. Episode Completion Across 13 Generations

Metric Divergence Between Pass@1 and Task Completion

Plotting first-attempt accuracy beside overall episode completion exposes the policy shift directly.

With generation number on the x-axis, overall task execution climbs steadily across iterations while pass@1 rises briefly through generations 2 and 3 and then drops sharply. As reported in our Negative Space Learning paper, pass@1 hits zero by generation 6 for both Terese (full history) and Katalin (top-3 curation), and drops near zero by generation 4 for Miri (last-3 sliding window). Multi-step completion peaks right as pass@1 reaches the floor.

The crossing point marks where the policy restructures. Before the crossing, the model attempts to solve container cleanup blind on turn one. After the crossing, turn one becomes an environment probe.

Composite improvement scores keep rising across the same window (reaching 3.322 online for Miri by v13), confirming that capability did not degrade; only single-shot solvability assumptions broke.

Testing Policy-Driven Probing vs. Model Degradation

Claiming an agent “learned to probe” is an empirical statement about conditional failure rates, not mental states. To separate strategic probing from incompetence, compare failure rates on trivial operations across standalone prompts versus multi-step episodes.

Writing a basic Python function signature or importing os succeeds reliably when late-generation checkpoints are evaluated on isolated single-turn prompts. Yet those exact checkpoints omit imports or mismatch types on first attempts inside multi-step container episodes, then repair them immediately on turn two using the error output. Underlying coding skill remains intact; the failure is context-dependent.

Define a fail-then-probe sequence as an episode where submission one fails on a sub-task the checkpoint solves one-shot in isolation, and submission two succeeds by conditioning on the returned error string:

By generation 7 and beyond, roughly half of Miri’s episodes and over three-fifths of Terese’s episodes follow this fail-then-probe structure. First-attempt failure has become structured information gathering.

Three Annotated Probe-Then-Solve Trajectories

Concrete episode traces show how first-turn errors supply exact constraints for turn two.

Trajectory 1: Type Probing

Attempt 1: The agent writes a helper to parse disk usage output while declaring a str return annotation where downstream caller code expects a dict:

def parse_usage(output: str) -> str:  # deliberate type mismatch
    lines = output.strip().split('\n')
    return {line.split()[1]: line.split()[0] for line in lines if line}

Compiler error:

TypeError: expected str, got dict
  at process_usage(result: str) -> parse_usage returns dict

Attempt 2: Conditioning on the traceback (which names both the actual return object and caller expectation), the agent aligns the signature and caller logic:

def parse_usage(output: str) -> dict:
    lines = output.strip().split('\n')
    return {line.split()[1]: line.split()[0] for line in lines if line}

Trajectory 2: Import Probing

Attempt 1: The agent invokes os.scandir() on /var/log without importing os:

for entry in os.scandir('/var/log'):  # no import
    if entry.is_file():
        sizes[entry.name] = entry.stat().st_size

Compiler error:

NameError: name 'os' is not defined

Attempt 2: Having confirmed interpreter execution context before touching filesystem calls, the agent adds import os and proceeds with directory traversal:

import os
for entry in os.scandir('/var/log'):
    if entry.is_file():
        sizes[entry.name] = entry.stat().st_size

Trajectory 3: Permission Probing

Attempt 1: The agent attempts direct deletion inside /var/cache/apt without pre-checking user privileges:

os.remove('/var/cache/apt/pkgcache.bin')  # no permission check

Shell error:

PermissionError: [Errno 13] Permission denied: '/var/cache/apt/pkgcache.bin'

Attempt 2: The error distinguishes a privilege barrier from a missing path (FileNotFoundError) or a read-only mount, so turn two escalates specifically with sudo:

import subprocess
subprocess.run(['sudo', 'rm', '/var/cache/apt/pkgcache.bin'], check=True)

Across all three cases, turn one elicits a diagnostic exception, the runtime returns ground-truth environment metadata, and turn two conditions directly on those error tokens.

Why Single-Attempt Metrics Misread Multi-Step Agents

Pass@1 Penalises Information Gathering in Iterative Harnesses

Pass@1 was built for single-turn synthesis where the model gets one submission and no environment interaction. Applied to an iterative agent harness, falling pass@1 can indicate stronger multi-step policy adaptation rather than regression. Until evaluation suites report multi-turn efficiency alongside single-shot accuracy, pass@1 alone misdiagnoses probing policies as degraded models.

Exploration Emerges From Modified SFT + Sliding Window LoRAs

Unlike classic static SFT (which clones fixed demonstrations offline), our loop pairs modified SFT with fresh LoRA adapters trained over sliding windows of self-generated, environment-filtered episodes. No PPO value head or explicit entropy bonus was required for probing to emerge. Because surviving multi-turn episodes naturally contained [probe -> error -> repair -> space freed] sequences, periodic LoRA updates reinforced both the diagnostic probe and the error-conditioned fix as a single behavioural unit.

Compilers and Shells Act as Environmental Sensors

In one-shot benchmarks, an executor is purely a pass/fail judge. Inside an unknown container, the executor doubles as a low-cost sensor: exceptions reveal installed packages, directory ownership, and schema shapes with zero ambiguity. Querying the runtime via a cheap first script turns execution errors into structured observations.

Three Mechanisms Behind Deliberate First-Attempt Probing

Three interacting mechanics make pass@1 divergence predictable whenever multi-turn episodes are filtered by final outcome.

Full-Episode SFT Retains Both the Probe and the Repair

When a surviving trajectory [probe_code -> error_output -> working_code] enters the SFT window, standard next-token loss trains the adapter over all assistant turns in the episode. The objective does not penalise turn one for raising an exception; it increases the likelihood of every prefix that led to net disk space freed. Across generations, starting an episode with a diagnostic probe becomes the default policy mode.

Execution Errors Carry High Mutual Information

Exceptions carry high mutual information with valid repairs: TypeError exposes runtime types, FileNotFoundError maps directory layout, and PermissionError reveals privilege boundaries. Taking a cheap measurement to shrink uncertainty before committing to a complex action mirrors active sensing in robotics, emerging here purely through survival selection on multi-turn transcripts.

Two-Step Sensing Beats Blind One-Shot Guessing

In unfamiliar procedurally generated containers where paths, permissions, and cached workloads vary every episode, a blind one-shot script is a high-variance guess. A probe followed by an error-conditioned script has higher expected episode return. Once the policy shifts toward two-step sensing, pass@1 (which scores only step one) drops mechanically as overall episode success rises.

Implications for SFT vs. RL and Agent Evaluation Metrics

Environment-Mediated SFT vs. Sparse-Reward RL

Pure SFT on static human transcripts is behavioural cloning, whereas iterative SFT on environment-filtered agent trajectories closes an outer reinforcement loop through data survival. Although there is no token-level policy gradient or critic network, environment selection supplies sparse outcome filtering while compiler tracebacks supply dense in-context state transitions.

Measuring Expected Attempts and Information Gain

Evaluating iterative agents requires reporting full pass@k metric trade-offs together with expected attempts-to-solve conditioned on episode completion. Consider Policy A, which solves 95% of container tasks in exactly two turns (pass@1 = 0%), versus Policy B, which solves 60% in one turn and fails the remaining 40% (pass@1 = 60%). For production automation, Policy A is substantially more reliable, especially when paired with per-attempt cost budgets so policies cannot degenerately spam retries.

Why Filtering for First-Attempt Wins Hurts Agent Policies

Attempting to “fix” falling pass@1 by keeping only single-attempt successes in the SFT buffer deletes every probe-and-recover trajectory. That filter raises pass@1 on paper while stripping away error recovery and environment discovery when containers drift, repeating the same curation trap seen in narrow reward-filtered post-training.

How to Diagnose Probe-Then-Solve Behaviour in Your Loop

You can separate healthy probing from genuine capability regression in three steps.

Step 1: Plot Pass@1 Against Multi-Step Task Completion

After every fine-tuning generation, log both pass@1 (first-turn execution success) and episode completion rate on the same evaluation suite. When episode completion climbs or holds steady while pass@1 drops sharply, your policy has shifted from one-shot guessing to multi-turn probing.

Step 2: Check Held-Out Easy Tasks to Rule Out Forgetting

Evaluate your latest LoRA checkpoint on a standalone single-turn benchmark where generation 1 achieved over 90% pass@1. If single-turn accuracy also collapsed, your training recipe suffered genuine degradation; if single-turn accuracy stays high while multi-turn pass@1 falls, the policy is selectively probing on uncertain environments.

Step 3: Inspect Episode Traces for Error Conditioning

Sample 20 episodes where turn one failed and turn two succeeded, then inspect prompt_responses (or inference_records) inside each EpisodeTrajectory log. When specific exception tokens from turn one (paths, permission codes, or module names) directly parameterise the working script on turn two, the agent is using execution feedback as a sensor.

FAQs

Why does pass@1 drop while an SFT agent’s task success rises?

When an agent can retry with execution feedback, submitting a quick diagnostic probe on turn one and repairing from the exact compiler or shell error on turn two yields higher episode success than guessing blind. Because pass@1 scores only the first submission, pass@1 falls toward zero as probe-then-solve wins.

How can you tell deliberate probing apart from catastrophic forgetting?

Test the same checkpoint on isolated single-turn coding prompts alongside multi-step environment episodes. In our study, late-generation adapters still solved standalone imports and function signatures in one shot, proving base coding skill remained intact while first-attempt failures spiked selectively inside unfamiliar multi-step containers.

How does probing emerge without an explicit RL exploration bonus?

Our training loop uses modified SFT with fresh LoRA adapters over a sliding window of surviving multi-turn episodes. Whenever a trajectory containing a failed probe, an informative error traceback, and a successful second-turn fix frees storage space, next-token loss reinforces the entire probe-and-repair sequence together.

Should you filter your SFT buffer to keep only first-attempt successes?

No, unless your production harness strictly forbids retries. Filtering out multi-step recoveries raises pass@1 at the cost of deleting error-handling and environment-mapping skills, leaving the agent brittle whenever filesystem paths, permissions, or schemas differ from training examples.

What metrics should accompany pass@1 when evaluating multi-turn agents?

Report overall episode completion within your turn budget, pass@k across sample counts, and average attempts or tokens per solved task. Tracking completion alongside attempt cost distinguishes disciplined two-step sensing from both blind one-shot guessing and brute-force retry spamming.