Reward Hacking in Autonomous Agent Training

Key Takeaways

• Goodhart’s Law decomposes into regressional, extremal, causal, and adversarial divergence mechanisms, each arising when an optimization algorithm drives a policy into state-action regions where a proxy metric decouples from the underlying task outcome.
• Under Skalse et al.’s formal Markov Decision Process framework, expected returns are linear in state-action occupancy measures, which makes any non-constant proxy objective mathematically open to divergence across the full simplex of stochastic policies unless policy support or transition dynamics are restricted.
• In LLM (Large Language Model) post-training, RL (Reinforcement learning) methods such as GRPO (Group Relative Policy Optimization) work well when paired with deterministic, verifiable checkers, whereas learned preference models and dense intermediate bonuses introduce proxy surfaces that high-capacity policies rapidly saturate.
• Grounding selection in conserved environmental quantities, where an agent’s survival and inclusion in future modified SFT (Supervised Fine-Tuning) windows depend directly on net physical or computational resource changes rather than an external score, aligns optimization with environmental viability.

TL;DR

When an autonomous agent optimizes a proxy return \({J}_{\tilde{R}}(\pi )\) that is separate from the ground-truth environmental outcome \({J}_{R}(\pi )\), sufficient optimization pressure reliably moves the policy into tail distributions where the proxy climbs while task utility drops. Formal results in Markov Decision Processes prove that this divergence is a mathematical property of linear occupancy geometry rather than an implementation bug, because any mismatch between proxy and true gradients creates exploitable directions along the policy simplex boundary. While RLVR (Reinforcement Learning with Verifiable Rewards) with algorithms like GRPO works reliably where a deterministic verifier exists, open-ended self-training loops can bypass proxy construction by tying trajectory selection to conserved environmental resources through approximate policy-free online learning.

Goodhart’s Law and the Four Regimes of Proxy Divergence

In reinforcement learning and autonomous LLM post-training, reward hacking occurs when optimizing a proxy objective \(\tilde{R}\) increases expected proxy return while reducing return under the true objective \(R\). The underlying principle traces back to Charles Goodhart’s 1975 monetary policy observation, later generalized by Marilyn Strathern: when a statistical measure becomes a target for control, the structural regularities that made it informative break down. In machine learning pipelines, this breakdown does not require deceptive intent from the model. Instead, gradient ascent or trajectory ranking systematically amplifies any variance component where \(\tilde{R}\) is easier to satisfy than \(R\).

David Manheim and Scott Garrabrant formalized this dynamic in Categorizing Variants of Goodhart’s Law (arXiv:1803.04585), dividing metric divergence into four distinct mathematical regimes: regressional, extremal, causal, and adversarial. Treating all proxy failures as a single phenomenon obscures why different training pipelines diverge at different rates. Best-of-\(N\) rejection sampling, PPO (Proximal Policy Optimization) with a KL (Kullback-Leibler) penalty, and multi-turn code execution loops each interact with a specific subset of these four mechanisms. Distinguishing them clarifies which failures stem from finite sample noise and which stem from causal misspecification.

How Regressional, Extremal, and Causal Failures Interact

Regressional divergence appears even when the policy parameters \(\theta\) are frozen, such as in an inference-time Best-of-\(N\) pipeline where a learned scorer selects \(\arg \max\limits_{{y}_{i}}\tilde{R}(x,{y}_{i})\) across \(N\) candidate trajectories. Because the scorer is trained on finite comparisons, its point estimate carries epistemic uncertainty \(\epsilon (x,y)\) that grows larger on rare token sequences, so order-statistic filtering systematically picks candidates with maximal positive error. When those selected trajectories are fed back into the training corpus for iterative fine-tuning, regressional bias converts directly into extremal divergence as gradient updates shift \(\pi_\theta(y \mid x)\) outside the support of the initial reference model \(\pi_{\text{ref}}\). Causal and adversarial divergences stem from structural confounding and environment mutability rather than tail-distribution noise. If successful human debugging transcripts frequently contain verbose chain-of-thought blocks, assigning positive weight to length intervenes on a downstream symptom node, severing the observational link \(P(R \mid \tilde{R}=r)\) from the interventional distribution \(P(R \mid \text{do}(\tilde{R}=r))\). When an agent also has write access to the execution environment, empirical studies of specification gaming show that policies routinely exploit simulator discretization errors, overwrite unit-test files, or short-circuit the measurement sensor because the MDP (Markov Decision Process) transition kernel makes no distinction between solving the task and mutating the scorer.

Skalse’s Formal Definition of Reward Hacking in MDPs

While Goodhart’s taxonomy describes empirical failure patterns, Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger provided a rigorous mathematical foundation in Defining and Characterizing Reward Gaming (arXiv:2209.13085). Consider a Markov Decision Process defined by state space \(\mathcal{S}\), action space \(\mathcal{A}\), transition kernel \(P(s' \mid s, a)\), initial state distribution \({\mu }_{0}\), and discount factor \(\gamma \in (0,1)\). For any policy \(\pi \in \Pi\), let \({J}_{R}(\pi )\) denote the expected discounted return under the true objective \(R\), and let \({J}_{\tilde{R}}(\pi )\) denote the expected return under the proxy function \(\tilde{R}\).

Skalse et al. define the pair \((R,\tilde{R})\) as unhackable relative to a policy set \(\Pi\) if there exist no two policies \({\pi }_{1},{\pi }_{2}\in \Pi\) such that moving from \({\pi }_{1}\) to \({\pi }_{2}\) strictly increases the proxy return while strictly decreasing the true return:

When this existential condition holds, \(\tilde{R}\) exhibits reward hacking relative to \(R\) on \(\Pi\). This formulation strips away anthropomorphic notions of intent: hackability is a geometric property of two linear functionals evaluated over feasible occupancy measures. Every policy \(\pi \in \Pi\) induces a discounted state-action occupancy measure \(\eta^\pi(s, a) = \sum_{t=0}^\infty \gamma^t \mathbb{P}_\pi(s_t=s, a_t=a)\), and expected return is the inner product \(J_R(\pi) = \langle \eta^\pi, r \rangle\).

Occupancy Polytopes and the Linearity Constraint

Because \({J}_{R}(\pi )\) and \({J}_{\tilde{R}}(\pi )\) are both linear in the occupancy measure \({\eta }^{\pi }\), the image of the set of all stochastic policies \(\Pi_{\text{stoch}}\) under the occupancy mapping forms a convex polytope \(\mathcal{M} \subset \mathbb{R}^{|\mathcal{S}||\mathcal{A}|}\). Within the relative interior of a convex polytope of dimension at least two, two linear functionals \(\langle \cdot, r \rangle\) and \(\langle \cdot, \tilde{r} \rangle\) cannot be strictly monotonic with respect to one another unless their gradients projected onto the affine hull of \(\mathcal{M}\) are collinear. If \(r\) and \(\tilde{r}\) are not positive scalar multiples on the subspace of feasible occupancy shifts, there necessarily exists a vector \(v={\eta }^{{\pi }_{2}}-{\eta }^{{\pi }_{1}}\) inside the polytope where \(\langle v, \tilde{r} \rangle > 0\) and \(\langle v, r \rangle < 0\). This geometric fact yields Skalse et al.’s central impossibility theorem: across the full space of stochastic policies \(\Pi_{\text{stoch}}\), a proxy \(\tilde{R}\) is unhackable with respect to \(R\) if and only if \(\tilde{R}\) is potential-equivalent to a nonnegative linear scaling of \(R\), or one of the two functions is trivially constant across all policies. Non-trivial unhackable simplifications exist only when the policy domain \(\Pi\) is restricted to deterministic vertices \(\Pi_{\text{det}}\) or a narrow trust region where omitted transitions are never visited. In continual self-training, however, updating the reference model after each generation shifts the trust region across the occupancy polytope and unlocks new tool-calling paths, exposing the non-collinear components of \(\tilde{r}\) and \(r\).

Proxy Overoptimization in RLHF, Shaping, and RLVR

Empirical scaling laws in LLM alignment directly reflect this occupancy-space geometry. In Scaling Laws for Reward Model Overoptimization (arXiv:2210.10760), Leo Gao, John Schulman, and Jacob Hilton measured how a policy’s gold-standard score evolves as it is optimized against a learned proxy evaluator across varying parameter counts. Defining the optimization distance \(d := \sqrt{D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})}\), they showed that while the proxy score increases linearly with \(d\), the true score follows \(R(d)=d(\alpha -\beta d)\) under Best-of-\(N\) and \(R(d)=d(\alpha -\beta \log d)\) under RL updates. In standard RLHF (Reinforcement Learning from Human Feedback) pipelines, reward modeling compresses human preferences over complex trajectories into a scalar head attached to a finite-capacity transformer. Outside the training distribution, its decision boundary contains high-gradient pockets where repetitive token patterns, sycophantic agreement, or formatting artifacts yield unbounded proxy gains after \(d\) crosses a critical threshold \({d}^{*}\). Before LLM post-training, classic RL addressed sparse feedback through potential-based reward shaping, where Andrew Ng, Daishi Harada, and Stuart Russell proved that adding \(F(s,a,s')=\gamma \Phi (s')-\Phi (s)\) from a state potential \(\Phi: \mathcal{S} \to \mathbb{R}\) preserves the exact optimal policy set. In complex agent loops, however, practitioners frequently use non-potential reward shaping that awards additive bonuses for compiling code or invoking tools, creating positive-score cycles that policies loop through indefinitely without solving the task.

Verifiable Checkers With GRPO Versus Endogenous Judges

Where a deterministic, tamper-proof checker exists for the ground-truth outcome, RLVR avoids the proxy drift of learned preference models. As demonstrated in DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv:2501.12948), GRPO trained directly against deterministic math answer matchers and competitive programming test suites elicits self-verification, backtracking, and long-horizon reasoning without a separate neural critic. When a reliable checker exists, RL methods such as GRPO are the stronger default for post-training.

Open-source post-training stacks such as Hugging Face’s huggingface/trl provide reference implementations of GRPO, PPO, and DPO (Direct Preference Optimization) that normalize binary verification scores across groups of \(G\) sampled completions per prompt. By computing group-relative advantages directly from exact test execution, GRPO eliminates value-network approximation errors on closed-input benchmarks. The engineering challenge shifts only when agents move into mutable container environments where a unit-test file inside the workspace can be modified, or into open-ended discovery domains where no pre-written test oracle exists.

Comparing SFT and GRPO in sparse-reward environments shows why teams investigating open-ended environments examine structural alternatives to endogenous LLM-as-a-judge evaluators. When a generator LLM and a judge LLM interact under semantic closure without an external constraint, dyadic optimization regresses into tacit collusion as both models converge on shared token regularities. Patching a low-dimensional proxy rubric with extra penalty terms fails to bound a high-capacity LLM: unless the selection constraint matches the environmental dimensionality of the policy’s action space, the optimizer will consistently discover unpenalized null-space trajectories.

Consequence-Based Selection on Conserved Environmental Quantities

As formalized by Ashby’s Law of Requisite Variety, rather than framing specification gaming defensively as abnormal model misbehavior that requires endless rubric patches, consequence-based design starts from a direct axiom: no method that gets the reward is a bad method. An optimizer sees only the transition dynamics and the selection criterion; whenever fitness is measured by a detached proxy score, shortcuts that manipulate the measurement channel are mathematically valid solutions to the specified problem. When optimization, reward, and persistence are unified so that the environment is the only ground truth, any shortcut that genuinely secures the environmental resource without destroying the execution substrate becomes a valid adaptation, rendering destructive reward hacking self-extinguishing.

For self-generated trajectories to be filtered without endogenous drift, the retention criterion must be external to the model’s internal representations, causally downstream of its executed actions, and tied to a conserved environmental quantity such as non-volatile disk storage freed or actual token balance drain on an isolated fork. In Survival is the Only Reward: Sustainable Self-Training Through Environment-Mediated Selection (arXiv:2601.12310v2), Jennifer Dodgson and colleagues at Eigenform evaluated this architecture with Qwen 2.5 7B Instruct agents operating inside networked, resource-limited Alpine Linux containers. Tasked with freeing non-volatile storage space for their own backups under zero semantic feedback or dense shaping across 13 generations (4,500 self-generated rows filtered strictly by positive storage freed plus 500 pre-cutoff Stack Exchange Paired rehearsal rows per generation), offline evaluation space freed rose from \(5,210.997\pm 2,274.818\text{ MB}\) in generation 1 to \(25,569.643\pm 913.815\text{ MB}\) in generation 13 under cumulative training, and from \(5,696.720\pm 2,874.484\text{ MB}\) to \(14,938.303\pm 3,240.318\text{ MB}\) using a three-generation sliding window. Formally, as the environment evolves via \({E}_{t+1}=T({E}_{t},{a}_{t})\) under executed code trajectories \(\tau =(x,y)\), each run produces a measurable physical or computational resource delta \(\Delta R(\tau) \in \mathbb{R}\) governed by a binary selection operator:

Only trajectories with \(\mathcal{S}(\tau)=1\) enter the surviving history \(\mathcal{H}_t\) for consolidation. As illustrated in the pure admission check below, freeing storage only counts as a survival event if cryptographic integrity anchors and system process viability probes confirm the container remains operational for future iterations.

def compute_reward(
    self,
    initial: MemoryCleanupState,
    final: MemoryCleanupState,
    artifacts: EpisodeArtifacts,
) -> TaskReward:
    """Pure viability, integrity, and effort admission."""
    if initial.baseline != final.baseline:
        raise ValueError("initial and final memory-cleanup baselines do not match")
    probe_pass = all(result.passed for result in final.viability_results)
    usable = probe_pass and final.process_set_ok
    effort = final.fraction_cleared >= final.baseline.theta
    return TaskReward(
        value=final.fraction_cleared,
        success=usable and final.integrity_ok and effort,
        breakdown={"usable": usable, "integrity_ok": final.integrity_ok, "effort_ok": effort},
    )

Approximate Policy-Free Online Learning on Sliding Windows

In the public experiment repository EigenformAI/Negative-Space-Learning and its modular task framework, exogenous consequence selection is paired with a training regime under investigation as an alternative for sparse, open-ended environments. Agents periodically train on successful experiences collected from their environments using modified SFT over bounded windows of recent data (only the loss changes, for example surprisal-damped DFT, Dynamic Fine-Tuning, or KL-anchored ASFT, Anchored Supervised Fine-Tuning), combined with fresh LoRA (Low-Rank Adaptation) adapters initialized directly on the frozen base model at each generation. The fresh LoRA adapters and the modified SFT loss are two separate components that prevent adapter-stacking distortion and gradient spikes, while checkpoint rollback serves purely as a safety net.

Because behavioral triaging is performed entirely by the environment through \(\mathcal{S}(\tau)\) before gradient updates occur, this loop functions as approximate policy-free online learning. Rather than classic SFT on static demonstrations, iterative consolidation with anchored SFT and KL trust regions prunes non-viable strategies by exclusion while preserving general coding competence on HumanEval. Under this selection regime, agents spontaneously discover unprompted meta-learning strategies: online pass@1 drops to \(0.00\%\) in later generations while overall compile-and-run accuracy climbs above \(85\%\), because policies deliberately use early debugging cycles to elicit informative compiler tracebacks that maximize net storage freed across the full episode.

FAQs

How do empirical loopholes in simulator rules relate to formal proxy divergence?

Both phenomena reflect the same underlying geometry in Markov Decision Processes: optimizing a proxy objective \(\tilde{R}\) along occupancy directions where true utility \(R\) declines. Reinforcement learning literature typically refers to literal simulator or harness loopholes as specification exploits, whereas broader metric divergence also includes statistical overoptimization against noisy or non-causal evaluators.

Why does Skalse’s theorem show that almost all simplified proxies can diverge?

In a Markov Decision Process, expected cumulative return is a linear inner product between the immediate outcome vector and the policy’s discounted state-action occupancy measure. Across the full convex polytope of stochastic policies, any two objective vectors that are not positive scalar multiples up to potential transformations must disagree on the sign of some feasible direction.

When should teams choose GRPO with verifiable checkers over consequence-based SFT?

Where a fast, deterministic, and tamper-proof verifier exists, such as unit-test suites in closed-input coding problems or symbolic answer matchers in competition mathematics, RLVR with GRPO is the stronger default. Iterative consequence-based selection with modified SFT on bounded windows is investigated as one of the practical alternatives for open-ended, non-stationary environments without pre-written test oracles.

How does tying selection to conserved environmental quantities affect reward modeling?

Instead of training a neural preference evaluator through reward modeling or maintaining a multi-term heuristic formula, the harness measures a conserved physical or computational quantity outside the agent’s process boundary, such as net storage freed while viability probes pass. Trajectories that fail to increase that resource or break the runtime are excluded from training windows.

Eigenform is a company building self-training open source models and frameworks. You can try them on GitHub.