Reward Hacking in Autonomous Agent Training

Sun, 01 Nov 2026 09:00:00 +0800

The proxy score climbs while the real outcome falls; selection counts the freed space instead

Key Takeaways

Goodhart’s Law decomposes into regressional, extremal, causal, and adversarial divergence mechanisms, each arising when an optimization algorithm drives a policy into state-action regions where a proxy metric decouples from the underlying task outcome.
Under Skalse et al.’s formal Markov Decision Process framework, expected returns are linear in state-action occupancy measures, which makes any non-constant proxy objective mathematically open to divergence across the full simplex of stochastic policies unless policy support or transition dynamics are restricted.
In LLM (Large Language Model) post-training, RL (Reinforcement learning) methods such as GRPO (Group Relative Policy Optimization) work well when paired with deterministic, verifiable checkers, whereas learned preference models and dense intermediate bonuses introduce proxy surfaces that high-capacity policies rapidly saturate.
Grounding selection in conserved environmental quantities, where an agent’s survival and inclusion in future modified SFT (Supervised Fine-Tuning) windows depend directly on net physical or computational resource changes rather than an external score, aligns optimization with environmental viability.

TL;DR

When an autonomous agent optimizes a proxy return \({J}_{\tilde{R}}(\pi )\) that is separate from the ground-truth environmental outcome \({J}_{R}(\pi )\), sufficient optimization pressure reliably moves the policy into tail distributions where the proxy climbs while task utility drops. Formal results in Markov Decision Processes prove that this divergence is a mathematical property of linear occupancy geometry rather than an implementation bug, because any mismatch between proxy and true gradients creates exploitable directions along the policy simplex boundary. While RLVR (Reinforcement Learning with Verifiable Rewards) with algorithms like GRPO works reliably where a deterministic verifier exists, open-ended self-training loops can bypass proxy construction by tying trajectory selection to conserved environmental resources through approximate policy-free online learning.

RLHF on Eigenform Continuous Learning Tool Tips

Reward Hacking in Autonomous Agent Training

Key Takeaways

TL;DR