
TL;DR: Ashby’s law of requisite variety states that only variety can destroy variety: a regulator holding an outcome in a narrow band needs at least as many distinct responses as the disturbances it faces. RLHF (Reinforcement Learning from Human Feedback) holds a high-capacity LLM (large language model) to a low-dimensional target (helpful, harmless, honest) by building a high-variety supervisory apparatus around it, and most of its known failure modes are that apparatus running out of variety. The RLHF alternatives differ mainly in where they relocate the variety, into preference data with DPO (Direct Preference Optimisation), a judge model with RLAIF (Reinforcement Learning from AI Feedback), a verifier with RLVR (Reinforcement Learning with Verifiable Rewards) or the environment itself (filtered self-training), and the last choice is what lets the update rule shrink to SFT (supervised fine-tuning) on whatever survived.
Ashby’s Law: Only Variety Can Destroy Variety
Choosing how a model’s behaviour is shaped is a central question in recursive self-improvement, and Ashby’s law gives a way to compare the options.
W. Ross Ashby stated the law of requisite variety in chapter 11 of An Introduction to Cybernetics (1956) as a game, and every argument about RLHF alternatives is at bottom an argument about the quantity that game measures: where a training signal gets its variety. A disturbance D picks a move, a regulator R replies, and a table of outcomes decides what happens to the essential variable E that R is trying to hold steady. If R always replies the same way, the outcomes vary as much as D does.
With two replies R can halve the variety of outcomes, with three cut it to a third, and never further. Measured logarithmically, the variety left in the outcomes is at least the variety of the disturbances minus the variety of the regulator. The full text is hosted by Principia Cybernetica with the estate’s permission.
Two consequences carry over to training. First, regulation is variety reduction: the controller’s job is to stop the environment’s variety from reaching the variable it protects, and it can do so only by matching the variety it faces, move for move.
Second, the regulator’s variety is a budget rather than a style choice. Fix how narrow the target band must be and how varied the disturbances are, and the law tells you how much variety the regulator must contain, wherever you decide to put it. The Principia Cybernetica entry adds a buffering constant for the variety that passive constraints absorb for free, and warns against the overstrong reading that the regulator must match every disturbance one for one.
Conant and Ashby: Every Good Regulator Models Its System
Conant and Ashby added a companion theorem in 1970: every good regulator of a system must be a model of that system. A regulator that cannot distinguish two situations will treat them alike, so any difference in the outcome they deserve goes uncorrected. Read as a statement about post-training, a reward signal that cannot tell two situations apart cannot shape behaviour differently across them, and that blind spot is exactly what a policy under optimisation pressure will find. Aliman and Kester (2019) made the same point for value alignment: a utility function that must not violate human ethical intuitions has to model those intuitions and carry their variety.
What the RLHF Apparatus Is Paying For
The canonical recipe runs from Christiano et al.’s 2017 work on deep reinforcement learning from human preferences to InstructGPT in 2022: collect pairwise comparisons, fit a reward model, then optimise the policy with PPO (Proximal Policy Optimisation) against that reward model plus a KL (Kullback-Leibler) penalty towards the supervised reference. Lambert’s RLHF book (2025) walks through every stage and the engineering that surrounds it: preference data curation, reward model training, rollout infrastructure, KL control, and repeated rounds as the policy drifts away from the data the reward model saw.
Overoptimisation: When the Reward Model Runs Out of Variety
Through Ashby’s lens, each piece is variety that the apparatus must supply because the target supplies none. “Be helpful” is low-dimensional. The prompt distribution the policy meets is not, so the regulator has to produce distinct verdicts across a practically unbounded set of situations, and the reward model carries that variety on behalf of the labellers. It carries less than it seems to.
Gao et al. (2022) measured what happens when a policy optimises a proxy reward model: the gold reward rises, peaks and then falls as the KL distance from the reference grows, with the peak arriving later as the reward model gets larger and sees more comparisons. In Ashby’s terms the proxy runs out of variety, and the policy keeps exploring into the region where the proxy can no longer tell good from bad. The KL penalty that caps the policy’s distance from the reference is the law’s bound rewritten as a trust region.
Where RLHF Alternatives Put the Variety
Sorted by Ashby’s question, where does the variety come from, the RLHF alternatives fall into three groups. The first keeps human preferences and drops the learned reward model.
DPO: RLHF Without the Reward Model
DPO (Rafailov et al., 2023) rewrites the reward in terms of the policy and the reference, so the preference pairs can be fitted with a classification loss and no separate model or on-policy rollouts. The variety still comes from the preference data. What DPO removes is one component of the apparatus and the instability that came with it, while the KL-to-reference term and the coverage limit of an off-policy dataset stay.
RLAIF: A Judge Model Supplies the Variety
The second group of reward model alternatives replaces the human labellers with a model. Constitutional AI (Bai et al., 2022) has the model critique and revise its own outputs against written principles, then trains a preference model on AI-generated comparisons, and Lee et al. reported in 2023 that this approach matched RLHF on summarisation and dialogue. The variety now comes from the judge. A judge can produce verdicts at any scale, but a policy trained against it is regulated by the judge’s blind spots, and the judge is another LLM with the same kind of capacity gap the first one had.
RLVR: Verifiers Supply the Variety
The third group lets the environment judge. Tulu 3 (Lambert et al., 2024) named the recipe reinforcement learning with verifiable rewards (RLVR) and applied it where a function can check the answer: math problems, exact-match tasks and instruction constraints. The reward model disappears because the environment already contains the variety. A checker returns a different verdict for every distinct wrong answer without anyone labelling it.
DeepSeek-R1 (2025) is the clearest large-scale case. Its reasoning stage trained with GRPO (Group-Relative Policy Optimisation) on rule-based rewards for accuracy and format, and the report says neural reward models were avoided because they invite reward hacking at scale. GRPO itself, introduced in DeepSeekMath, drops the critic and normalises each sample’s reward against the group sampled for the same prompt, so the only learned component is the policy. The regulator is the checker, and the checker does not run out of variety as the policy improves.
Filtered Self-Training: SFT on Survivors
Push the third group one step further and the update rule stops being reinforcement learning (RL) at all. STaR (Self-Taught Reasoner; Zelikman et al., 2022) samples rationales, keeps those whose answer matched the reference, fine-tunes on them, and restarts from the original model each round. ReST-EM (Reinforced Self-Training, modified to use expectation-maximisation; Singh et al., 2023) framed the same loop as expectation-maximisation with a binary verifier and scaled it to math and code with PaLM 2. The environment’s verdicts are the whole regulator, and the learning rule is supervised fine-tuning on survivors.
The published recipes of 2025 layer the groups rather than pick one. Qwen3 runs a long chain-of-thought cold start, a reasoning RL stage with GRPO over query-verifier pairs, a thinking-mode fusion stage and a general RL stage, and Kimi K2 runs a joint RL stage over real and synthetic environments with a self-critic rubric for the parts no checker reaches. A Hugging Face survey from August 2026 covers the frontier reports and finds GRPO variants the usual starting point, with the default loss in the GRPOTrainer of TRL (Transformer Reinforcement Learning) now DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) rather than the original. Verifiable signal where it exists, learned signal where it does not: the variety is bought from the cheapest source for each part of the target.
Simple Rules in Rich Environments
The three groups point to one general principle. Holding a high-capacity model to a low-dimensional behavioural constraint takes a high-complexity supervisory apparatus, because the apparatus has to manufacture the variety the constraint lacks. Putting the same model inside a high-dimensional environment with hard, non-negotiable constraints lets the learning rule be low-complexity, because the environment manufactures the variety for free.
A file system, a compiler, a test suite, a simulator or a market produces a distinct consequence for each distinct action, and that consequence is a verdict nobody had to label. Once contact with the environment is compulsory, selection needs no separate judge: propose, execute, observe, keep what survived.
What a Simple Update Rule Buys and Costs
What the simple rule buys is the absence of a proxy. There is no reward model to overoptimise, no critic to go stale, no clipping ratio to tune, and the reference-KL term shrinks to a stability device for the update rather than a trust region around a model that might be wrong. The verdicts also stay informative as the policy improves, because the environment’s variety does not depend on the policy’s competence, which is the one property a learned reward model cannot promise. That is the update rule we use for sparse-feedback agents: modified SFT, with loss changes only, on bounded windows of recent successful experience, trained into fresh LoRA (Low-Rank Adaptation) adapters, with GRPO the default wherever a reliable checker answers often enough.
What it costs is also plain. Filtering on survivors is positive-only, so failures teach nothing except by their absence, and when successes are rare the loop is sample-hungry. It also inherits everything the environment accepts, and that is a feature rather than a defect.
When reward, optimisation and persistence are the same quantity, a shortcut the environment accepts is information about the environment, not a flaw in the learner, and no method that gets the reward is a bad method. That stance, collapsing reward and persistence, is what makes the environment the only ground truth.
None of this says RL needs dense rewards or that RLVR is a stopgap. Where a checker is reliable and successes arrive often enough, group-relative RL extracts more from each batch than a keep-or-drop filter, because it also learns from the failures in the group, and RLVR is the same relocation of variety into the environment with a stronger optimiser on top. The choice between the two is set by how sparse success is, how long the episodes run, and whether the agent’s actions change the environment between updates, the regime where catastrophic remembering and a sliding window keeps the loop learning.
Choosing by Environment Variety, Not by Fashion
Three questions settle most cases: how much of the target the environment can check, how varied the situations the policy will meet are relative to what the available signal can distinguish, and whether the agent changes the environment it is trained in. The answers map onto the groups above.
- A diffuse preference target in open-ended conversation leaves no choice: the variety has to come from people or from a judge, so preference methods stay, DPO on well-covered preference data is the cheapest of them, and the overoptimisation curve sets how far the policy may move.
- A verifiable target with frequent successes is RLVR territory: a checker as the reward, a group-relative policy gradient, and the GRPOTrainer in TRL as the reference implementation.
- A verifiable target with sparse successes, long episodes or an environment the agent modifies favours the simple rule: filter the survivors, fine-tune on a bounded window of recent ones, and let the environment stay the regulator.
- A mixed target gets the layered recipe: SFT for format, preference optimisation for style, verifiable RL for whatever a function can check, in that order.
Ashby’s law does not pick a winner among them. It says that the variety a target demands must be paid for somewhere, and that the cheapest place to pay is wherever reality already produces the verdicts. That is the whole case for RLHF alternatives that let the environment judge, and it is also the limit of the case: where nothing in the environment can tell a good answer from a bad one, the apparatus is not overhead, it is the only regulator there is.
FAQs
Is DPO an RLHF alternative or RLHF without the reward model?
Both readings are fair. DPO keeps the preference data and the KL-anchored objective, and drops the reward model and the on-policy rollouts. In Ashby’s terms the source of variety is unchanged, so DPO inherits the coverage limit of its preference set. It is an alternative to the apparatus, not to the signal.
Does the law mean a learned reward model can never work?
No. It means a reward model works only as far as its variety reaches, which is the overoptimisation curve measured empirically: gold reward rises, then falls, as the policy moves away from the data. Larger reward models, more comparisons and a KL budget push the peak further out. They do not remove it.
When should I prefer RLVR over filtering successes into SFT?
When the checker is reliable, successes arrive often enough for group-relative advantages to carry signal, and the environment stays fixed between updates, RL on verifiable rewards is the stronger default and learns from failures too. Filtering into SFT suits sparse success, long episodes and environments the agent itself changes.
What makes an environment high-variety enough for a simple rule?
It must return a distinct, non-negotiable consequence for each distinct action, independent of the policy’s competence: tests actually passed, a file actually deleted, a balance actually moved. A rubric scored by a model does not qualify, because its variety is the scorer’s. A compiler, a sandbox or a simulator does.
What are the main alternatives to RLHF?
The main alternatives are DPO, which drops the reward model but keeps preference data; RLAIF, which uses a judge model instead of human labellers; RLVR, which uses a verifier as the reward; and filtered self-training, which fine-tunes on episodes the environment accepted. They differ in where the training signal gets its variety.


