TL;DR: PowerPlay and OOPS answer the question of where the next task comes from inside the agent: the next task is the cheapest one the current solver cannot do, found by a search that also produces the solver that can. Procedural generation answers it from outside: a generator produces more tasks than anyone can train on, and the curriculum is a sampling policy over that distribution. For continual learning the useful comparison is the gate each design uses to accept a task, what that gate costs, and what it protects.
What PowerPlay and OOPS Search For
The weight level adaptation required for continual learning is a tricky mechanical problem. The question of what to learn, what to prioritise, and what to do when you run out of problems is a deeper epistemological issue. Here we run through some of the options.
Where an agent’s next task comes from is a central design question in recursive self-improvement.
Before procedural generation made task distributions cheap, Schmidhuber’s answer to the supply of new tasks was a search. The Optimal Ordered Problem Solver (OOPS, 2002, journal version 2004) takes a sequence of tasks and searches for programs that solve them. Once a program solves the first n tasks it is frozen, and the search for task n+1 splits its time between extending that frozen prefix, so earlier solutions become callable subroutines, and a fresh search from scratch. The ordering is bias-optimal in Levin’s sense, and the demonstration solved Towers of Hanoi up to 30 disks, where the shortest solution alone needs over a billion moves.
PowerPlay: Searching Pairs of Tasks and Solver Changes
PowerPlay (2011) removes the external task sequence. The system searches the space of pairs, a new task plus a modification of the current solver, ordered by their computational cost given the stored experience, and stops at the first pair where the modified solver solves the new task and still solves every task learned before. A task carries its own test, and “solve an old task in less time or space” counts as a new task, so compression of the repertoire is progress too. Schmidhuber’s abstract makes the two claims that matter here: the cost of validating new tasks need not grow with the size of the repertoire, and the search for novelty keeps breaking the generalisation of the present solver on purpose.
Procedural Generation Turns Task Invention Into Sampling
Procedural generation makes the opposite bet: do not search for the next task, generate millions and choose among them. Procgen (Cobbe, Hesse, Hilton and Schulman, 2019) is the reference benchmark: 16 game-like environments whose levels are generated from a seed, so an agent trained on a few hundred seeds can be tested on seeds it never saw. The paper’s finding was that diverse environment distributions are essential to train and evaluate RL agents at all, and that agents trained on small level sets overfit to them. MiniGrid mazes and Crafter worlds follow the same design: the task identity is a seed, and the generator’s parameters define the family.
Curriculum Design Over a Generator: PAIRED, PLR, ACCEL and POET
Unsupervised environment design (UED), introduced with PAIRED (Dennis et al., 2020), states the sampling problem as a game. A teacher sets the free parameters of an underspecified environment and is rewarded for the regret of the student: the gap between an antagonist agent’s return and the student’s return on the same level. Regret is high exactly where PowerPlay’s gate would fire, on levels that are solvable (the antagonist proves it) but not yet solved by the student. Impossible levels score near zero regret, and so do solved ones.
Prioritized Level Replay as a Procedural Curriculum
Prioritized Level Replay (PLR; Jiang, Grefenstette and Rocktäschel, 2020) keeps random generation and adds a curator. Every level the agent has seen gets a learning-potential score, a regret estimate built from TD errors, and the next level is drawn from a mix of that score and a staleness term that favours levels not visited for a while. The result is an implicit easy-to-hard curriculum with no teacher network, and the staleness term is the replay buffer of continual learning under another name.
The follow-up, Replay-Guided Adversarial Environment Design (Jiang et al., 2021), showed that PLR is UED: if the agent updates only on curated replayed levels and never on freshly generated ones, it inherits the minimax-regret guarantee of PAIRED. The authors call the resulting class dual curriculum design, a generator paired with a curator.
ACCEL (Parker-Holder et al., 2022) adds evolution to that class: take a high-regret level from the buffer, make a small edit such as moving a block, and keep the edit if regret stays high. ACCEL is the closest thing inside UED to PowerPlay’s search, since the next task is a small modification of a known one, ranked by a “can almost do” score.
POET: Open-Ended Environment and Agent Search
POET (Wang, Lehman, Clune and Stanley, 2019) frames the same problem as open-ended search with a population instead of a single solver. Environments are mutated, each new environment is kept only if it passes a minimal criterion (not too easy and not too hard for the current agents), and agents are periodically transferred between environments so that a skill learned on one terrain can unlock another. Its growing archive of environment-agent pairs follows the quality-diversity tradition of MAP-Elites.
Reference implementations of PAIRED, PLR, Robust PLR and ACCEL share one codebase, Meta’s dcd repository, now a public archive, with each method selected by flags in a single arguments file.
What Each Approach Pays for Its Gate
PowerPlay’s gate is exact. A task joins the repertoire only when the modified solver passes it and every earlier task, so the archive is also the regulariser: forgetting is ruled out by construction rather than discouraged by a loss term. The price is that every task needs a decidable test, every candidate must be run against the archive or an incremental version of it, and the task space has to be enumerable in order of cost. Those conditions hold for self-invented programs on a small recurrent network and rarely hold for an agent operating a shell.
The Regret Gate in Procedural Generation Curricula
The regret gate is an estimate. Positive value loss stands in for regret, it is cheap to compute from rollouts the agent runs anyway, and it is noisy: a level may be scored as promising because the value function is wrong about it. Nothing guarantees a generated level is solvable, which is why Robust PLR updates only on curated levels and PAIRED keeps an antagonist as an existence proof. Forgetting is handled statistically, with the same replay buffer logic that continual fine-tuning uses, and it can fail in the same ways.
Task Variation Curricula for Self-Training LLM Agents
Today procedural generation increasingly means a model writing code. OMNI-EPIC (Faldor, Zhang, Cully and Clune, 2024) has a foundation model write the environment and reward code for the next task, keeps the tasks that a second model judges interesting and learnable for the agent’s current skills, and archives the result. It is POET’s loop with a language model as the mutation operator.
A Hybrid Curriculum for Self-Training Agents
For an agent that trains on its own successful episodes, the designs that work in practice borrow from both traditions. A generator supplies variations over a few task families and cycles through them, so that a bounded window of recent episodes stays mixed rather than collapsing onto one family. A threshold schedule raises the fraction of the job an episode must complete to count as a success, generation by generation up to a cap, which is PowerPlay’s “simplest still unsolvable” turned into a dial the environment controls. And an archive of previously discovered results rejects rediscoveries, so a rule or fix the agent has already produced earns nothing: OOPS’s reuse of old solutions, turned into a novelty gate on new data.
Where a reliable checker returns frequent feedback, RL with verifiable rewards and GRPO-style updates remain the stronger default, and regret-style curricula slot into them directly. The loop we investigate is a serious alternative for sparse or delayed feedback: the environment selects which episodes count, and periodically a fresh LoRA adapter is trained on a bounded window of recent successes with a modified SFT loss, where the loss changes and the data is the agent’s own accepted episodes. Because acquiring experience and optimising on it are separate steps with no explicit policy gradient between them, we call the result approximate policy-free online learning. The curriculum pieces above decide what experience is there to acquire, and the open questions are the ones any sliding-window schedule raises, stability across adapters and forgetting between windows, with rollback to the previous adapter as a safety net rather than the mechanism.
A third option skips the question of where tasks come from: put the agent in a real environment and treat any adaptation that keeps it operating as valid learning, with the environment deciding by its own state which episodes count. That is the setting of negative-space learning, and the NSL Workbench repository implements a recursive self-training loop on it: the agent attempts tasks, the successful attempts shape the training data, and the trained model attempts the next round.
FAQs
Is PowerPlay a form of curriculum learning?
Yes, with the curriculum generated by the learner. Classic curriculum learning orders a fixed dataset from easy to hard by a hand-designed score. PowerPlay has no dataset: the next task is whatever the search finds cheapest among the tasks the current solver fails, so the easy-to-hard ordering is a by-product of the solver’s own state.
Does procedural generation guarantee generalisation?
No. It makes generalisation measurable, because held-out seeds exist, and Procgen showed that agents trained on a few hundred levels overfit to them. Moving the held-out score takes a broad generator plus a curriculum aimed at the agent’s frontier, and even then the agent generalises only within the family the generator can produce.
Can an LLM replace the procedural generator?
Partly. A model can write environment code and reward functions, as OMNI-EPIC does, and environment evolution for terminal agents (Fan et al., 2026) raises shell-task difficulty generation by generation for models such as Claude Opus 5. What it cannot supply alone is the acceptance gate: whether a task is solvable, novel and worth training on.
What is the difference between PowerPlay and procedural generation?
PowerPlay searches inside the agent for the cheapest new task that the current solver cannot yet do, together with a solver change that fixes it. Procedural generation produces tasks from outside, using a seeded generator, and the curriculum becomes a sampling policy over that distribution. They differ in the cost and strength of their acceptance gates.
What is prioritized level replay in curriculum learning?
Prioritized Level Replay keeps random level generation and adds a curator. Each level seen gets a learning-potential score based on value loss, and the next level is sampled from that score mixed with a staleness term. The result is an implicit easy-to-hard curriculum without training a separate teacher network.
