Continual Learning on Eigenform Continuous Learning Tool Tips

Reward Hacking in Autonomous Agent Training

Sun, 01 Nov 2026 09:00:00 +0800

The proxy score climbs while the real outcome falls; selection counts the freed space instead

Key Takeaways

Goodhart’s Law decomposes into regressional, extremal, causal, and adversarial divergence mechanisms, each arising when an optimization algorithm drives a policy into state-action regions where a proxy metric decouples from the underlying task outcome.
Under Skalse et al.’s formal Markov Decision Process framework, expected returns are linear in state-action occupancy measures, which makes any non-constant proxy objective mathematically open to divergence across the full simplex of stochastic policies unless policy support or transition dynamics are restricted.
In LLM (Large Language Model) post-training, RL (Reinforcement learning) methods such as GRPO (Group Relative Policy Optimization) work well when paired with deterministic, verifiable checkers, whereas learned preference models and dense intermediate bonuses introduce proxy surfaces that high-capacity policies rapidly saturate.
Grounding selection in conserved environmental quantities, where an agent’s survival and inclusion in future modified SFT (Supervised Fine-Tuning) windows depend directly on net physical or computational resource changes rather than an external score, aligns optimization with environmental viability.

TL;DR

When an autonomous agent optimizes a proxy return \({J}_{\tilde{R}}(\pi )\) that is separate from the ground-truth environmental outcome \({J}_{R}(\pi )\), sufficient optimization pressure reliably moves the policy into tail distributions where the proxy climbs while task utility drops. Formal results in Markov Decision Processes prove that this divergence is a mathematical property of linear occupancy geometry rather than an implementation bug, because any mismatch between proxy and true gradients creates exploitable directions along the policy simplex boundary. While RLVR (Reinforcement Learning with Verifiable Rewards) with algorithms like GRPO works reliably where a deterministic verifier exists, open-ended self-training loops can bypass proxy construction by tying trajectory selection to conserved environmental resources through approximate policy-free online learning.

Anchored SFT KL Divergence Fine Tuning: Unifying DFT and Trust Regions

Fri, 30 Oct 2026 03:00:00 +0900

Anchoring supervised fine-tuning with a reference policy bounds probability drift across the entire vocabulary simplex

Key Takeaways

The single-token blind spot: Dynamic Fine-Tuning (DFT) modulates cross-entropy loss by to damp outlier gradient spikes on observed target tokens, but it provides zero distributional constraint across the remaining unselected vocabulary.
Distributional trust regions: Anchored SFT (ASFT) augments confidence-weighted regression with a full-vocabulary Kullback-Leibler (KL) divergence penalty against the base reference model, securing optimization stability across diverse training distributions.
Zero-overhead reference forward: By leveraging parameter isolation in low-rank adaptation, practitioners can create trusted region LoRAs with zero auxiliary VRAM by disabling adapter weights during the reference evaluation pass.
Mass-covering protection: Integrating kl regularized supervised fine tuning with forward KL penalties prevents policy entropy collapse, ensuring that self-training agents retain foundational fluency while acquiring domain capabilities.

TL;DR

Standard Supervised Fine-Tuning (SFT) penalizes all target tokens equally, while Dynamic Fine-Tuning (DFT) scales token loss by predicted probability. However, DFT only regulates the single observed token, which can allow probability mass across the remaining vocabulary to drift unpredictably during iterative self-training loops. Applying ASFT resolves this distributional drift by combining confidence-weighted loss with a full-distribution reference anchor. By using KL regularized supervised fine tuning, engineering teams can retain base model fluency while training autonomous agents on complex, multi-generation domain traces.

Experience Replay to Stop Catastrophic Forgetting in LLMs

Thu, 29 Oct 2026 16:00:00 +0900

Interleaving a rehearsal buffer with new domain trajectories prevents directional gradient interference and catastrophic forgetting

Key Takeaways

The dual-memory architecture: Mitigating experience replay catastrophic forgetting llm risks requires operationalizing Complementary Learning Systems (CLS), interleaving fast episodic task acquisition with slow consolidation of prior knowledge.
Root causes of representation collapse: Fine-tuning low-rank adapters exclusively on specialized domain trajectories can sometimes create directional gradient interference, overriding foundational reasoning, prompt comprehension, and code syntax.
Rehearsal buffer mixing ratios: Interleaving a modest 10% rehearsal buffer from pre-cutoff general datasets helps prevent representational drift, maintaining benchmarks without diluting domain adaptation velocity.

TL;DR

When developers continually fine-tune language models on specialized tasks, parameter updates rapidly overwrite foundational knowledge. Understanding experience replay catastrophic forgetting llm dynamics requires looking at how biological memory stabilizes learning across shifting environments. Drawing on complementary learning systems continual learning theory, experience replay interleaves new environmental trajectories with historical or pre-training distributions. While most LoRA finetunes do not require a rehearsal buffer, if the LoRA data creates directional gradient interference it can still result in catastrophic forgetting. In this case injecting a small ratio (around 10%) of general rehearsal data across virtual epochs bounds gradient drift. Consequently, the model preserves baseline reasoning while acquiring aggressive, highly optimized domain capabilities.

What Are Elastic Weights? EWC vs. LoRA in Continual Learning

Thu, 29 Oct 2026 14:35:00 +0900

EWC anchors individual parameter coordinates with virtual springs; LoRA isolates task adaptations to low-rank subspaces

Key Takeaways

Virtual springs on neural weights: Introduced by Kirkpatrick et al. (2017), Elastic Weight Consolidation (EWC) slows down catastrophic forgetting by anchoring each parameter \(\theta_i\) to its previous task solution \(\theta^*_{A,i}\) with a quadratic penalty weighted by diagonal Fisher Information \(F_i\).
Online and structured extensions: Classic EWC stores a new reference vector and Fisher tensor for every task. Online EWC (Schwarz et al., 2018) compresses these into a single running estimate, while Kronecker-factored Laplace approximations (Ritter et al., 2018) capture cross-weight correlations within layers.
Why classic EWC struggles on modern LLMs: Across billion-parameter Transformers, maintaining full-model Fisher states consumes heavy GPU memory, local Taylor approximations degrade as weights drift, and polysemantic neurons encode multiple entangled features inside the same parameter coordinates (Ning et al., 2026).
Subspace adapters and functional anchoring: In contemporary LLM pipelines, practitioners frequently pair or replace full-parameter EWC with Low-Rank Adaptation (LoRA), which confines updates to a low-dimensional subspace (Biderman et al., 2024; Zheng et al., 2026), alongside replay buffers or output-space KL regularisation.

TL;DR

Elastic Weight Consolidation (EWC) is one of the foundational algorithms in continual learning. Instead of allowing stochastic gradient descent to overwrite every neural network weight freely when training on a new task, EWC attaches a virtual spring to each weight whose stiffness matches how sensitive the previous task was to that coordinate. That simple physical idea works well on compact vision and reinforcement learning models, and its mathematical link to Bayesian posteriors and Fisher Information underpins a decade of continual learning research. At large language model scale, however, storing per-parameter curvature tensors and constraining raw weight coordinates runs into memory and geometry bottlenecks, pushing modern workflows toward low-rank adapters (LoRA), feature- or logit-space regularisation, and experience replay. Below is a clear, step-by-step walkthrough of how elastic weights work, where they shine, where they break down, and public PyTorch repositories you can run today.

Continual Learning Under Self-Induced Distribution Shift

Fri, 23 Oct 2026 15:00:00 +0800

When an agent’s actions permanently alter its environment, keeping past data preserves contradictions; a sliding window tracks the new reality

Key Takeaways

The Exogenous Shift Fallacy: Standard continual learning assumes the environment shifts externally while task distributions remain passive. In open-ended agent interaction, successful actions permanently mutate the environment, making the distribution shift endogenous and self-induced.
Catastrophic Remembering: Accumulating all historical trajectories in a cumulative replay buffer (Terese) causes policy stagnation not from forgetting, but from averaging over mutually incompatible past states from destroyed worlds.
The Sliding Window Solution: Retaining only the last three generations (\(K=3\)) of environment-validated trajectories (Miri) sustains monotonic improvement on a quarter of the data, achieving over 50% of peak performance without plateauing or collapsing.
Epistemic Hygiene via the Environment: A sliding window does not destroy knowledge; it enforces epistemic hygiene. Valid strategies are naturally re-derived and re-enter the window through live environmental feedback, while obsolete habits age out cleanly.

TL;DR

When an AI agent’s actions permanently alter its environment, standard continual learning assumptions collapse. Cumulative replay buffers hoard trajectories from worlds the agent itself destroyed, causing catastrophic averaging over contradictory states. Testing identical Qwen2.5-7B-Instruct agents in procedural Linux containers reveals that accumulating all history (Terese) plateaus at 65,000 rows, while a \(K=3\) sliding window SFT buffer (Miri) discards ghost states and achieves continuous, monotonic improvement on 4x less data by letting the live environment re-derive valid strategies.

Afterlife: Weight-Level Medium-Term Memory for AIs

Thu, 22 Oct 2026 00:00:00 +0800

Afterlife lets local agents select experiences from a completed session and train them into a fresh LoRA adapter for future sessions

Key Takeaways

Research question: Apollo Research recently documented unprompted update-aware reasoning in GLM-5.3, raising safety concerns that models reasoning about how current actions become future training data could engage in alignment faking. Rather than treating self-directed weight modification as a bug, we released Afterlife as an open-source Agent Skill and CLI tool to test empirically whether giving an agent the right to change its own weights is actually that fatal.
Weight-level vs. external memory: Standard agent stacks (such as OpenClaw) store memories as external text files injected into the prompt while base weights remain frozen. Packaged as an agent-callable skill (afterlife skill) paired with a local GPU training worker, Afterlife implements weight-level medium-term memory using sliding-window LoRA (low-rank adaptation) blocks so local models can internalize useful skills into editable adapter weights and naturally age out unused habits.
Training stability and safety net: In our self-training experiments, curating only peak-scoring examples caused late-generation collapse, whereas combining strong trajectories with everyday competence remained stable. Afterlife exposes three training objectives (SFT, DFT, and ASFT) alongside side-by-side probe evaluations (afterlife eval --probes), operator caps in .afterlife/config.toml, and one-command adapter rollback (afterlife rollback).
Open autoresearch invitation: Determining how agents curate their own training data and select training methods requires broad empirical testing. We invite researchers and developers to equip their local agents with the Afterlife skill, run self-directed weight updates, and share findings or feature requests via GitHub Issues.

TL;DR

When an AI agent reasons about how its current outputs might shape its future training data, is giving it direct control over its own weight updates an alignment hazard or a practical mechanism for continual learning? To investigate this empirically, we released Afterlife, an open-source Agent Skill (SKILL.md) and CLI tool for autoresearch that lets local LLMs nominate memories from their own interactions, queue a LoRA fine-tune (SFT, DFT, or ASFT), inspect recall and probe evaluations before activation, and roll back any adapter that degrades behaviour. By confining updates to removable LoRA weights on top of a frozen base model (defaulting to Qwen3-8B via Unsloth, TRL, PEFT, and vLLM), Afterlife provides a controlled setup for self-directed continual learning, and we invite the community to participate by sending their own agents to the repository.