Engraved illustration of two scenes: a slate of pinned notes being wiped clean beside a candle and hourglass, and a brass machine fitted with a glowing amber cartridge while stamped cards wait in a tray
Notes in context are quick to write and quick to wipe; an adapter in the weights takes a training run to fit and stays.

TL;DR: An agent that adapts at test time can keep what it learns in its context, as a scratchpad or an episodic memory it re-reads on every call, or in its weights, as a LoRA (Low-Rank Adaptation) adapter trained on the trajectories that worked. The first is fast, cheap and disposable; the second is slow, persistent and changes what the model does before it reads anything. You can use either; most of our own deployed loops need both, and thus the useful question is which lessons belong at which timescale.

What Test-Time Adaptation Means for an LLM Agent

Test-time adaptation, where an agent keeps adapting after deployment, is a building block of recursive self-improvement.

Test-time adaptation is the family of methods that change a trained model’s behaviour using the test inputs themselves, after training has finished and without labels. The idea comes from computer vision. Sun et al. (2020) attached a self-supervised rotation-prediction head to a classifier and took a few gradient steps on each test image before classifying it, so the network could follow a distribution shift it had never seen in training.

The survey by Liang et al. (2024) sorts the field by what the model sees at test time: a whole target domain, a single batch, or a stream it must adapt to online. An LLM (Large Language Model) agent sits in the last category. Its test input is an environment plus its own interaction history, and it has two places to store what it learns from that history: its activations, through the context window, and its weights, through gradient updates. The rest of this post is about choosing between them.

The Fast Route: Scratchpads and Episodic Memory in Context

The in-context route is older than Transformers. Hochreiter et al. (2001) trained an LSTM (Long Short-Term Memory) network by ordinary gradient descent on many small function-fitting tasks and showed that the trained network ran a learning algorithm inside its activations: given a few input-output pairs of a new function, it fitted the function with no weight change at all. In-context learning in LLMs is the same effect at scale.

Explicit scratchpads give that capacity a workspace. Nye et al. (2021) let a model emit intermediate computation into a scratchpad before its final answer and found large gains on multi-step arithmetic and program execution; chain-of-thought prompting then made the same move general. Within one episode, the scratchpad is where an agent keeps the state of its task: what it has tried, what the tools returned, what the last error said.

Episodic Memory Across Episodes

Across episodes the scratchpad becomes episodic memory. Reflexion (Shinn et al., 2023) has the agent write a short verbal reflection after each failed trial, stores the reflections in a capped buffer, and prepends them to the next attempt; with no weight update at all it lifted HumanEval pass@1 to 91 percent for the model it wrapped. This is test-time learning entirely in the activations: the lesson exists only as text the model reads again.

The Slow Route: Test-Time Training into LoRA Adapters

The weight-level route takes the gradient step for real. Tandon et al. (2025) turn long context into a continual learning problem: a standard Transformer with sliding-window attention continues next-token training on the context it is reading, compressing that context into its weights, and their 3B models scale with context length the way full attention does while keeping constant inference latency. Context that the fast route would re-read on every call is, in that design, written into parameters once.

LoRA Adapters as the Unit of Test-Time Training

For a deployed LLM the practical unit of a weight update is a LoRA adapter: a low-rank pair of matrices on a frozen base, cheap to train, small to store and swappable per request. Akyürek et al. (2024) trained one adapter per task on augmented versions of the few demonstration pairs each ARC (Abstraction and Reasoning Corpus) puzzle provides, and an 8B model reached 53 percent on the public validation set, with per-task adapters beating a single shared one.

The Selection Problem in Test-Time Training

What makes this route expensive is not the gradient step but the selection problem before it. ARC hands the model demonstrations; an agent in an environment has only its own trajectories and must decide which of them deserve to become training data. Moradi et al. (2025) answer with a learned verifier: sample several candidates per query, keep the highest-scoring one above a threshold, and train only low-rank adapter parameters on the pairs that survive, for up to a 32 percent relative gain over the base model on their benchmarks.

Choosing the Timescale: Where Each Lesson Belongs

In context (scratchpad, episodic memory)In weights (LoRA test-time training)
What changesActivations, through the promptAdapter parameters on a frozen base
Takes effectOn the next callAfter the next training run
PersistsUntil the buffer is trimmed or clearedUntil the adapter is replaced
CapacityBounded by the context windowBounded by adapter rank and data
ReversalDelete the textRoll back to the previous adapter
NeedsNothing beyond inferenceGPU time and a selection signal
Failure modeAttention dilution, stale notes re-readForgetting, training on bad trajectories

Most test-time adaptation decisions reduce to one split. Information local to one task or one episode (the state of the file system, which hypotheses were tried this session, the error the last run produced) belongs in context, because it will be false next episode and costs nothing to discard. Regularities that recur across episodes (how this environment’s tools behave, which strategies keep paying off, the conventions of a codebase) belong in weights, because carrying them in context means re-reading the same lesson on every call and hoping attention finds it.

Two further tests settle the hard cases. If the environment changes faster than you can train, stay in context: an adapter trained on last week’s regularities is a liability this week. And if a lesson has recurred often enough that you would paste it into the system prompt, that is the signal to train it into an adapter instead, because a system prompt is a scratchpad nobody consolidated. Re-read memory is not learning, and the system prompt is where that confusion usually hides.

Running Both: Scratchpad Inside the Episode, Adapters Between Episodes

The two routes are nested, not rival. In the self-training agent loop we are investigating, the agent keeps a textual scratchpad that persists across episodes within a session, capped in length so it stays readable, and writes to it as it explores, plans and executes. That is the fast route, and it carries the episode-level state that would be wasteful to train on: current hypotheses, tool output, the last failure.

Adapters Between Episodes in a Self-Training Loop

Between generations the slow route runs. The environment decides which trajectories succeeded, and those alone, from a bounded window of recent generations, become training rows. A fresh LoRA adapter is then trained on them with a modified SFT (Supervised Fine-Tuning) loss (loss changes only, on a frozen base) and swapped in for the next generation; the adapter and the loss are two separate components, one bounding what can move and the other how far. Because nothing in that loop optimises a reward, and the only selection is which experiences the environment allowed to persist, we call the mechanism approximate policy-free online learning. Nightly LoRA training is the same loop on a slower clock.

The comparison with reinforcement learning should be stated plainly. Where a reliable verifier exists, reinforcement learning with verifiable rewards, GRPO (Group-Relative Policy Optimisation) included, is the stronger default, and the verifier-filtered test-time training above is close kin to it. The filtered-experience loop is a serious alternative under investigation for environments whose only feedback is sparse, delayed, or visible as the agent’s continued operation, and it raises its own questions about stability and forgetting. Rolling back to the previous adapter is a safety net, not the mechanism; the mechanism is environment-driven selection.

The tooling for the slow route is standard. PEFT (Parameter-Efficient Fine-Tuning) trains the adapters on a frozen base, often in 4-bit through QLoRA (Quantised LoRA), and inference servers such as vLLM load several adapters at once and pick one per request without reloading the base model, so swapping in a new generation’s adapter is a configuration change rather than a deployment. The fast route needs only a buffer with a cap and a prompt template that reads it.

FAQs

Is in-context learning a form of test-time adaptation?

Yes, by the definition: the model’s behaviour changes on the test input without labels or gradient steps. It is adaptation at the activation level, and a trained Transformer can implement something close to a gradient step in context on its examples. What differs is persistence: the adapted state lasts as long as the context, and the weights stay unchanged.

Does test-time training need labels?

No. Vision methods use a self-supervised auxiliary task or entropy minimisation; language models use next-token prediction on the context itself; agents use the environment’s outcome as a filter and train only on trajectories that succeeded. Each case manufactures its training signal from the test data, which is what separates test-time adaptation from ordinary fine-tuning on a curated set.

How do I stop weight-level adaptation from forgetting?

The remedies for catastrophic forgetting under continual fine-tuning apply unchanged: keep the base frozen and train fresh LoRA adapters, bound the training window so stale data expires, interleave a few general rehearsal rows, and anchor the loss to the reference model’s distribution. Keep the previous adapter for rollback, and treat rollback as a safety net.

When should I pick a scratchpad over a LoRA update?

When the lesson is specific to this episode, when the environment changes faster than you can train, or when you cannot yet tell which trajectories deserve to become training data. A scratchpad costs nothing to try and nothing to discard. Move a lesson into weights once it has recurred across enough episodes to justify the GPU time.

What is test-time training for LLMs?

Test-time training continues gradient updates on the input a model is reading, after pretraining has finished and without external labels. A long-context variant trains the model with next-token prediction on its context, compressing that context into weights. For agents, the environment’s outcome acts as the filter for which trajectories become training data.