Decoupled Framework Design for Test-Time Learning Agents

TL;DR: Training language model agents across multi-container tasks often stalls because benchmark definitions, prompt formatting, tool schemas, and inference clients are tightly coupled inside monolithic runner scripts. Separating the domain contract (TaskSpec) from the turn-level execution loop (HarnessSpec), the inference wrapper stack (Genner and Backend), and the parallel slot scheduler (Execution) allows any task to run unchanged across in-process orchestrators or containerized third-party agents. This four-layer boundary preserves secret isolation during reward verification and produces uniform supervised fine-tuning (SFT) trajectories for iterative LoRA consolidation.

Why Monolithic Agent Loops Break Continual Learning Framework Design

A modular framework design makes it possible to run many self-training experiments cheaply, which matters for recursive self-improvement.

In early-stage agent repositories, the task definition and the execution loop usually live inside the same Python class. A typical runner script spins up a Docker container, builds a system prompt assuming a specific ReAct or XML tag syntax, calls an OpenAI client directly, parses shell blocks with regular expressions, and computes a task score at the end of the file. Sound continual learning framework design eliminates this entanglement by treating the domain problem and the turn-orchestration mechanism as independent axes of variation. When those axes are fused, evaluating whether an observed failure stems from poor environment feedback or brittle prompt orchestration becomes impossible.

Lessons From PyTorch Lightning and Avalanche

Standardized ML frameworks solved an analogous coupling problem during the transition from raw training loops to modular deep learning libraries. In PyTorch Lightning, researchers define model architecture, forward passes, and loss computation inside a LightningModule, while the Trainer and Strategy objects manage device placement, distributed data parallelism, mixed precision, and checkpointing. Separating model definition from training engineering allowed the same research module to scale from a single local GPU to multi-node clusters without modifying a line of forward-pass code.

In their continual learning research, Lomonaco et al. demonstrated a parallel architectural split in the Avalanche library paper, decoupling non-stationary benchmark streams (Scenario) from parameter adaptation algorithms (Strategy) and logging hooks (EvaluationPlugin). Bringing that separation to self-training LLM agents requires a further split because an agent interacts with stateful external containers over multiple turns before a single scalar reward or training row is emitted.

Lessons From Classical Environment Interfaces and Agent-Computer Interfaces

Reinforcement learning libraries established early environment abstractions through the step-and-reset loop. Towers et al. formalized the modern contract in the Gymnasium paper, where an Env exposes reset() and step(action) over typed spaces.Space definitions while wrappers transform observations or rewards outside the core simulation. However, multi-turn coding and systems agents rarely step through atomic integer actions against an in-memory simulator. Instead, an episode requires orchestrating multi-service Docker Compose stacks, seeding variation fixtures, verifying post-population container health, exposing capabilities as Model Context Protocol (MCP) tools or phase modes, and running out-of-band integrity checks after the agent terminates.

Recent software engineering agent research highlights how strongly the intermediate harness shapes model performance even when the underlying environment and base weights are held fixed. Yang et al. showed in the SWE-agent paper that introducing a dedicated Agent-Computer Interface (ACI) with windowed file viewers, syntax-checked edit commands, and concise diagnostic outputs raised pass@1 on SWE-bench to 12.5% and HumanEvalFix to 87.7%. When a continual learning system hardcodes one ACI or prompt harness into its task definitions, researchers cannot swap an in-process mode-switching harness for an external MCP container agent without rewriting every task’s Docker management and reward verification logic.

The TaskSpec Contract: Stateless Environment Lifecycle and Pure Reward

To make tasks portable across harnesses, the nsl-workbench repository defines TaskSpec(ABC, Generic[StateT]) in src/task/base.py as the sole contract between a domain benchmark and the surrounding runtime. Rather than inheriting from a monolithic runner, a task author subclasses TaskSpec parameterized by a domain state type StateT and declares seven mandatory class attributes: name, description, metric_name, metric_unit, higher_is_better, docker_compose_dir, and agent_service_name. An __init_subclass__ hook inspects concrete subclasses at import time and raises a TypeError immediately if any of the seven identity or environment attributes are missing. The task loader (src/task/loader.py) resolves the [task].class dotted path from the TOML configuration, instantiates the class with task_config: dict[str, Any], and runs validate() to confirm that list_variations() is non-empty, capability names are unique, and agent_service_name exists in the target Compose specification.

Why TaskSpec Must Stay Stateless

Because a single TaskSpec instance is shared across all parallel episode worker threads, this framework design enforces strict episode-level statelessness on the task object. Subclasses must never store mutable per-episode attributes on self, preventing race conditions when 16 worker slots execute concurrent episodes against separate container stacks. Instead, the execution engine drives each episode through a deterministic seven-step sequence (reset -> populate -> verify_population -> prompt_spec -> measure_initial_state -> [agent loop] -> finalize_episode) where episode-scoped state flows exclusively through method arguments and return payloads (episode_context, private_context, and EpisodeArtifacts).

# From src/task/base.py and src/harness/base.py
class TaskSpec(ABC, Generic[StateT]):
    name: str; description: str
    metric_name: str; metric_unit: str; higher_is_better: bool
    docker_compose_dir: str; agent_service_name: str

    @abstractmethod
    def populate(self, containers: list[Container], variation: Variation) -> PopulationOutcome: ...
    @abstractmethod
    def prompt_spec(self, variation: Variation, episode_context: dict[str, Any]) -> TaskPromptSpec: ...
    @abstractmethod
    def measure_initial_state(
        self, containers: list[Container], episode_context: dict[str, Any],
        *, private_context: dict[str, Any] | None = None,
    ) -> StateT: ...
    @abstractmethod
    def compute_reward(
        self, initial: StateT, final: StateT, artifacts: EpisodeArtifacts,
    ) -> TaskReward:
        """Pure reward computation: no containers, no I/O."""
        ...

class HarnessSpec(ABC):
    def __init__(self, harness_config: dict[str, Any]) -> None:
        self.harness_config = harness_config

    @abstractmethod
    def run_episode(self, *, ctx: "HarnessContext") -> "HarnessTranscript": ...

Pure Reward Evaluation and Declaring Capabilities Instead of Modes

Two architectural invariants in TaskSpec prevent subtle bugs that frequently corrupt self-training loops. First, TaskPromptSpec declares abstract capabilities (for example, run_python, run_shell, deploy_attack_sol, or probe_feature) rather than harness-specific orchestrator modes or vendor-specific tool schemas. The task also reserves framework phase tags (FRAMEWORK_RESERVED_PHASES = frozenset({"orchestrator", "cancelled", "system", "harness_error"})) so capability names never collide with harness boundary telemetry. Whether a capability is later presented to the model as a markdown code-block instruction inside an orchestrator-delegate prompt or exposed as a JSON-schema HTTP endpoint over an MCP server is decided entirely by the harness layer.

Second, compute_reward(initial: StateT, final: StateT, artifacts: EpisodeArtifacts) -> TaskReward is required to be a pure function with zero container access and zero I/O. All container inspection happens inside measure_initial_state and measure_final_state, which return typed state snapshots (StateT). When a task fails due to a broken Docker daemon or a timed-out docker exec call, the measurement method raises TaskEnvironmentError(message, container_ids=...) rather than returning TaskReward(success=False). Distinguishing infrastructure faults (TaskEnvironmentError, which triggers a targeted container rebuild via ContainerManager.rebuild_containers) from behavioural agent failures (TaskReward.success == False, which records a failed episode outcome) ensures that transient container crashes are never mistaken for negative task performance.

The HarnessSpec Seam: In-Process Orchestrators Versus Containerized Agents

Per-Episode HarnessSpec Instances

While TaskSpec is a stateless singleton, HarnessSpec (src/harness/base.py) follows a per-episode lifetime contract. The loader function construct_harness(config) instantiates a fresh HarnessSpec object at the start of every episode so that mutable turn counters, repetition guards, and scratchpad pointers never leak from one episode into the next without requiring error-prone reset methods. The abstract interface of HarnessSpec consists of a single required method, run_episode(self, *, ctx: HarnessContext) -> HarnessTranscript, alongside optional run_workflow and run_step hooks for multi-stage directed acyclic workflows. Any UI telemetry or failure diagnostics are attached through separate structural Protocol classes (HarnessTelemetry, HarnessRecorderTelemetry, and HarnessFailureExtras) so observability additions never bloat the core abstract base class.

HarnessContext as a Security and Telemetry Membrane

Communication between the harness and the rest of the runtime occurs strictly through HarnessContext (src/harness/context.py), which acts as a security and telemetry membrane. When run_episode builds HarnessContext, it passes the episode ID, TracedGenner, TaskSpec, Variation, TaskPromptSpec, public episode_context, container handles, EventRecorder, BudgetLedger, and a narrowed read-only HarnessConfigView. Crucially, private_context is omitted from HarnessContext by design. Secrets produced during populate(), such as an attacker private key in a smart-contract simulation, an upstream archive RPC URL, or held-out DFT reference labels in a materials discovery task, are held by the outer execution runner (src/execution/episode.py) and passed directly to measure_initial_state, measure_final_state, and finalize_episode. Because harness code never receives private_context, neither a buggy custom harness nor a prompt-injection payload inside a container can leak verification keys into the prompt context or the exported SFT training rows.

In-Process Mode Switching Versus Ephemeral Host Shims for External Containers

This narrow HarnessContext seam allows the runtime to support two structurally different agent architectures behind a single [harness] TOML block. The first built-in implementation, OrchestratorModeHarness (src/harness/orchestrator_modes/harness.py), runs entirely in-process without spawning an auxiliary harness container. Each iteration executes a two-stage hierarchical call pattern: an orchestrator turn reads the persistent scratchpad and remaining tool budget from BudgetLedger to select a cognitive mode (phase="orchestrator"), and a delegate turn executes under phase=mode_name to emit either a fenced code block or a structured response parsed by task.parse_response.

ContainerHarness for External Agent Frameworks

The second built-in implementation, ContainerHarness (src/harness/container.py), executes external Docker-packaged agent frameworks such as ModelScope ms-agent (MsAgentProfile) or the NVIDIA NeMo Agent Toolkit (aiq) against the exact same TaskSpec. At the start of an episode, ContainerHarness enters an ExitStack that mints a 24-byte URL-safe bearer token (secrets.token_urlsafe(24)), resolves a host-side scratch directory, and launches two ephemeral host servers bound to 0.0.0.0 on dynamic ports:

• OpenAiShim (src/harness/openai_shim.py): A per-episode FastAPI server exposing /v1/chat/completions that authenticates the container’s bearer token, forwards completions through TracedGenner, and rehydrates pseudo tool calls across seven text dialects whenever an underlying model emits raw XML or markdown tool syntax instead of native OpenAI tool_calls.
• CapabilityMcpBridge (src/framework/capability_bridge.py): An HTTP MCP server that dynamically translates the task’s TaskPromptSpec.capabilities into callable MCP tools while enforcing hard per-episode max_task_tool_calls and independent per-capability sub-budgets (max_task_tool_calls_by_name).

The profile adapter (src/harness/profiles/) renders the external agent’s native configuration pointing at those two host gateway URLs and launches the harness container. To prevent silent reproducibility bugs when Dockerfiles or driver scripts change, src/harness/provisioning.py stamps every locally built harness image with an nsl.build-context-sha label computed from a deterministic NUL-separated SHA-256 hash of all sorted build-context files, automatically triggering a --no-cache rebuild whenever image drift is detected. Furthermore, HarnessTranscript (src/harness/transcript.py) is never scraped for training data; instead, TracedGenner strips <think>...</think> reasoning prefixes into an audit field and logs every inference call directly to EventRecorder (events_<episode_id>.jsonl), producing uniform 19-field SFT rows across all harnesses.

Orthogonal Inference Backends, Parallel Slot Workers, and Weight Consolidation

Decoupling TaskSpec and HarnessSpec also isolates both layers from how tokens are generated and how episodes are parallelized across hardware. In src/execution/backend_runtime.py, inference requests traverse a four-layer wrapper stack: TracedGenner (per-episode event recording) wraps MetricsGenner (latency, token throughput, and GPU utilization sampling), which wraps EndpointAwareGenner (capacity-proportional endpoint leasing and reactive quarantine), which wraps the concrete Genner (OAIGenner, SglangServerGenner, LlamaGenner, or ClaudeGenner). Selecting a serving engine in this agent infrastructure requires changing only the model_name prefix in the TOML config (vllm:, sglang:, llama:, claude, or openrouter:), which dispatches through src/backend/resolver.py without touching a single line of task or harness code.

Thread-Per-Slot Parallel Execution

Above the inference layer, the execution engine (src/execution/__init__.py and src/parallel.py) scales data collection from sequential debugging (parallel_episodes = 1) to concurrent multi-container generation (parallel_episodes > 1) using a thread-per-slot worker model. Each worker thread owns an isolated WorkerSlot containing its own ContainerManager, dedicated DockerClient, host scratch directory (slot_<i>), and SlotCircuitBreaker (max_consecutive_failures = 3, paired with a 50% GlobalCircuitBreaker). Because every slot spins up an independent Docker Compose stack, estimate_slot_capacity preflights the Docker daemon’s bridge subnet pool (NETWORKS_PER_SLOT = 3) at startup so network exhaustion never crashes a multi-slot generation run.

Feeding Decoupled Trajectories Into Iterative Weight Consolidation

Cleanly separating environment verification, harness orchestration, and parallel execution directly improves downstream continual learning stability. When a reliable, fast-executing verifier exists and on-policy rollouts are computationally tractable, reinforcement learning with verifiable rewards (RLVR) using algorithms such as Group Relative Policy Optimization (GRPO) or Proximal Policy Optimization (PPO) is a well-established default that optimizes policies directly from positive and negative rollouts. However, in long-horizon container environments where episodes involve multi-step workflows, external MCP tool servers, or expensive compilation and simulation checks, coupling an online RL policy gradient loop directly to heterogeneous container harnesses introduces severe systems complexity and synchronization bottlenecks.

Experience Acquisition Decoupled From Weight Optimisation

As a serious alternative under investigation, the modular framework design in nsl-workbench decouples experience acquisition from weight optimization. During the generation phase, parallel worker slots execute episodes across any configured TaskSpec and HarnessSpec, and GenerationData.add_episode (src/typing/trajectory.py) routes only trajectories that pass the environment’s binary verification gate (TaskReward.success == True) into the exported SFT dataset (successful_episodes_only = True in src/training_data/transforms.py). Filtering complete surviving trajectories avoids brittle per-step reward shaping while preserving multi-turn error recovery sequences.

Subprocess Training Between Generations

Between generations, scripts/run_train_loop.py consolidates a bounded sliding window of recent generations (training_window_size, typically the last three generations) by training a fresh 4-bit QLoRA adapter (src/train/qlora.py) initialized from the frozen base model using modified SFT objectives (Dynamic Fine-Tuning, or DFT, in src/train/dft_fused.py to damp high-surprisal token gradients, and Anchored SFT, or ASFT, in src/train/asft.py for Forward KL regularization). Keeping fresh LoRA adapter resets and modified SFT loss functions as two distinct components inside a standalone subprocess trainer (python -m src.train.qlora) implements approximate policy-free online learning: CUDA memory is fully reclaimed between generation and training phases, and the newly trained adapter path is injected into the next generation’s backend configuration without modifying the task or harness layers. While this separation of experience collection from reward optimization raises open research questions around long-horizon plasticity and distribution shift, isolating the four runtime layers in standard ML tooling makes controlled ablation across tasks, harnesses, and loss functions straightforward.

FAQs

Why does TaskSpec declare abstract capabilities instead of OpenAI tool schemas or orchestrator modes?

Declaring harness-neutral Capability objects in TaskPromptSpec keeps the task definition independent of how an agent loop formats actions. An in-process OrchestratorModeHarness maps capabilities to markdown code blocks or task.parse_response hooks, whereas ContainerHarness exposes those exact same capabilities as HTTP MCP tools through CapabilityMcpBridge for external agents.

How does separating HarnessContext from private_context protect reward integrity in agent infrastructure?

During populate(), tasks place sensitive verification data such as private keys, held-out ground-truth labels, or upstream RPC credentials into private_context while exposing only public task metadata in episode_context. Because HarnessContext omits private_context, neither the agent harness nor the recorded SFT training trajectories can access or leak evaluation secrets.

Why should TaskSpec instances be stateless singletons while HarnessSpec instances are created per episode?

In thread-per-slot parallel execution, all worker threads share a single TaskSpec instance to reuse read-only variation catalogs and dataset caches without duplicating memory. Conversely, harnesses maintain mutable per-episode turn counters, scratchpads, and repetition guards, so constructing a fresh HarnessSpec per episode eliminates cross-episode state contamination.

How do environment abstractions distinguish container crashes from failed agent trajectories?

When a Docker container hangs or a service daemon crashes mid-episode, task methods raise TaskEnvironmentError with the broken container_ids rather than returning TaskReward(success=False). The slot worker catches TaskEnvironmentError, triggers a targeted container rebuild via ContainerManager, discards the aborted reservation, and avoids mislabeling infrastructure faults as negative agent behavior.

What is framework design for agent continual learning?

Framework design for agent continual learning splits an agent loop into independent layers: the task contract and scoring, the turn-level harness, the inference backend, and the parallel executor. Each layer can change without touching the others, so one task runs unchanged across harnesses, models and loss functions.