SFT vs RL for sparse-feedback agents: modified SFT trains a small adapter on a window of accepted episodes and archives the failures, GRPO groups that all failed give no gradient, and RLVR learns from a verifier that marks every rollout
Match the method to the signal you have

TL;DR: Where a fast, reliable verifier checks rollouts cheaply, online reinforcement learning such as GRPO or RLVR learns from both successes and failures and generalises further than supervised fine-tuning. However, when multi-turn agent episodes are expensive to sample, pass rates are very low, and only a binary end-of-episode check exists, most sampled RL groups have zero reward variance and produce no gradient. As an alternative under investigation, periodically retraining fresh LoRA adapters from the base model using modified SFT over a bounded sliding window of passing episodes separates experience collection from reward optimisation, a loop we call approximate policy-free online learning.

Why Choose Between SFT and RL at All?

The SFT vs RL question usually gets a quick answer. For post-training, reinforcement learning is the established default when rollouts can be checked reliably, because it optimises the task outcome directly and generalises beyond its demonstrations. That default assumes you can sample enough attempts per prompt for rewards to vary within a batch. Multi-turn agent workloads frequently break that assumption.

The trade-off becomes acute in continual recursive self-improvement deployments when five constraints coincide:

  • feedback is strictly binary: the environment detects whether an episode survived or passed, without grading partial progress;
  • success is rare: early policies fail on nearly every attempt;
  • each rollout is expensive: one attempt runs a multi-turn container or tool session rather than a single completion;
  • the environment shifts over time: APIs, repositories, and task distributions change after deployment;
  • compute is one or two GPUs: leaving little budget for parallel rollout workers or separate critic networks.

Without matching the algorithm to that regime, teams stand up an online RL pipeline only to discover that most rollout groups contribute zero gradient, or fall back to a one-off static SFT run that stops adapting the day the agent ships.

What Is the Difference Between SFT and RL?

Supervised fine-tuning (SFT) and reinforcement learning (RL) differ in how they turn trajectories into gradients. SFT raises the token likelihood of target trajectories under a cross-entropy objective, whereas RL samples completions from the current policy and weights updates by relative return so higher-scoring trajectories gain probability mass and lower-scoring ones lose it.

Classic SFT fits a fixed set of human demonstrations or distilled teacher traces once before deployment and never updates again. By contrast, the loop we are investigating pairs two distinct components for continual adaptation:

  • modified SFT trained exclusively on the agent’s own episodes that passed a task-owned check inside a bounded recency window;
  • fresh LoRA adapters retrained periodically from the frozen base model rather than stacked across rounds.

Because no scalar reward function specifies an optimisation direction in advance and updates run on periodic batches of surviving trajectories, we call this setup approximate policy-free online learning. Separating experience collection from reward optimisation introduces its own stability and forgetting trade-offs, yet it changes actual model weights, which matters because AI agent memory limits prevent prompt notes alone from building durable skill.

How Modified SFT and Fresh LoRAs Work in NSL

Negative Space Learning (NSL), our research loop, pairs those two components in four steps, with its simplest schedule running as scheduled nightly LoRA adapters over recent passing episodes:

  1. A deterministic task gate checks environment state after each episode; failed episodes are archived and contribute zero training rows.
  2. Every model turn inside a passing episode is exported as a training row labelled by the surviving trajectory.
  3. A fresh LoRA adapter trains from the frozen base model across a sliding window of the last training_window_size generations (default 3).
  4. training.inner_loss optionally replaces standard cross-entropy with probability-weighted Dynamic Fine-Tuning (DFT) or KL-regularised Anchored SFT (ASFT).

We evaluated PPO, DPO, and GRPO on our sparse container tasks before focusing on this loop, and we retain RL trainers in our research codebase for tasks with denser per-group variance.

SFT vs GRPO, PPO, and RLVR in Sparse-Feedback Environments

In sparse-feedback agent tasks, modified SFT plus fresh LoRAs and online RL algorithms differ in sample efficiency, memory footprint, and how they treat failed rollouts:

DimensionModified SFT + Fresh LoRAPPOGRPORLVR
Training signalBinary pass/fail from environment gateScalar return per rolloutGroup-normalised return across rolloutsBinary or graded verifier per rollout
Extra networks in memoryNoneLearned value criticNoneDepends on policy optimiser
Samples needed per updateAny single passing episodeLarge rollout batchesMultiple rollouts per promptMultiple rollouts per prompt
Learns from failed rolloutsNo (failures archived)YesYes, when group rewards varyYes, when group rewards vary
Primary bottleneckImitates noisy detours inside passing tracesCritic memory and high value varianceZero gradient when an entire group failsRequires a fast, cheap per-prompt verifier

PPO Memory Cost and Value-Network Variance

The Proximal Policy Optimization paper scores actions against a learned value critic before taking clipped policy-gradient steps. Holding a critic of comparable size alongside the policy roughly doubles training memory, and estimating value targets from rare binary outcomes at the end of long multi-turn episodes introduces high variance.

GRPO Zero-Variance Groups Under Rare Pass Rates

The DeepSeekMath GRPO paper removes the value network by sampling a group of completions per prompt and normalising each reward against the group mean and standard deviation: A_i = (r_i - mean(r)) / std(r). GRPO works well with binary rewards when rollouts are cheap, yet whenever every completion in a group receives the same 0 reward, advantages vanish.

At a 5% episode success rate with eight rollouts per prompt, an entire group fails with probability 0.95^8, or roughly 66%. Two thirds of rollout compute therefore produces zero gradient while still paying the full wall-clock cost of multi-turn container execution. Modified SFT extracts gradient signal from the very first episode that passes.

RLVR Strengths and Verifier Constraints

Reinforcement learning with verifiable rewards (RLVR), popularised in the Tulu 3 post-training report, pairs policy optimisation such as GRPO with deterministic checkers like maths answer matchers or fast unit tests. Where that check is cheap enough to run many rollouts per prompt, RLVR is the stronger default; RL with learned reward models or RLAIF covers broader tasks at the cost of proxy drift. Multi-turn infrastructure or research tasks sit in between: an end-of-episode check exists, but rollouts are slow and successes start sparse.

Where Modified SFT and Fresh LoRAs Differ From Online RL

Formally, filtered SFT and binary-reward policy gradients are closely related, yet four practical mechanics distinguish periodic sliding-window SFT plus fresh LoRAs from online RL.

Learning From Rare Single Successes

A single surviving episode immediately yields supervised training rows for the next batch window. Group-relative RL requires successes to co-occur alongside failures inside the same prompt group before moving the policy, making rare early breakthroughs slower to capture when episode budgets are tight.

Separating Experience Collection From Reward Optimisation

An explicit reward function or learned judge fixes the optimisation target ahead of time. By using the environment gate strictly to filter which episodes enter a bounded recency window, habits that stop surviving as tools or workloads shift age out automatically once older generations leave the window.

Bounding Drift With Sliding Windows and Fresh LoRAs

Retraining a brand-new LoRA adapter from the frozen base model each cycle ensures earlier rounds influence weights only through trajectories still inside the active window, while previous checkpoints remain on disk strictly as a rollback safety net if held-out probes regress. In our Negative Space Learning paper, the three-generation sliding-window lineage improved across 13 generations, whereas cumulative training plateaued and cherry-picking best generations collapsed.

Choosing the window length also governs the plasticity-stability balance central to continual fine-tuning forgetting across recurring deployment cycles.

Token-Level Reweighting With DFT and Anchored SFT

Standard cross-entropy treats every token inside a passing trajectory as equally desirable, including exploratory detours. Dynamic Fine-Tuning scales each token’s loss by the model’s current probability so high-surprisal noise pulls less strongly, addressing the implicit inverse-probability weighting analysed in the Dynamic Fine-Tuning paper.

Complementing token weighting, the Anchored SFT KL paper adds forward KL regularisation toward a frozen reference policy, which in our loop is simply the base model with the LoRA adapter disabled. Both objectives plug into training.inner_loss alongside the fresh-adapter schedule.

With on-policy rollouts and binary 0/1 outcomes, the expected REINFORCE gradient E[r * grad log pi(y)] matches the cross-entropy gradient on passing trajectories up to a success-rate scalar. Filtering rollouts by outcome places this approach in the rejection-sampling family introduced in the expert iteration framework alongside ReST and RAFT.

Reasoning bootstrapping methods such as the STaR self-taught reasoner apply the same filter-and-fine-tune principle to chain-of-thought rationales. Our loop departs from pure on-policy rejection sampling in three ways: its multi-generation window mixes trajectories from recent prior adapters, each cycle resets to a fresh LoRA on the base model, and DFT or ASFT modifies token-level loss weights.

Where Reinforcement Learning Outperforms Modified SFT

Online RL remains stronger than modified SFT on four fundamental dimensions:

  • Direct negative feedback: RL pushes probability mass away from failed actions within mixed-outcome groups, whereas modified SFT simply archives failed episodes.
  • True objective optimisation: RL maximises expected task return directly rather than imitating trajectories that happened to clear a threshold.
  • Broader out-of-distribution generalisation: Controlled experiments on SFT memorisation vs RL show RL transferring across unseen rule and visual shifts where SFT memorises training patterns.
  • Ceiling above existing traces: Policy gradients can sharpen action selection beyond average accepted trajectories rather than averaging noisy detours.

Consequently, modified SFT with fresh LoRAs is not a universal replacement for RL; where rollouts are cheap and verifiable, RLVR or GRPO is the right first choice.

How to Benchmark Modified SFT Against RL on Your Own Agent

Whether verifiable-reward RL creates new capabilities beyond base-model support remains actively debated: controlled evaluations in the RLVR reasoning boundary study show RLVR lifting pass@1 substantially while base models catch up at large pass@k. To compare modified SFT and online RL fairly on your own sparse-feedback agent, isolate the update rule:

  1. Hold the environment and budget constant: give every arm the same base checkpoint, task generator, and total episode wall-clock budget.
  2. Compare matched arms: run sliding-window modified SFT with fresh LoRA adapters alongside GRPO on the identical binary outcome signal.
  3. Log signal efficiency alongside task score: record pass@1, pass@k, the fraction of GRPO prompt groups with zero reward variance, and GPU hours per point gained.
  4. Evaluate held-out task distributions: test every generation on unseen environment variants to separate memorisation from generalisation.
  5. Run across multiple self-training generations: multi-round loops expose distribution drift and forgetting that single-epoch benchmarks miss.

Failure Modes When Comparing Modified SFT and RL

Keep four failure modes in view during multi-generation tests:

  • Loose gates teach sloppy habits: modified SFT imitates whatever clears the survival threshold, making gate design critical.
  • Episode-level labels reinforce detours: every turn inside an accepted multi-turn trace receives positive supervision unless token weighting tempers noisy steps.
  • Narrow windows risk memorisation: track held-out task variants every cycle and keep prior adapters ready as a rollback safety net.
  • Contradictory task pressures stall both paradigms: when one deployment demands opposing behaviours simultaneously, shared weights face conflicting multi-domain gradients under both RL and SFT.

FAQs

Which is better for agent post-training, SFT or RL?

Neither method wins across every regime. Online reinforcement learning such as GRPO or RLVR learns from failures, optimises expected return directly, and generalises further when rollouts are cheap and verifiable. Periodic modified SFT with fresh LoRA adapters is a practical alternative when multi-turn episodes are expensive and binary successes are rare.

What is the difference between modified SFT and GRPO?

Modified SFT trains on surviving trajectories inside a recent sliding window using cross-entropy or probability-weighted token losses. GRPO samples a group of completions per prompt and scales updates by within-group reward variance; it excels with fast verifiers, but yields zero gradient whenever every rollout in a group fails.

What is approximate policy-free online learning?

It is a continual-learning loop that pairs two components: periodic modified SFT over a bounded sliding window of environment-verified episodes and fresh LoRA adapters retrained from the frozen base model. Because no reward function sets a fixed optimisation direction and updates run in batches, it approximates online adaptation without policy gradients.

Does reinforcement learning require dense rewards to work?

No, reinforcement learning does not require dense step-by-step rewards. Algorithms such as GRPO and RLVR train effectively on sparse binary pass/fail checks whenever rollouts are cheap enough to sample mixed successes and failures per prompt; sparse binary feedback only stalls RL when multi-turn episodes are too costly to fill varied groups.

When should I use RLVR instead of modified SFT?

Use RLVR whenever an automatic checker (such as unit tests, compiler checks, or exact mathematical verifiers) evaluates rollouts quickly enough to sample multiple completions per prompt within your compute budget. Reach for modified SFT with fresh LoRA adapters when episodes take minutes in stateful sandboxes and early pass rates sit near zero.