Explore-Exploit Tradeoff Inside Online SFT Loops

Thu, 05 Nov 2026 09:00:00 +0800

Direct strategies chase immediate reward; instrumental strategies pay for future learnability when the whole episode trains the next adapter

TL;DR: In an online SFT loop, the policy at step \(t+1\) is trained on the filtered trajectories produced at step \(t\), turning every output into a candidate training row and creating a training-level explore-exploit tradeoff that a frozen policy never faces. A trajectory’s value splits into its immediate return plus a discounted improvement in the next policy’s future value, allowing selection over multiple update cycles to preserve low-return instrumental strategies (such as a fast first attempt that triggers an informative error message) without any curiosity bonus or exploration reward in the loss. Under RLVR the same tension appears as entropy collapse and a pass@1 gain that the base model undoes at large \(k\), which is why RLVR carries the exploration term explicitly through entropy controls and pass@k objectives while online selection loops carry it through the environment filter and update cadence.

Online SFT on Eigenform Continuous Learning Tool Tips

Explore-Exploit Tradeoff Inside Online SFT Loops