Anchored SFT KL Divergence Fine Tuning: Unifying DFT and Trust Regions

Fri, 30 Oct 2026 03:00:00 +0900

Anchoring supervised fine-tuning with a reference policy bounds probability drift across the entire vocabulary simplex

Key Takeaways

The single-token blind spot: Dynamic Fine-Tuning (DFT) modulates cross-entropy loss by to damp outlier gradient spikes on observed target tokens, but it provides zero distributional constraint across the remaining unselected vocabulary.
Distributional trust regions: Anchored SFT (ASFT) augments confidence-weighted regression with a full-vocabulary Kullback-Leibler (KL) divergence penalty against the base reference model, securing optimization stability across diverse training distributions.
Zero-overhead reference forward: By leveraging parameter isolation in low-rank adaptation, practitioners can create trusted region LoRAs with zero auxiliary VRAM by disabling adapter weights during the reference evaluation pass.
Mass-covering protection: Integrating kl regularized supervised fine tuning with forward KL penalties prevents policy entropy collapse, ensuring that self-training agents retain foundational fluency while acquiring domain capabilities.

TL;DR

Standard Supervised Fine-Tuning (SFT) penalizes all target tokens equally, while Dynamic Fine-Tuning (DFT) scales token loss by predicted probability. However, DFT only regulates the single observed token, which can allow probability mass across the remaining vocabulary to drift unpredictably during iterative self-training loops. Applying ASFT resolves this distributional drift by combining confidence-weighted loss with a full-distribution reference anchor. By using KL regularized supervised fine tuning, engineering teams can retain base model fluency while training autonomous agents on complex, multi-generation domain traces.

ASFT on Eigenform Continuous Learning Tool Tips

Anchored SFT KL Divergence Fine Tuning: Unifying DFT and Trust Regions

Key Takeaways

TL;DR