Reproducible ML Pipelines and Recipe Hash Provenance

Fri, 13 Nov 2026 09:00:00 +0800

Row lineage, recipe hash and atomic publication: the checks behind each training export

TL;DR: Static supervised fine-tuning pipelines treat dataset preparation as a one-off preprocessing script, but continual agent loops regenerate their own fine-tuning corpus at every generation boundary. If a post-collection transform silently drops a metadata key, mutates a prompt without recording lineage, or leaves a partially flushed JSONL file on disk during a process crash, the resulting weight update permanently diverges from its intended specification. In nsl-workbench, src/training_data/transforms.py we close these failure modes by validating a strict 19-field row schema with parent_row_id ancestry checks, compiling sequential TrainingDataTransform stages into a canonical SHA-256 recipe_hash with exact row-level deltas, and enforcing atomic directory publication verified by path canonicality and cryptographic digests before each training pass begins.

Data Attribution on Eigenform Continuous Learning Tool Tips

Reproducible ML Pipelines and Recipe Hash Provenance