Reproducible ML Pipelines and Recipe Hash Provenance

TL;DR: Static supervised fine-tuning pipelines treat dataset preparation as a one-off preprocessing script, but continual agent loops regenerate their own fine-tuning corpus at every generation boundary. If a post-collection transform silently drops a metadata key, mutates a prompt without recording lineage, or leaves a partially flushed JSONL file on disk during a process crash, the resulting weight update permanently diverges from its intended specification. In nsl-workbench, src/training_data/transforms.py we close these failure modes by validating a strict 19-field row schema with parent_row_id ancestry checks, compiling sequential TrainingDataTransform stages into a canonical SHA-256 recipe_hash with exact row-level deltas, and enforcing atomic directory publication verified by path canonicality and cryptographic digests before each training pass begins.

Why Iterative Self-Training Demands Reproducible ML Data Provenance

Reproducible ML pipelines let a team trust that each generation of a self-improving agent was trained on exactly the data it claims, which is essential for recursive self-improvement.

When an engineering team fine-tunes a language model on a static benchmark corpus, a bug in a preprocessing script is an isolated inconvenience: once spotted, an engineer fixes the script, re-runs the export, and overwrites the output file. Enforcing reproducible ML inside an iterative self-training loop is a harder systems problem because each generation’s model checkpoint determines which actions the agent takes in the next generation’s environment containers. If Generation 3 ingests a silently corrupted or improperly filtered JSONL file, the resulting QLoRA adapter shifts the exploration distribution of Generation 4, which in turn alters the survivor pool of Generation 5. By the time an evaluation metric degrades several generations later, reconstructing whether the regression came from optimizer dynamics, environment drift, or a subtle change in row filtering is impossible without cryptographic provenance.

Pineau et al. listed missing training data, under-specified training procedures and unavailable code among the reasons ML results are hard to reproduce in a NeurIPS reproducibility report. In autonomous agent loops, episodes are collected, filtered and rewritten automatically between weight updates, so relying on manual checklists lets lineage drift unnoticed.

Datasheets and Machine-Verified Manifests

Standardizing provenance at the dataset artifact level is a core principle of data-centric AI, shifting engineering rigor from ad-hoc model tweaking to deterministic corpus construction. Gebru et al. proposed datasheets for datasets, a standard document that records a dataset’s motivation, composition, collection process and recommended uses. In a continual learning system, that record cannot be a static markdown document written once by a researcher: it must be a machine-verified JSON manifest emitted and cryptographically hashed at every generation boundary.

Structural Row Lineage Versus Post-Hoc Gradient Attribution

When a fine-tuned model exhibits unexpected behavior on a downstream task, researchers often reach for gradient-based data attribution methods to identify which examples drove the parameter shift. Koh et al. demonstrated in the influence functions paper that classic robust-statistics influence functions can approximate the counterfactual effect of upweighting or removing a single training example via inverse-Hessian-vector products (IHVPs) without retraining the network from scratch. While this mathematical formulation works cleanly on convex objectives and small networks, applying it across multi-generation LLM trajectories introduces severe computational and structural bottlenecks.

Why Influence Functions Cannot Replace Row Provenance

Scaling second-order gradient estimators to billions of parameters requires curvature approximations that remain expensive across sequential self-training rounds. Grosse et al. showed in the EK-FAC influence paper that Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC) can scale influence functions to 52B-parameter LLMs to surface supporting pretraining sequences. However, when a 19-field trajectory row is rewritten, filtered by workflow step, or stripped of internal metadata before QLoRA optimization, gradient estimates on the final token sequence cannot tell you which upstream episode turn produced a mutated row or which transform dropped its siblings. Cryptographic row provenance and deterministic transform deltas solve this structural trace problem on CPU before the first forward pass runs.

Enforcing the 19-Field Row Schema and Parent-Row Lineage

In EigenformAI/nsl-workbench, the boundary between raw episode execution and supervised fine-tuning is governed by src/training_data/transforms.py. When an agent finishes executing an episode inside its container environment, src/harness/training_row_adapter.py converts each recorded inference turn (TrajectoryRecord) into a flat dictionary via record_to_row and enriches it with episode-level metadata via enrich_training_rows_for_episode. Before any filtering or rewriting transform runs, validate_training_row_groups inspects every EpisodeTrainingRows group and enforces a strict 19-field schema defined by REQUIRED_ROW_FIELDS.

What the 19 Required Fields Guarantee

Rather than permitting loose dictionaries where optional keys silently vanish when a harness changes, validate_training_row_groups computes REQUIRED_ROW_FIELDS - set(row) and raises a ValueError listing the sorted missing keys if any of the 19 fields is absent. Every field is also checked for strict JSON serializability via _assert_json_serializable, which runs json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=True) so non-serializable Python objects never leak into downstream files. The 19 required keys divide into four functional groups that preserve end-to-end data attribution from the raw inference call to the final SFT token stream:

1. Identity and Derivation Lineage (row_id, parent_row_id, source_episode_id, source_row_index): Every turn receives a globally unique row_id (f"{episode_id}:{source_row_index}"). When a TrainingDataTransform synthesizes a rewritten row from an existing turn, it assigns a new row_id and sets parent_row_id to the ancestor’s row_id, which validate_training_row_groups checks against source_row_ids and the current batch (allowed_parent_ids).
2. Supervision Payload and Turn Context (prompt, raw_response, interaction_type, source_interaction_type, workflow_step, actor_role): These fields capture the rendered conversation history (prompt), the clean model completion (raw_response), and the turn’s workflow role. Separating interaction_type from source_interaction_type lets a transform reclassify a turn while keeping its original harness label, and _validate_sft_rows confirms that every final prompt and raw_response is a str.
3. Whole-Episode Credit Assignment (episode_id, episode_index, generation_id, episode_score, episode_score_scope): In multi-turn environment selection, individual tool calls do not receive dense per-step proxy scores; the environment evaluates the final outcome of the full trajectory. To prevent turn-level credit confusion, validate_training_row_groups enforces row.get("episode_score_scope") == "whole_episode" and checks that row.get("episode_id") == group.episode_id.
4. Execution Diagnostics and Extensible Provenance (timestamp, success, error_message, record_meta): Turn-level execution status and structured task metadata live in record_meta. Tasks use record_meta to attach domain-specific provenance (such as pair_kind, artifact_route, or allowlisted public_provenance hashes in tasks/materials_adsorption/training.py), which _row_observability aggregates into sorted counters inside the export report.

REQUIRED_ROW_FIELDS = frozenset({
    "row_id", "parent_row_id", "prompt", "raw_response", "interaction_type",
    "source_interaction_type", "timestamp", "success", "error_message",
    "episode_id", "episode_index", "generation_id", "episode_score",
    "episode_score_scope", "source_episode_id", "source_row_index",
    "workflow_step", "actor_role", "record_meta",
})

def build_training_export(
    generation_data: Any,
    transforms: Sequence[TrainingDataTransform],
    context: TrainingDataExportContext,
) -> TrainingDataExport:
    recipe = build_export_recipe(transforms)
    groups = generation_data.get_successful_training_row_groups()
    validate_training_row_groups(groups)
    source_row_ids = set(_snapshot_row_index(groups))
    transform_reports: list[dict[str, Any]] = []
    for transform in transforms:
        before = _snapshot_row_index(groups)
        groups = transform.transform_export_rows(context, groups)
        validate_training_row_groups(groups, source_row_ids=source_row_ids)
        after = _snapshot_row_index(groups)
        transform_reports.append(_compute_transform_delta(transform, before, after))
    rows = _flatten_episode_groups(groups)
    _validate_sft_rows(rows)
    return TrainingDataExport(rows=rows, report={"per_transform": transform_reports}, recipe=recipe)

Deterministic TrainingDataTransform Pipelines and Cryptographic recipe_hash Provenance

General-purpose dataset libraries such as huggingface/datasets cache .map() and .filter() operations by hashing pickled Python functions and closure bytecode via dill. While automatic bytecode fingerprinting is convenient for interactive notebooks, it is fragile in long-running reproducible ML systems: minor Python interpreter upgrades, module import order changes, or harmless code formatting edits alter function bytecode and invalidate cache hashes even when the logical transformation is identical. Conversely, if a transform reads external mutable state that closure pickling misses, the fingerprint stays constant while the exported training data silently changes.

To eliminate bytecode fragility, src/training_data/transforms.py defines TrainingDataTransform as a @runtime_checkable Python Protocol with three explicit members: a name string property, a config() -> Mapping[str, Any] method, and transform_export_rows(context, episodes) -> list[EpisodeTrainingRows]. When build_export_recipe compiles a sequence of transforms, it inspects each transform at its sequence index, extracts its fully qualified class path via _transform_class_path (f"{cls.__module__}.{cls.__qualname__}"), validates that transform.config() returns a JSON-serializable mapping, and computes an individual config_hash using _json_hash(config_payload). Here _json_hash serializes any payload through _canonical_json (json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=True)) and computes its UTF-8 SHA-256 hex digest.

Canonical JSON and the recipe_hash

The ordered list of transform specifications is then wrapped into {"schema_version": 1, "transforms": transform_specs} and hashed once more with _json_hash to produce the pipeline’s recipe_hash. Because _canonical_json enforces lexicographical key sorting, compact , and : separators with no whitespace variance, and strict ASCII escaping, the resulting 64-character recipe_hash is completely invariant to dictionary insertion order, operating system locale, or Python runtime memory addresses. Furthermore, src/execution/generation.py records export_recipe_hash inside the generation manifest (ensure_generation_manifest), metadata.json, and checkpoint.json, binding the exact transform pipeline configuration to the generation’s execution lineage.

Auditing the NSL Workbench: Per-Transform Row Hash Snapshots and Deltas

In a self-training loop, understanding what was filtered out of the fine-tuning dataset is just as important as inspecting the rows that survived. Before the transform loop begins, build_training_export calls generation_data.get_successful_training_row_groups(), enforcing successful_episodes_only: True at the collection boundary so failed episodes are excluded first. It then records raw_successful_row_count and steps through the sequence of TrainingDataTransform stages, surrounding each stage with _snapshot_row_index(groups).

The helper _snapshot_row_index verifies that every row_id in groups is unique and maps row_id -> _canonical_row_hash(row), where _canonical_row_hash is the SHA-256 digest of the canonical JSON representation of the 19-field row dictionary. Immediately after transform.transform_export_rows(context, groups) returns and passes validate_training_row_groups(groups, source_row_ids=source_row_ids), a second snapshot after is taken and passed to _compute_transform_delta(transform, before, after). Comparing before_ids = set(before), after_ids = set(after), and retained = before_ids & after_ids yields five exact stage metrics:

• rows_in and rows_out: Total row counts (len(before) and len(after)) entering and leaving the stage.
• rows_removed (len(before_ids - after_ids)): Survivor turns dropped by the stage’s filtering criteria (such as non-rewrite workflow steps in MaterialsRewriteTransform).
• rows_added (len(after_ids - before_ids)): Synthesized row_id entries, each required to link to a valid ancestor via parent_row_id.
• rows_modified (sum(1 for row_id in retained if before[row_id] != after[row_id])): Retained rows whose 19-field payload changed (such as scrubbed record_meta), detected via SHA-256 inequality.

Reading the Export Report

All per-stage delta dictionaries are stored in report["per_transform"] inside training_data_export_report.json, alongside aggregate totals and task observability counters (rows_by_pair_kind, rows_by_artifact_route). If a task supplies a row_observability hook, _task_row_observability ensures its keys do not collide with reserved report fields before merging. When a generation yields an unexpected drop in trainable rows, engineers can inspect per_transform in training_data_export_report.json to isolate the responsible transform, or call regenerate_sft_export(generation_dir, task) to replay the export deterministically from all_episodes.jsonl.

Crash-Consistent Atomic Publication and Strict Path Verification

Even a deterministic transform pipeline fails reproducible ML standards if an out-of-memory kill, power loss, or storage timeout interrupts the process while writing sft_training_rows.jsonl. If a trainer resumes and reads a truncated JSONL file or a stale directory pointer, the next adapter trains on an unverified subset of the generation’s data. In src/training_data/transforms.py, publish_training_export eliminates partial-write corruption by implementing a multi-step atomic staging protocol backed by POSIX filesystem semantics and os.fsync.

First, publish_training_export validates export_id via _validate_export_id, enforcing that it starts with an ASCII alphanumeric character and contains only ASCII letters, digits, ., _, or - (blocking path-traversal tokens). Instead of writing to exports/sft/<export_id>/ directly, it creates a hidden staging directory on the same filesystem inside exports/sft/ using tempfile.mkdtemp(prefix=f".tmp-{active_export_id}-", dir=str(export_root)) and writes three artifacts using durable I/O helpers from src/typing/training.py:

1. sft_training_rows.jsonl (save_generation_training_data): Writes newline-terminated JSON rows, flushes the stream, calls os.fsync(handle.fileno()), and calls _fsync_directory on the parent directory descriptor (O_RDONLY | O_DIRECTORY).
2. export_recipe.json (save_durable_json): Writes export.recipe.to_dict() to a PID-and-UUID temporary file, calls os.fsync, atomically replaces the target via os.replace, and fsyncs the parent directory.
3. training_data_export_report.json (double-checked SHA-256): Computes rows_hash = _jsonl_sha256(rows_path) in 1 MB chunks, writes the report via save_durable_json, and re-computes _jsonl_sha256(rows_path) to verify the JSONL file did not mutate during staging.

Atomic Rename and the latest.json Pointer

Once all three files are durably flushed inside tmp_dir, publish_training_export executes tmp_dir.rename(final_dir) followed by _fsync_directory(export_root). Because tmp_dir and final_dir share the parent directory exports/sft/, rename is an atomic POSIX directory-entry operation: no reader can observe final_dir with missing or half-written files. Only after final_dir is durable does publish_training_export call save_durable_json(latest_payload, export_root / "latest.json") to commit the active pointer (export_id, relative paths, export_recipe_hash, training_row_count, and sft_training_rows_sha256). If an error occurs before latest.json commits, the except handler removes tmp_dir (or removes final_dir if latest.json does not match active_export_id), leaving any prior export intact.

Pre-Training Verification in Sliding-Window Self-Training Loops

Matching the optimization method to the task’s feedback structure is critical in agent continual learning. Reinforcement learning methods such as Group Relative Policy Optimization (GRPO) have proved effective with verifiable feedback, and where a reliable checker exists, reinforcement learning (RL) is the stronger default. We are investigating a serious alternative for regimes with expensive multi-turn container episodes: agents periodically train on successful experiences collected from their environments, using modified SFT (loss changes only, such as Dynamic Fine-Tuning or Anchored SFT) over bounded windows of recent generations paired with fresh LoRA adapters initialized from the frozen base model. In this setup, which we call approximate policy-free online learning, modified SFT and fresh LoRA adapters act as two separate components that decouple experience collection from reward optimization, with rollback kept strictly as a safety net.

Because each fresh LoRA adapter trains over a bounded window of recent generations, scripts/run_train_loop.py gathers window paths via _collect_training_window_paths(generation_root, end_generation_id, window_size), calling resolve_latest_sft_training_rows_path(generation_dir) on each generation folder. Sliding-window fine-tuning requires strict verification of every window entry: in a 3-generation window, one corrupted directory taints a third of the training data batch.

To block symlink hijacking, path traversal, or silent bit rot before QLoRA starts, resolve_latest_sft_training_rows_path enforces four checks:

• Relative Path Consistency: Validates export_id in exports/sft/latest.json and confirms sft_training_rows_path equals exports/sft/<export_id>/sft_training_rows.jsonl.
• Strict Path Canonicality (.resolve(strict=True)): Verifies that export_root, active_export_dir, and row_path each equal their own .resolve(strict=True) path, rejecting any symbolic link along the export tree.
• SHA-256 Digest Re-Verification: Confirms sft_training_rows_sha256 is a 64-character lowercase hex digest and matches _jsonl_sha256(row_path) on disk.
• Cross-Manifest Agreement: In scripts/run_train_loop.py, _generation_phase_complete verifies that active_sft_export_id in metadata.json matches export_id in latest.json.

FAQs

What makes an ML pipeline reproducible in a self-training loop?

Each generation must rebuild its training data deterministically and verifiably. That takes a strict row schema, a canonical hash of every transform and its configuration, per-stage row counts, and atomic publication of the export. With these in place, a team can trace any regression back to the exact data and recipe behind the adapter.

What is a recipe hash in machine learning?

A recipe hash is one SHA-256 digest of the ordered data transforms and their configurations used to build a training set. Two runs with the same recipe hash applied the same pipeline. Because the JSON is canonical, with sorted keys and fixed separators, the hash does not change with dictionary order, machine or Python version.

Why not fingerprint data transforms by pickling their code?

Pickled bytecode hashes change with Python versions, import order or formatting edits, even when the logic is identical. They can also miss external state that changes the output. Hashing each transform’s class path and JSON config is portable and stable, so the same pipeline produces the same recipe hash on any machine.

How do you trace a rewritten training example back to its source?

Give every row a unique ID, and set a parent_row_id on any row a transform creates from another. A validator then checks each parent against the source rows, so every rewritten example traces back to its original episode turn. This gives exact data attribution without running gradient-based estimators.

Should you use row provenance or influence functions to debug training data?

Use row provenance on every run and keep influence functions for deeper analysis. Provenance checks schema, row counts and hashes in milliseconds on CPU. Influence functions need costly inverse-Hessian approximations on GPUs, so they suit post-hoc questions about how specific examples shifted a model’s predictions.

How do you verify a training export was not corrupted before fine-tuning?

Publish each export atomically and verify it before training starts. The export is staged in a temporary directory, renamed in one step, and recorded in latest.json with a SHA-256 of the rows file. The trainer recomputes that digest and rejects symlinks, so it reads only the bytes attested at export time.