TL;DR

  • Writing a good benchmark by hand doesn’t scale: earlier attempts needed a human to review every batch of questions, and inconsistent judgment calls between batches produced incompatible rubric structures, duplicate evidence between questions, and scoring gates that wrongly zeroed correct answers.
  • The fix, and how we now build every Groundtruth benchmark, is a single fixed prompt that generates an entire 50-question benchmark - questions, reference answers, a machine-readable rubric and validation evidence - from a prepared set of documents, stopping for human approval exactly once.
  • That one stop is a blueprint allocating all 50 questions before any are drafted. Everything after it is checked by scripts instead of a person, because the specific defects a human review used to catch turned out to be ones a script catches just as well.

Key Takeaways

  • Reviewing every batch of questions by hand doesn’t produce a consistent benchmark. Earlier builds without a fixed structure produced three benchmarks with three incompatible rubric shapes, evidence reused across sections, and gates that zeroed mostly correct answers.
  • One human checkpoint, not many, is enough - if it’s the right one. The run stops once, at a blueprint allocating all 50 questions before any are drafted; a script can’t tell a lopsided allocation or evidence already claimed by another section from a good one, but a person can, in minutes, before any drafting happens.
  • Allocate every item before writing any of them. On one build, 9 of the last 22 questions collided with earlier ones because an early section had already used up the best evidence - invisible until you try to draft the last question, and easy to prevent by planning all 50 up front.
  • Four scripted gates replace what used to be manual review, and two passes stay manual. Blueprint counts, frozen questions, per-section rubric checks and final validation now run automatically; only running the judge against known, pre-scored answers, and an independent expert review of flagged items, still need a person.

Groundtruth Dynamic Benchmarking builds its question sets from source documents instead of writing them by hand - benchmark authoring handled by AI end to end. The skill that does this, build-source-grounded-groundtruth-benchmarking-geology, runs inside Claude Code against a prepared text corpus and produces the files the harness grades with.

The first benchmarks were built interactively, with a review after every authoring batch. The current flow is a single prompt that converges on a finished custom LLM benchmark: one round of LLM benchmark generation, structured the same way every time. This post describes how it is structured and which earlier problem each part answers — one piece of the wider process covered in how to benchmark AI models, and one input to comparing what different agents actually do with a benchmark like this, covered in AI agent benchmarks.

A fixed template with slots for measurements

The prompt, in full, is AUTHORING.md. Most of it is fixed text that applies to every corpus: how to allocate items, how gates work, how anchors are verified. A few bracketed slots take facts about the specific corpus, such as which document dominates and by how much, which extraction mode was used on each file, how many character-damaged strings there are and where, and which reporting codes are and are not in play.

The split exists because leaving the structure to judgement produced three benchmarks with three different shapes. Each rubric was valid against its own schema file, and together they were unreadable by one grading harness. The fixed template, and a fixed table of element shapes in the skill, now define the structure once.

The slots hold measurements and never rules. If a corpus seems to need a fixed rule changed, that is treated as a sign the rule is wrong for every corpus. The change goes into the template, the benchmarks already built are re-checked against the new rule, and only the items that fail it are revised.

The one stop is the blueprint

Before any question is drafted, the skill writes BLUEPRINT.md: a section scheme and a table allocating all 50 items, each with its id, section, difficulty, source document, the capability it tests, and the passage it intends to anchor to. The run presents that blueprint and waits for approval.

The earlier workflow stopped after every authoring batch as well. Those reviews mostly confirmed what scripts can confirm, and the real problems they caught, such as anchor errors, id collisions and template fixtures, are now machine checks. The blueprint review stayed because its catches are of a different kind: a section outside the declared scope, a lopsided allocation, evidence already used up by another section. A script sees a well-formed blueprint. A reader sees a wrong one. At that point a fix takes minutes; after drafting it can mean a rebuild.

On approval, the skill writes the line APPROVED into the blueprint and runs to the end without waiting again.

Allocate everything before drafting anything

Allocating all 50 items up front looks like process for its own sake until you skip it. On one build, 9 of the last 22 items collided with items already written, because an early cross-cutting section had already consumed the best evidence.

The template also sets balance rules at the blueprint stage.

  • Sections as even as the evidence allows. Ten sections of five, or eight of six or seven, are preferred over a lopsided split, because scores are aggregated without weighting and a large section would dominate the total.
  • A cap per source document. The cap comes from how rich each document’s content is, not from its page count. Without one, a corpus where a single report holds most of the words becomes a benchmark on that report with the others as appendices. Where several documents address the same question in different settings, cross-document comparison items are the most valuable items available.
  • A fixed difficulty profile. 10 Easy, 17 Moderate, 17 Hard and 6 Expert, with at least one Easy item in every section. Easy means a genuine reasoning on-ramp, never a trivia lookup.

Four machine gates

After approval, each stage has to pass a scripted check before the run moves on. On failure the skill fixes the problem and re-checks. The rubric structure each gate checks against is covered in Dynamic Benchmarking: How We Did It.

  1. Blueprint. Exactly 50 items, section and difficulty counts that add up to the declared profile, an Easy item in every section, a source document and intended anchor for every item, and per-document counts within the caps.
  2. Questions frozen. All 50 questions are written to the questions file before any reference answer, gate or component text exists anywhere, and the file is scanned for marking language or anything that reveals an answer.
  3. Per section, straight after authoring it. Components add up to exactly 10 with a single gate at C1 worth 2 to 4 points. Calibration fixtures reconcile, a failed gate zeroes every component, and each item has the four-fixture pattern. Every evidence locator passes a verbatim substring check against the extracted text, and a span audit is clean.
  4. Final. The skill’s own validation checks, schema validation of the machine-readable rubric, the element-shape table, and a validation document that records every judgement call, low-confidence claim and unresolved conflict.

Gate 2 approximates something the skill prefers to do with separate agents: writing questions without seeing the answers. In a single run, the questions file is frozen first and every rubric is written from the frozen question text and the corpus alone. If a rubric needs intent that the question does not state, the question is treated as defective and rewritten, instead of the intent being quietly added to the rubric.

Two reasons to stop early

The run may stop before finishing for exactly two reasons. The first is that the corpus cannot support 50 defensible items. Then the skill reports how many it can support and why, instead of padding the set with weak questions. The second is that the session is close to running out of capacity. Then it writes its state and stops cleanly. It never compresses quality to finish, because a template fixture or an unverified anchor is worse than an honest partial benchmark.

Resuming

The working files under _work/ are what make one prompt enough: the blueprint, a PENDING.json listing every item not yet fully built, and the checking scripts. Running the same prompt again is a resume. If the blueprint lacks the approval line, the skill presents it again. If it has it, the skill re-runs the machine gates over everything on disk, reads the pending list, and continues from the first unfinished item. It never regenerates finished items and never renumbers ids.

Rules that came from earlier builds

A few fixed rules in the template trace directly to problems we hit.

Continuous question ids. Questions are numbered straight through the sections: A1 to A5, B6 to B10, C11 to C15. Component ids always run C1, C2 and upward, with C1 as the gate, so a section-C question numbered C1 would collide with its own gate in the same grading call.

One claim per gate. A gate may test one core claim or one coherent mechanism, never a conjunction. A gate that requires naming four host rocks is a four-part gate however it is worded, and an answer correct on three of the four scores zero, the same as an answer about the wrong subject entirely. The gate tests the single distinction the item exists for, and the enumeration moves into the components. A two-way contrast such as distinguishing A from B counts as one claim.

No figures. The author can open the PDFs and see maps and cross-sections. The candidate reads extracted text. No question may depend on a map, section, stratigraphic column, drill plan or photograph.

Attribution. Where a corpus covers several properties or operators, no item attributes one party’s resource, grade or history to another, and a question that could be read either way names the property.

What stays manual

Two passes after the build remain non-optional, because neither the blueprint stop nor the machine gates can see how the rubric actually grades. Every calibration fixture is fed through the production judge and compared with its expected score, and every item the validation document flags gets an independent geological review.

The full prompt is in AUTHORING.md, and the fixture pass is described in a companion piece, “Testing the Rubric Before You Test the Model.” The principles behind the rubric design are covered in Dynamic Benchmarking: How We Did It, and this generation step is one of six in the wider process laid out in how to benchmark AI models. Once a benchmark like this exists, comparing what different agents actually do with it is a separate question, covered in AI agent benchmarks.