TL;DR
- Writing a good benchmark by hand doesn’t scale. Earlier attempts needed a human to review every batch of questions. Inconsistent judgment calls between batches then produced incompatible rubric structures, duplicate evidence between questions, and gates that wrongly zeroed correct answers.
- The fix is a single fixed prompt, and it is how we now build every Groundtruth benchmark. From a prepared set of documents it generates an entire 50-question benchmark: questions, reference answers, a machine-readable rubric and validation evidence. It stops for human approval exactly once.
- That one stop is a blueprint allocating all 50 questions before any are drafted. After it, scripts check everything instead of a person. The defects a human review used to catch turned out to be ones a script catches just as well.
Key Takeaways
- One human checkpoint, not many, is enough, if it’s the right one. The run stops once, at a blueprint allocating all 50 questions before any are drafted. A script can’t tell a lopsided allocation or evidence already claimed by another section from a good one. A person can, in minutes, before any drafting happens.
- Allocate every item before writing any of them. On one build, 9 of the last 22 questions collided with earlier ones because an early section had already used up the best evidence. The problem stays invisible until you draft the last questions, and planning all 50 up front prevents it.
- Four scripted gates replace what used to be manual review, and two passes stay manual. Blueprint counts, frozen questions, per-section rubric checks and final validation now run automatically. Only two passes still need a person: running the judge against known, pre-scored answers, and an independent expert review of flagged items.
Groundtruth Dynamic Benchmarking builds each custom LLM benchmark from source documents instead of writing questions by hand. The skill that does the benchmark authoring, build-source-grounded-groundtruth-benchmarking-geology, runs inside Claude Code against a prepared text corpus. It produces the files the harness grades with, and the results feed the Groundtruth leaderboard.
From batch reviews to one prompt
We built the first AI model benchmarks interactively, with a review after every authoring batch. The current flow is a single prompt that produces a finished benchmark, structured the same way every time. This post describes how it is structured and which earlier problem each part answers.
A custom LLM benchmark from a fixed template
The prompt, in full, is AUTHORING.md. Most of it is fixed text that applies to every corpus: how to allocate items, how gates work, how to verify anchors. A few bracketed slots take facts about the specific corpus. Examples are which document dominates and by how much, and which extraction mode each file went through. Others record how many character-damaged strings there are and where, and which reporting codes are and are not in play.
The split exists because leaving the structure to judgement produced three benchmarks with three different shapes. Each rubric was valid against its own schema file, and together they were unreadable by one grading harness. The fixed template, and a fixed table of element shapes in the skill, now define the structure once.
The slots hold measurements and never rules. If a corpus seems to need a fixed rule changed, we treat that as a sign the rule is wrong for every corpus. The change goes into the template, we re-check the benchmarks already built against the new rule, and revise only the items that fail it.
The one stop is the blueprint
Before drafting any question, the skill writes BLUEPRINT.md. It holds a section scheme and a table allocating all 50 items. Each row gives the item’s id, section, difficulty and source document, the capability it tests, and the passage it intends to anchor to. The run presents that blueprint and waits for approval.
The earlier workflow stopped after every authoring batch as well. Those reviews mostly confirmed what scripts can confirm. The real problems they caught, such as anchor errors, id collisions and template fixtures, are now machine checks. The blueprint review stayed because its catches are of a different kind: a section outside the declared scope, a lopsided allocation, evidence already used up by another section. A script sees a well-formed blueprint. A reader sees a wrong one. At that point a fix takes minutes; after drafting it can mean a rebuild.
On approval, the skill writes the line APPROVED into the blueprint and runs to the end without waiting again.
Allocate everything before drafting anything
Allocating all 50 items up front looks like process for its own sake until you skip it. On one build, 9 of the last 22 items collided with items already written, because an early cross-cutting section had already consumed the best evidence.
The template also sets balance rules at the blueprint stage.
- Sections as even as the evidence allows. Ten sections of five, or eight of six or seven, beat a lopsided split. The harness aggregates scores without weighting, so a large section would dominate the total.
- A cap per source document. The cap comes from how rich each document’s content is, not from its page count. Without one, a corpus where a single report holds most of the words becomes a benchmark on that report with the others as appendices. Where several documents address the same question in different settings, cross-document comparison items are the most valuable items available.
- A fixed difficulty profile. 10 Easy, 17 Moderate, 17 Hard and 6 Expert, with at least one Easy item in every section. Easy means a genuine reasoning on-ramp, never a trivia lookup.
Four machine gates
After approval, each stage has to pass a scripted check before the run moves on. On failure the skill fixes the problem and re-checks.
- Blueprint. Exactly 50 items, with section and difficulty counts that add up to the declared profile. Every section has an Easy item, every item has a source document and intended anchor, and per-document counts stay within the caps.
- Questions frozen. The skill writes all 50 questions to the questions file before any reference answer, gate or component text exists anywhere. It then scans the file for marking language or anything that reveals an answer.
- Per section, straight after authoring it. Components add up to exactly 10 with a single gate at C1 worth 2 to 4 points. Calibration fixtures reconcile, a failed gate zeroes every component, and each item has the four-fixture pattern. Every evidence locator passes a verbatim substring check against the extracted text, and a span audit is clean.
- Final. The skill’s own validation checks, schema validation of the machine-readable rubric, and the element-shape table. A validation document also records every judgement call, low-confidence claim and unresolved conflict.
Gate 2 approximates something the skill prefers to do with separate agents: writing questions without seeing the answers. In a single run, the skill freezes the questions file first and writes every rubric from the frozen question text and the corpus alone. Sometimes a rubric needs intent that the question does not state. The skill then treats the question as defective and rewrites it, instead of quietly adding the intent to the rubric.
Two reasons to stop early
Both stopping conditions, and what the skill does at each one, are written into the AUTHORING.md prompt itself, not left to runtime judgement. The run may stop before finishing for exactly two reasons. The first is that the corpus cannot support 50 defensible items. Then the skill reports how many it can support and why, instead of padding the set with weak questions. The second is that the session is close to running out of capacity. Then it writes its state and stops cleanly. It never compresses quality to finish, because a template fixture or an unverified anchor is worse than an honest partial benchmark.
Resuming
The working files under _work/ are what make one prompt enough: the blueprint, a PENDING.json listing every item not yet fully built, and the checking scripts. Running the same prompt again is a resume. If the blueprint lacks the approval line, the skill presents it again. If it has it, the skill re-runs the machine gates over everything on disk, reads the pending list, and continues from the first unfinished item. It never regenerates finished items and never renumbers ids.
Rules that came from earlier builds
A few fixed rules in the template trace directly to problems we hit.
Continuous question ids
Question ids run straight through the sections: A1 to A5, B6 to B10, C11 to C15. Component ids always run C1, C2 and upward, with C1 as the gate. A section-C question numbered C1 would collide with its own gate in the same grading call.
One claim per gate
A gate may test one core claim or one coherent mechanism, never a conjunction. A gate that requires naming four host rocks is a four-part gate however it is worded. An answer correct on three of the four scores zero, the same as an answer about the wrong subject entirely. The gate tests the single distinction the item exists for. The enumeration moves into the components, in line with the rubric’s gate-and-components split. A two-way contrast such as distinguishing A from B counts as one claim.
No figures
The author can open the PDFs and see maps and cross-sections. The candidate reads extracted text. No question may depend on a map, section, stratigraphic column, drill plan or photograph.
Attribution
Where a corpus covers several properties or operators, no item attributes one party’s resource, grade or history to another. A question that could be read either way names the property.
What stays manual
Two passes after the build remain non-optional, because neither the blueprint stop nor the machine gates can see how the rubric actually grades. We feed every calibration fixture through the production judge and compare it with its expected score. In addition, every item the validation document flags gets an independent geological review.
The full prompt is in AUTHORING.md.
