
TL;DR: A domain-specific LLM benchmark is a set of questions and scoring rules built to test an AI model on one specialist field, such as law, medicine, finance, science or geology. A model that scores well on general tests can still fail these tasks. Compare how each benchmark is built, where its evidence comes from and how it is scored, and do not rank raw scores from unrelated tests against each other.
A model can perform well on general knowledge tests and still struggle with legal reasoning, medical question answering, financial documents or geological records. Specialist evaluation needs representative tasks and a scoring method that reflects the work being tested. This is where dynamic benchmarking in specialist fields becomes useful. This guide compares domain-specific LLM benchmarks on GitHub and explains what each measures, where its limits lie and how to select one for a real evaluation project.
Which GitHub Repositories Benchmark LLMs on Specialist Domains?
1. Groundtruth Dynamic Benchmarking (Eigenform): Geology and Source-Grounded Reasoning
Description
Groundtruth Dynamic Benchmarking is an open-source benchmark for evaluating LLMs on geological reasoning and source-grounded knowledge. It tests whether a model can answer questions using real geological records rather than relying only on general textbook knowledge.
Why it works
- Generates benchmark questions from geological source corpora.
- Uses evidence-anchored rubrics to define what a correct answer must contain.
- Grades candidate answers with an independent LLM judge.
- Supports pointwise scoring from 0 to 10.
- Supports pairwise comparison with answer order swapped to expose position bias.
- Separates model evaluation from harness evaluation.
The benchmark includes geological material such as exploration reports, government records and technical filings. Its rubric design uses pass/fail gates, partial-credit components and evidence locators.
Is it up to date?
The public repository and benchmark page show an evolving benchmark with multiple rubric configurations and a live leaderboard. Check the current rubric and submission documentation before running an evaluation.
Best for
- Geological reasoning.
- Evaluating AI on mineral exploration and technical records.
- Comparing models on source-grounded questions.
- Comparing retrieval or agent harnesses while keeping the model fixed.
Additional notes
Geology is the first edition. The evaluation harness is designed to run schema-based rubrics beyond geology, while the current rubric-authoring workflow is geoscience-focused. Leaderboard submissions are self-reported and are not independently rerun, so this should be considered when interpreting results.
2. LegalBench: Legal Reasoning Tasks
Description
LegalBench is an open research benchmark for evaluating legal reasoning in English-language LLMs.
Why it works
- Contains 162 legal reasoning tasks.
- Includes tasks contributed by 40 contributors.
- Covers different types of legal text and reasoning.
- Uses task-specific input-output examples.
- Includes classification, extraction, generation and question-answering tasks.
Examples include identifying whether evidence qualifies as hearsay, extracting legal definitions and answering questions about legal rules.
Is it up to date?
The repository remains publicly accessible. Its README describes the project as an ongoing open-science effort. Check the current task definitions and dataset versions before comparing results.
Best for
- Evaluating legal language understanding.
- Testing specific legal reasoning tasks.
- Comparing model behaviour across different legal question formats.
- Building experiments around legal task datasets.
Additional notes
LegalBench is a collection of tasks, not a single simulation of a lawyer’s full workflow. A high score on a particular task does not establish that a model can manage an entire legal matter, conduct current legal research or produce a complete legal work product.
3. MedQA: Medical Knowledge and Question Answering
Description
MedQA provides code and data for evaluating medical question answering. Its dataset is based on medical examination questions, including US, Mainland China and Taiwan question sources.
Why it works
- Provides medical question-answering datasets.
- Includes question-bank files and train, development and test splits.
- Includes textbook material in English and simplified Chinese.
- Provides baseline information-retrieval code.
- Supports experiments on medical knowledge retrieval and answer selection.
Is it up to date?
The repository is publicly available, but its core paper and dataset date to 2020. Treat it as an established medical QA benchmark rather than a newly refreshed clinical evaluation suite.
Best for
- Medical knowledge question answering.
- Exam-style multiple-choice evaluation.
- Research on retrieval and medical textbook grounding.
- Comparing models on established medical QA datasets.
Additional notes
Exam-style questions are not the same as real clinical workflows. MedQA does not, by itself, evaluate the full process of reviewing a patient record, managing uncertainty, communicating with a patient or making a clinical decision.
4. FinanceBench: Financial Document Question Answering
Description
FinanceBench evaluates LLM performance on open-book financial question answering using company filings and related financial documents.
Why it works
- Focuses on questions about publicly traded companies.
- Uses financial documents as source material.
- Provides annotated questions, answers and evidence.
- Includes financial document metadata and source references.
- Supports analysis of whether a model can retrieve and use relevant financial information.
The full benchmark described in the repository contains 10,231 questions. The open-source sample includes 150 annotated examples.
Is it up to date?
The repository remains available, but its benchmark paper and original evaluation are from 2023. Treat it as a useful financial document QA reference, not a continuously refreshed measure of current model performance.
Best for
- Financial filing question answering.
- Testing evidence retrieval from 10-K and 10-Q documents.
- Evaluating numerical and document-grounded responses.
- Studying hallucinations in financial QA.
Additional notes
The public sample is smaller than the full benchmark. Check dataset access and licensing conditions before using the data in a commercial evaluation.
5. ScienceQA: Scientific Question Answering
Description
ScienceQA is a benchmark for multimodal scientific question answering. It combines questions with supporting context and, in many cases, visual information.
Why it works
- Covers science questions across multiple topics.
- Includes text and image-based examples.
- Supports evaluation of reasoning with contextual and visual inputs.
- Provides a dataset for studying multimodal question answering.
Is it up to date?
The project is an established research benchmark. Check its current repository, dataset version and evaluation scripts before using it in a new comparison.
Best for
- Scientific question answering.
- Multimodal reasoning experiments.
- Evaluating models on questions with diagrams or images.
- Research into science education and visual reasoning.
Additional notes
ScienceQA is broad rather than focused on one specialist scientific profession. It should not be treated as a direct substitute for a benchmark built around research papers, laboratory workflows or geological source records.
Domain-Specific LLM Benchmarks Compared
| Domain | Repository | What it tests | Dataset or source material | Evaluation approach |
|---|---|---|---|---|
| Geology | Groundtruth | Source-grounded geological reasoning | Geological reports, archives and technical records | Independent judge with evidence-anchored rubrics; pointwise and pairwise |
| Law | LegalBench | Legal reasoning tasks | Legal text and task-specific datasets | Task-specific input-output evaluation |
| Medicine | MedQA | Medical knowledge and exam-style QA | Medical examination question banks and textbooks | Multiple-choice answer evaluation |
| Finance | FinanceBench | Financial document question answering | Company filings and annotated evidence | Human-annotated answers and evidence-based review |
| Science | ScienceQA | Scientific question answering | Science questions, context and images | Answer evaluation across text and multimodal tasks |
These benchmarks should not be ranked against one another using their raw scores. They test different domains, datasets, question formats and grading methods. A score of 80% on one benchmark does not mean a model is more capable than a model scoring 70% on another.
Why General LLM Benchmarks Miss Specialist Skills
Broad Knowledge vs Applied Domain Reasoning
General benchmarks often test broad knowledge, language understanding or reasoning across a wide range of subjects.
Specialist work requires more specific abilities.
A legal model may need to distinguish a rule from an exception. A medical model may need to interpret a question in the context of clinical terminology. A financial model may need to identify the right figure in a filing and explain how it was calculated.
A geology model may need to connect information across exploration reports, drill logs, geological units and technical records.
Knowing a fact is different from applying it to a source document. Domain-specific benchmarks help measure that gap.
Evidence, Citations and Source-Grounded Answers
Many specialist tasks require more than a plausible answer.
The answer may need to point to:
- A specific paragraph in a legal document.
- A relevant medical passage or study.
- A figure in a financial filing.
- A geological record or technical report.
A model can produce a fluent response while citing the wrong source or inventing supporting details.
Benchmarks should therefore assess evidence quality when source-grounded answers are part of the intended task.
Groundtruth uses evidence locators in its geological rubrics. FinanceBench also provides annotated evidence associated with financial questions. These approaches make it possible to inspect why an answer received credit.
Task Coverage and Domain-Specific Error Types
Each domain has different failure patterns.
- Law: Incorrectly applying a rule, confusing legal concepts or missing an exception.
- Medicine: Selecting an incorrect answer, misreading clinical terminology or overlooking relevant context.
- Finance: Using the wrong reporting period, misreading a figure or giving an unsupported calculation.
- Science: Misinterpreting a diagram, confusing concepts or drawing a conclusion unsupported by the supplied context.
- Geology: Misidentifying a geological unit, confusing evidence across records or making an unsupported interpretation.
A general benchmark may not contain enough examples to expose these errors. A domain benchmark can define tasks that make them measurable.
How Domain-Specific LLM Benchmarks Are Built
Choosing a Representative Source Corpus
The source corpus determines what the benchmark can measure.
A useful corpus should have:
- Clear provenance.
- Appropriate usage and redistribution rights.
- Coverage of the intended domain.
- Enough detail to support meaningful questions.
- Stable identifiers or evidence locations.
For example, a financial benchmark may use company filings. A legal benchmark may use statutes and judicial opinions. A geological benchmark may use government archives and exploration reports.
The source material should reflect the intended evaluation task. A benchmark built from introductory textbooks will not necessarily measure the ability to reason over complex professional records.
Creating Questions and Scoring Rubrics
Questions should test a defined capability. They should also be answerable from the supplied source material.
A scoring rubric should specify:
- The required elements of a correct answer.
- Which errors invalidate an answer.
- Which components receive partial credit.
- What evidence supports each criterion.
- How ambiguous or incomplete responses are handled.
For more detail on building reliable ground-truth data for AI, the key principle is traceability. A reviewer should be able to follow a scoring decision back to the source evidence.
Evaluating Models and Reviewing Judge Reliability
The evaluation method should match the benchmark design.
Possible approaches include:
- Exact-match scoring for fixed-answer tasks.
- Multiple-choice accuracy for exam-style questions.
- Human review for complex or ambiguous answers.
- Rubric-based scoring for answers with multiple required components.
- Independent model judging, with calibration and human checks, for suitable open-ended tasks.
LLM judges need validation. Their identity, prompts, scoring rules and consistency should be recorded. Position bias, preference for longer answers and unsupported confidence can affect results.
A benchmark should not assume that an LLM judge is correct simply because it produces a numerical score.
Keeping Benchmarks Fresh and Resistant to Leakage
A fixed dataset can become less useful when questions and answers appear in model training data.
Benchmark maintainers should consider:
- Refreshing questions when source material changes.
- Tracking dataset versions.
- Recording model and judge versions.
- Keeping evaluation sets separate from training data where possible.
- Publishing enough methodology for results to be reproduced.
Dynamic question generation can help refresh an evaluation as new source material becomes available. It does not remove every risk of leakage or guarantee that generated questions are valid. The generation and calibration process still needs review.
How to Choose a Domain-Specific Benchmark
Match the Benchmark to the Actual Work
Use this decision guide:
- Legal reasoning: Start with LegalBench for task-level legal reasoning. Add workflow-specific tests if you need to evaluate research, drafting or multi-step legal work.
- Medical knowledge QA: MedQA is a useful starting point for exam-style questions. Select a different evaluation design for clinical workflows.
- Financial documents: FinanceBench is relevant when the task involves answering questions from company filings and supporting the answer with evidence.
- Scientific QA: ScienceQA is useful for text and image-based science questions. Choose a narrower benchmark when evaluating a particular research discipline.
- Geological reasoning: Groundtruth is designed for source-grounded geological questions and rubric-based evaluation.
Before selecting a benchmark, check five things:
- Does the dataset represent the work the model will perform?
- Can the answers be verified against reliable source material?
- Does the scoring method measure the capability you care about?
- Can the evaluation be reproduced with recorded model and judge versions?
- Is the dataset current enough for the intended use?
A benchmark is useful when it measures a real capability. A large dataset or an impressive leaderboard is not enough on its own.
Conclusion
Domain-specific LLM benchmarks help teams evaluate capabilities that broad leaderboards may not measure. LegalBench focuses on legal reasoning tasks. MedQA covers medical examination questions. FinanceBench tests financial document QA. ScienceQA evaluates scientific question answering with text and visual context. Groundtruth provides a source-grounded approach to geological reasoning.
The right benchmark depends on the task, source material and scoring method. Use general benchmarks to understand broad model capabilities, then add domain-specific evaluation when specialist knowledge, evidence or workflows matter.
FAQs
What are the best domain-specific LLM benchmarks on GitHub for evaluating models in law, medicine, finance and science, and why do general benchmarks miss specialist skills?
LegalBench, MedQA, FinanceBench and ScienceQA are useful starting points for law, medicine, finance and science. Groundtruth provides a geology-focused benchmark. General benchmarks may not test the source-grounded reasoning and specialised tasks required in these fields.
What is a domain-specific LLM benchmark?
It is an evaluation dataset or task suite designed to test model performance in a particular field. It may use specialist questions, source documents, task-specific scoring or evidence-based rubrics.
Why are general LLM benchmarks insufficient for professional domains?
They may not cover the terminology, source material, reasoning patterns or error types found in specialist work. Domain-specific tests provide more targeted evidence about performance on those tasks.
What are examples of medical and legal LLM benchmarks?
MedQA evaluates medical question answering using examination-style datasets. LegalBench contains legal reasoning tasks such as hearsay classification, definition extraction and rule-based question answering.
How do you evaluate an LLM on geological knowledge?
Use questions grounded in geological source material and score answers against evidence-anchored rubrics. The evaluation should check whether the answer is supported by the source, not simply whether it sounds geologically plausible.
What is Groundtruth Dynamic Benchmarking?
Groundtruth is an open-source geology benchmark that generates questions and source-anchored rubrics from geological records. It evaluates candidate answers using an independent judge and supports model and harness evaluation tracks.


