Domain-specific LLM benchmarks: one AI model connected to five specialist test sets for law, medicine, finance, science and geology
A model can pass general tests and still fail specialist ones

TL;DR: A domain-specific LLM benchmark is a set of questions and scoring rules built to test an AI model on one specialist field, such as law, medicine, finance, science or geology. A model that scores well on general tests can still fail these tasks. Compare how each benchmark is built, where its evidence comes from and how it is scored, and do not rank raw scores from unrelated tests against each other.

A model can perform well on general knowledge tests and still struggle with legal reasoning, medical question answering, financial documents or geological records. Specialist evaluation needs representative tasks and a scoring method that reflects the work being tested. This is where dynamic benchmarking in specialist fields becomes useful. This guide compares domain-specific LLM benchmarks on GitHub and explains what each measures, where its limits lie and how to select one for a real evaluation project.

Which GitHub Repositories Benchmark LLMs on Specialist Domains?

1. Groundtruth Dynamic Benchmarking (Eigenform): Geology and Source-Grounded Reasoning

Description

Groundtruth Dynamic Benchmarking is an open-source benchmark for evaluating LLMs on geological reasoning and source-grounded knowledge. It tests whether a model can answer questions using real geological records rather than relying only on general textbook knowledge.

Why it works

  • Generates benchmark questions from geological source corpora.
  • Uses evidence-anchored rubrics to define what a correct answer must contain.
  • Grades candidate answers with an independent LLM judge.
  • Supports pointwise scoring from 0 to 10.
  • Supports pairwise comparison with answer order swapped to expose position bias.
  • Separates model evaluation from harness evaluation.

The benchmark includes geological material such as exploration reports, government records and technical filings. Its rubric design uses pass/fail gates, partial-credit components and evidence locators.

Is it up to date?

The public repository and benchmark page show an evolving benchmark with multiple rubric configurations and a live leaderboard. Check the current rubric and submission documentation before running an evaluation.

Best for

  • Geological reasoning.
  • Evaluating AI on mineral exploration and technical records.
  • Comparing models on source-grounded questions.
  • Comparing retrieval or agent harnesses while keeping the model fixed.

Additional notes

Geology is the first edition. The evaluation harness is designed to run schema-based rubrics beyond geology, while the current rubric-authoring workflow is geoscience-focused. Leaderboard submissions are self-reported and are not independently rerun, so this should be considered when interpreting results.

Description

LegalBench is an open research benchmark for evaluating legal reasoning in English-language LLMs.

Why it works

  • Contains 162 legal reasoning tasks.
  • Includes tasks contributed by 40 contributors.
  • Covers different types of legal text and reasoning.
  • Uses task-specific input-output examples.
  • Includes classification, extraction, generation and question-answering tasks.

Examples include identifying whether evidence qualifies as hearsay, extracting legal definitions and answering questions about legal rules.

Is it up to date?

The repository remains publicly accessible. Its README describes the project as an ongoing open-science effort. Check the current task definitions and dataset versions before comparing results.

Best for

  • Evaluating legal language understanding.
  • Testing specific legal reasoning tasks.
  • Comparing model behaviour across different legal question formats.
  • Building experiments around legal task datasets.

Additional notes

LegalBench is a collection of tasks, not a single simulation of a lawyer’s full workflow. A high score on a particular task does not establish that a model can manage an entire legal matter, conduct current legal research or produce a complete legal work product.

3. MedQA: Medical Knowledge and Question Answering

Description

MedQA provides code and data for evaluating medical question answering. Its dataset is based on medical examination questions, including US, Mainland China and Taiwan question sources.

Why it works

  • Provides medical question-answering datasets.
  • Includes question-bank files and train, development and test splits.
  • Includes textbook material in English and simplified Chinese.
  • Provides baseline information-retrieval code.
  • Supports experiments on medical knowledge retrieval and answer selection.

Is it up to date?

The repository is publicly available, but its core paper and dataset date to 2020. Treat it as an established medical QA benchmark rather than a newly refreshed clinical evaluation suite.

Best for

  • Medical knowledge question answering.
  • Exam-style multiple-choice evaluation.
  • Research on retrieval and medical textbook grounding.
  • Comparing models on established medical QA datasets.

Additional notes

Exam-style questions are not the same as real clinical workflows. MedQA does not, by itself, evaluate the full process of reviewing a patient record, managing uncertainty, communicating with a patient or making a clinical decision.

4. FinanceBench: Financial Document Question Answering

Description

FinanceBench evaluates LLM performance on open-book financial question answering using company filings and related financial documents.

Why it works

  • Focuses on questions about publicly traded companies.
  • Uses financial documents as source material.
  • Provides annotated questions, answers and evidence.
  • Includes financial document metadata and source references.
  • Supports analysis of whether a model can retrieve and use relevant financial information.

The full benchmark described in the repository contains 10,231 questions. The open-source sample includes 150 annotated examples.

Is it up to date?

The repository remains available, but its benchmark paper and original evaluation are from 2023. Treat it as a useful financial document QA reference, not a continuously refreshed measure of current model performance.

Best for

  • Financial filing question answering.
  • Testing evidence retrieval from 10-K and 10-Q documents.
  • Evaluating numerical and document-grounded responses.
  • Studying hallucinations in financial QA.

Additional notes

The public sample is smaller than the full benchmark. Check dataset access and licensing conditions before using the data in a commercial evaluation.

5. ScienceQA: Scientific Question Answering

Description

ScienceQA is a benchmark for multimodal scientific question answering. It combines questions with supporting context and, in many cases, visual information.

Why it works

  • Covers science questions across multiple topics.
  • Includes text and image-based examples.
  • Supports evaluation of reasoning with contextual and visual inputs.
  • Provides a dataset for studying multimodal question answering.

Is it up to date?

The project is an established research benchmark. Check its current repository, dataset version and evaluation scripts before using it in a new comparison.

Best for

  • Scientific question answering.
  • Multimodal reasoning experiments.
  • Evaluating models on questions with diagrams or images.
  • Research into science education and visual reasoning.

Additional notes

ScienceQA is broad rather than focused on one specialist scientific profession. It should not be treated as a direct substitute for a benchmark built around research papers, laboratory workflows or geological source records.

Domain-Specific LLM Benchmarks Compared

DomainRepositoryWhat it testsDataset or source materialEvaluation approach
GeologyGroundtruthSource-grounded geological reasoningGeological reports, archives and technical recordsIndependent judge with evidence-anchored rubrics; pointwise and pairwise
LawLegalBenchLegal reasoning tasksLegal text and task-specific datasetsTask-specific input-output evaluation
MedicineMedQAMedical knowledge and exam-style QAMedical examination question banks and textbooksMultiple-choice answer evaluation
FinanceFinanceBenchFinancial document question answeringCompany filings and annotated evidenceHuman-annotated answers and evidence-based review
ScienceScienceQAScientific question answeringScience questions, context and imagesAnswer evaluation across text and multimodal tasks

These benchmarks should not be ranked against one another using their raw scores. They test different domains, datasets, question formats and grading methods. A score of 80% on one benchmark does not mean a model is more capable than a model scoring 70% on another.

Why General LLM Benchmarks Miss Specialist Skills

Broad Knowledge vs Applied Domain Reasoning

General benchmarks often test broad knowledge, language understanding or reasoning across a wide range of subjects.

Specialist work requires more specific abilities.

A legal model may need to distinguish a rule from an exception. A medical model may need to interpret a question in the context of clinical terminology. A financial model may need to identify the right figure in a filing and explain how it was calculated.

A geology model may need to connect information across exploration reports, drill logs, geological units and technical records.

Knowing a fact is different from applying it to a source document. Domain-specific benchmarks help measure that gap.

Evidence, Citations and Source-Grounded Answers

Many specialist tasks require more than a plausible answer.

The answer may need to point to:

  • A specific paragraph in a legal document.
  • A relevant medical passage or study.
  • A figure in a financial filing.
  • A geological record or technical report.

A model can produce a fluent response while citing the wrong source or inventing supporting details.

Benchmarks should therefore assess evidence quality when source-grounded answers are part of the intended task.

Groundtruth uses evidence locators in its geological rubrics. FinanceBench also provides annotated evidence associated with financial questions. These approaches make it possible to inspect why an answer received credit.

Task Coverage and Domain-Specific Error Types

Each domain has different failure patterns.

  • Law: Incorrectly applying a rule, confusing legal concepts or missing an exception.
  • Medicine: Selecting an incorrect answer, misreading clinical terminology or overlooking relevant context.
  • Finance: Using the wrong reporting period, misreading a figure or giving an unsupported calculation.
  • Science: Misinterpreting a diagram, confusing concepts or drawing a conclusion unsupported by the supplied context.
  • Geology: Misidentifying a geological unit, confusing evidence across records or making an unsupported interpretation.

A general benchmark may not contain enough examples to expose these errors. A domain benchmark can define tasks that make them measurable.

How Domain-Specific LLM Benchmarks Are Built

Choosing a Representative Source Corpus

The source corpus determines what the benchmark can measure.

A useful corpus should have:

  • Clear provenance.
  • Appropriate usage and redistribution rights.
  • Coverage of the intended domain.
  • Enough detail to support meaningful questions.
  • Stable identifiers or evidence locations.

For example, a financial benchmark may use company filings. A legal benchmark may use statutes and judicial opinions. A geological benchmark may use government archives and exploration reports.

The source material should reflect the intended evaluation task. A benchmark built from introductory textbooks will not necessarily measure the ability to reason over complex professional records.

Creating Questions and Scoring Rubrics

Questions should test a defined capability. They should also be answerable from the supplied source material.

A scoring rubric should specify:

  • The required elements of a correct answer.
  • Which errors invalidate an answer.
  • Which components receive partial credit.
  • What evidence supports each criterion.
  • How ambiguous or incomplete responses are handled.

For more detail on building reliable ground-truth data for AI, the key principle is traceability. A reviewer should be able to follow a scoring decision back to the source evidence.

Evaluating Models and Reviewing Judge Reliability

The evaluation method should match the benchmark design.

Possible approaches include:

  • Exact-match scoring for fixed-answer tasks.
  • Multiple-choice accuracy for exam-style questions.
  • Human review for complex or ambiguous answers.
  • Rubric-based scoring for answers with multiple required components.
  • Independent model judging, with calibration and human checks, for suitable open-ended tasks.

LLM judges need validation. Their identity, prompts, scoring rules and consistency should be recorded. Position bias, preference for longer answers and unsupported confidence can affect results.

A benchmark should not assume that an LLM judge is correct simply because it produces a numerical score.

Keeping Benchmarks Fresh and Resistant to Leakage

A fixed dataset can become less useful when questions and answers appear in model training data.

Benchmark maintainers should consider:

  • Refreshing questions when source material changes.
  • Tracking dataset versions.
  • Recording model and judge versions.
  • Keeping evaluation sets separate from training data where possible.
  • Publishing enough methodology for results to be reproduced.

Dynamic question generation can help refresh an evaluation as new source material becomes available. It does not remove every risk of leakage or guarantee that generated questions are valid. The generation and calibration process still needs review.

How to Choose a Domain-Specific Benchmark

Match the Benchmark to the Actual Work

Use this decision guide:

  • Legal reasoning: Start with LegalBench for task-level legal reasoning. Add workflow-specific tests if you need to evaluate research, drafting or multi-step legal work.
  • Medical knowledge QA: MedQA is a useful starting point for exam-style questions. Select a different evaluation design for clinical workflows.
  • Financial documents: FinanceBench is relevant when the task involves answering questions from company filings and supporting the answer with evidence.
  • Scientific QA: ScienceQA is useful for text and image-based science questions. Choose a narrower benchmark when evaluating a particular research discipline.
  • Geological reasoning: Groundtruth is designed for source-grounded geological questions and rubric-based evaluation.

Before selecting a benchmark, check five things:

  1. Does the dataset represent the work the model will perform?
  2. Can the answers be verified against reliable source material?
  3. Does the scoring method measure the capability you care about?
  4. Can the evaluation be reproduced with recorded model and judge versions?
  5. Is the dataset current enough for the intended use?

A benchmark is useful when it measures a real capability. A large dataset or an impressive leaderboard is not enough on its own.

Conclusion

Domain-specific LLM benchmarks help teams evaluate capabilities that broad leaderboards may not measure. LegalBench focuses on legal reasoning tasks. MedQA covers medical examination questions. FinanceBench tests financial document QA. ScienceQA evaluates scientific question answering with text and visual context. Groundtruth provides a source-grounded approach to geological reasoning.

The right benchmark depends on the task, source material and scoring method. Use general benchmarks to understand broad model capabilities, then add domain-specific evaluation when specialist knowledge, evidence or workflows matter.

FAQs

What are the best domain-specific LLM benchmarks on GitHub for evaluating models in law, medicine, finance and science, and why do general benchmarks miss specialist skills?

LegalBench, MedQA, FinanceBench and ScienceQA are useful starting points for law, medicine, finance and science. Groundtruth provides a geology-focused benchmark. General benchmarks may not test the source-grounded reasoning and specialised tasks required in these fields.

What is a domain-specific LLM benchmark?

It is an evaluation dataset or task suite designed to test model performance in a particular field. It may use specialist questions, source documents, task-specific scoring or evidence-based rubrics.

Why are general LLM benchmarks insufficient for professional domains?

They may not cover the terminology, source material, reasoning patterns or error types found in specialist work. Domain-specific tests provide more targeted evidence about performance on those tasks.

MedQA evaluates medical question answering using examination-style datasets. LegalBench contains legal reasoning tasks such as hearsay classification, definition extraction and rule-based question answering.

How do you evaluate an LLM on geological knowledge?

Use questions grounded in geological source material and score answers against evidence-anchored rubrics. The evaluation should check whether the answer is supported by the source, not simply whether it sounds geologically plausible.

What is Groundtruth Dynamic Benchmarking?

Groundtruth is an open-source geology benchmark that generates questions and source-anchored rubrics from geological records. It evaluates candidate answers using an independent judge and supports model and harness evaluation tracks.