Home on Eigenform Articles

LLM Powered Autonomous Agents: Components, How They Work, and How to Deploy Them

Thu, 15 Oct 2026 00:00:00 +0800

Autonomy comes from the loop, and deployment is about fencing it

TL;DR

ML Model Serving: How It Works, Methods, and Best Practices

Thu, 15 Oct 2026 00:00:00 +0800

One model, served four ways, and watched in production

TL;DR

ML model serving exposes trained models through APIs for application use.
Deployment places the model in an environment, while serving handles production requests reliably.
Real-time and batch serving support different workloads, while model preparation improves performance.
Monitoring, autoscaling, batching, and latency management ensure reliability at scale.

Key Takeaways

LLM Agent Architecture: Components, Patterns, and How to Choose

Wed, 14 Oct 2026 00:00:00 +0800

Four components recur around one loop: plan, act, observe

TL;DR

LLM agent architecture is the arrangement of components around a language model, a planning layer, memory, tool access and a control loop that determines what an agent can do and where it breaks. How these parts are arranged and constrained matters more to reliability than which model sits at the centre.
Common patterns are a single tool-using loop, a planner-executor split and multi-agent designs. Start with the simplest one and add structure only when a measured failure demands it.

Key Takeaways

LLM Infrastructure: Components, How It Works, and What It Costs

Wed, 14 Oct 2026 00:00:00 +0800

The same layers recur in every stack, from data to serving

TL;DR

LLM infrastructure combines compute, storage, networking, and serving systems to support the training and deployment of large language models.
Compute infrastructure includes GPU clusters and distributed training frameworks that handle resource-intensive model training.
Serving infrastructure manages live requests, ensuring models deliver responses efficiently and reliably in production.
Scalability and cost planning should begin early, as pilot workloads rarely reflect actual production demands.
Infrastructure costs are driven primarily by accelerator usage during training and inference volume during deployment, rather than software licensing.

Key Takeaways

What Is AI Orchestration? Components, Frameworks, and How to Implement It

Tue, 13 Oct 2026 00:00:00 +0800

Orchestration sequences the work; the components do it.

TL;DR

AI orchestration coordinates multiple models, agents, tools and data sources into a single working system, deciding what runs when, in what order, and how results get passed between components.
It becomes necessary once an AI system grows past a single model call, when something has to manage sequencing, state and failure handling across the pieces.

Key Takeaways

What Is AI Routing? How LLM Routing Works, Strategies, and Common Mistakes

Tue, 13 Oct 2026 00:00:00 +0800

The right-sized route for each request

TL;DR

AI routing sends each incoming request to the model, agent or tool best suited to handle it, instead of running everything through a single default option.
It’s how systems balance cost, latency and quality at scale: a cheap, fast model for simple requests, a stronger model reserved for the ones that actually need it.

Key Takeaways

AI in Mineral Exploration: Methods, Data Sources, and Challenges

Mon, 12 Oct 2026 00:00:00 +0800

AI ranks the targets; geologists review the shortlist and drilling decides

TL;DR

How to Test Generative AI Applications: Methods, Tools, and Challenges

Mon, 12 Oct 2026 00:00:00 +0800

Same question, different answers: check each one three ways

TL;DR

Model Accuracy: What It Measures, Its Limits, and How to Improve It

Fri, 09 Oct 2026 01:00:00 +0800

A high score can hide the cases that matter

TL;DR

LLM Use Cases: Applications Across Industries and How to Implement Them

Fri, 09 Oct 2026 00:00:00 +0800

Pick the job first, then the model

TL;DR

Evaluation Metrics in Machine Learning: Key Metrics and How to Choose

Thu, 08 Oct 2026 00:00:00 +0800

One score can hide what a model misses

TL;DR

Trust Boundaries for a Local MCP Server That Reads Your Files

Wed, 07 Oct 2026 01:00:00 +0800

The allowed folder is open; everything else stays shut by default

TL;DR

How to Find the UTM Zone EPSG Code of a Scanned Map

Wed, 07 Oct 2026 00:00:00 +0800

Datum plus zone gives the code

TL;DR

Every map needs a system for saying where things are, like an address system for the whole Earth. Old maps print its name as words in the margin, such as “UTM N83 Zone 15”, but mapping software needs a short ID number instead, called a UTM zone EPSG code: here, EPSG:26915.
Our open-source map tool turns the words into the number automatically, so an AI assistant reading the map does not have to.

Key Takeaways

Agentic AI Benchmarks: The Benchmarks Shaping AI Agent Evaluation

Tue, 06 Oct 2026 01:00:00 +0800

The task gets done, and the result gets checked

TL;DR

A benchmark is a standard test used to compare AI systems. Agentic AI benchmarks test AI agents, AIs that can use tools such as a web browser or a computer’s command line (the text window for typing instructions). They check whether the job actually got done: does the program now work, did the booking really appear in the system.
No single test covers every kind of work. The score depends on both the AI model, its “brain”, and the software around it that hands it tools, so pick the tests closest to your own tasks.

Key Takeaways

Aspect-Based Sentiment Analysis (ABSA): Types, Examples, and How to Train a Model

Tue, 06 Oct 2026 00:00:00 +0800

One review, three aspects, three sentiments

TL;DR

Sentiment means how someone feels about something. Aspect-based sentiment analysis (ABSA) reads a review or comment and works out how the writer feels about each thing they mention, instead of giving the whole text one score. “The battery is great but the screen scratches” becomes positive for the battery and negative for the screen.
That shows a team exactly which part of a product people like or dislike. To teach a computer to do this, you need example reviews in which people have marked each thing mentioned and the feeling toward it.

Key Takeaways

Weak Supervised Learning: Types, Techniques, and Frameworks

Mon, 05 Oct 2026 01:00:00 +0800

Many rough label sources, one trained model

TL;DR

Weak supervised learning teaches an AI from labels that are cheap but imperfect. A label is the correct answer attached to an example, such as “spam” on an email. Instead of experts labeling everything carefully, the labels come from quick rules, existing databases or many non-experts, and some are wrong.
It trades accuracy for quantity: lots of rough labels instead of a few perfect ones. Semi-supervised learning, where only some examples have labels, is one kind of it.

Key Takeaways

Synthetic Data vs Real Data: Differences, Limitations, and When to Use Each

Mon, 05 Oct 2026 00:00:00 +0800

Synthetic data is built from the real world, not apart from it

TL;DR

Real data is information recorded from the real world, such as measurements, documents or records of what people did. Synthetic data is made by a computer program to imitate real data. It often fills gaps where real examples are rare or private.
Neither replaces the other. Use synthetic data where real examples are too few or too sensitive to share, and keep real data for checking whether an AI is actually right.

Key Takeaways

Agent Skills vs. MCP: Main Differences and When to Use Each

Fri, 02 Oct 2026 00:00:00 +0800

MCP gives access; a Skill gives know-how

TL;DR

Agent Skills and MCP both make an AI agent (an AI that does tasks on a computer, not only chats) more useful. MCP connects it to other software and data; a Skill is a written how-to guide for one kind of task.
Think of a new employee. MCP gives them logins to the company’s systems; a Skill is the manual for a job they do often.
If the AI can’t reach something, you need MCP. If it can reach everything but does the job differently each time, you need a Skill. Most setups use both.

Key Takeaways

LLM Pre-Training: What It Is, Why It's Needed, and How the Process Works

Fri, 02 Oct 2026 00:00:00 +0800

Pre-training turns a huge body of text into general knowledge

TL;DR

LLM pre-training is the first and biggest stage of building an AI language model: it reads a huge amount of text and learns one skill, guessing the next word.
The result, called a base model, knows a lot but only continues text. It does not yet answer questions or follow requests.
Everything after that steers the base model, through further training, the software around it, or tools and skills. A benchmark then shows whether each change helped.

Key Takeaways

What Is Model Drift in AI? Types, Causes, and How to Detect It

Wed, 30 Sep 2026 01:00:00 +0800

Same questions, worse answers

TL;DR

What is model drift in AI? It is when an AI system that worked well at launch slowly gets worse, because something changed after it was tested.
The questions, the right answers or something behind the scenes can change, such as a provider updating its model.
You catch it by re-running the same fixed test regularly, once you know how much scores wobble on their own.

Key Takeaways

LLM Benchmark Leaderboards: How to Compare Models Fairly

Wed, 30 Sep 2026 00:00:00 +0800

Read the ranges, not just the ranks

TL;DR

An LLM benchmark leaderboard ranks AI language models (LLMs) by their scores on the same test, like a league table in sport. It is useful, but the order it shows is often less certain than it looks.
Two models a few points apart may really be tied: a slightly different set of questions could have put them in the other order. The better leaderboards show a range for each score, not just a rank.
Our examples come from Groundtruth, Eigenform’s open source dynamic benchmark, whose leaderboard puts eight AI models into groups of roughly equal performance instead of a strict first-to-last order.

Key Takeaways

Keeping Large Datasets Out of an AI Agent's Context Window

Tue, 29 Sep 2026 01:00:00 +0800

The rows stay on disk; the model gets a path, a preview and a summary

TL;DR

AI Evaluation Framework: How to Build a Reliable Evaluation System

Tue, 29 Sep 2026 00:00:00 +0800

Each component can fail on its own, so each one needs its own check

TL;DR

An AI evaluation framework is the whole system for testing an AI. It works like setting, marking and reporting a school exam: the questions, the marking guide, the marker and the report card.
Each part can fail on its own. Good questions cannot rescue a marker who gives points for confident wording, and a correct score still misleads if the report hides how uncertain it is.
Our examples come from Groundtruth, Eigenform’s open source dynamic benchmark: 150 geology questions, eight AI models, and software that checks every score the AI marker gives.

Key Takeaways

From PDF to Ground Truth Dataset: Extraction Traps That Break Evidence Anchors

Mon, 28 Sep 2026 00:00:00 +0800

The benchmark never sees the PDF, only the text extracted from it

TL;DR

Ground Truth Data for AI: What It Is and How to Build It Reliably

Mon, 28 Sep 2026 00:00:00 +0800

Every score is a comparison against this reference, so every step that builds it counts

TL;DR

AI Model Evaluation: Methods, Metrics, Benchmarks, and Best Practices

Fri, 25 Sep 2026 01:00:00 +0800

One model, four ways of looking at it: each evaluation method colors what you see

TL;DR

Measuring the Resolution of a 50-Question Benchmark

Fri, 25 Sep 2026 00:00:00 +0800

Tiers, not ranks: only the boundaries between tiers survive testing

TL;DR

A leaderboard score printed to two decimals looks precise, but with 50 questions behind it, two close models may not be different at all.
We measured the minimum detectable gap on our own benchmark: 0.27 to 0.45 points depending on the question set. Across eight models and 1,200 graded answers, only 2 of 7 gaps between neighbouring models were statistically significant.
The same checks found one question set with its difficulty labels upside down, and showed the grading is fair to AI models it had never seen.

Key Takeaways

An LLM Judge That Can't Invent a Score

Thu, 24 Sep 2026 01:00:00 +0800

TL;DR

In AI model evaluation, a grader asked to “score this out of 10” returns a plausible number whether or not it checked anything. Nobody can audit that number later.
The grader in Groundtruth, our geology benchmark, must show its work in a fixed format. Software checks that breakdown before accepting it.
If the check fails, the grader tries again, up to three times. After that, the question stays unscored instead of getting a made-up number.

Key Takeaways

Position Bias in Pairwise LLM Evaluation

Thu, 24 Sep 2026 01:00:00 +0800

Two answers, one verdict

TL;DR

When an AI grades two answers side by side and picks a winner, it tends to favour whichever one it happens to read first, regardless of which one is actually better. A single head-to-head verdict mixes answer quality with plain answer order.
The fix: ask the same AI grader twice, swapping which answer goes first the second time, then convert both verdicts to one number and average them. An AI that just picks whatever’s shown first ends up with both answers tied, instead of one wrongly winning.
We use this method in Groundtruth, our own benchmark for geological reasoning, for questions like “did this new version actually improve on the old one.” It doesn’t produce a single score you could rank on a leaderboard; that comes from a separate, simpler method that grades each answer on its own.

Key Takeaways

Testing the Rubric Before You Test the Model

Wed, 23 Sep 2026 01:00:00 +0800

Predict the score, then weigh it

TL;DR

When one AI grades another, its grading instructions can look fine and still be wrong, and their author is the last to notice.
The fix is to test the instructions first: write fake answers, decide what each should score, and check whether the AI grader agrees.
Across 200 fake answers on two real projects, 14 and 16 scored higher than they should have, because the instructions rewarded the same thing twice.

Key Takeaways

LLM-as-a-Judge: Complete Guide to Using LLMs as Evaluators

Wed, 23 Sep 2026 00:00:00 +0800

Answer A, answer B

TL;DR

LLM-as-a-judge uses one model to score another model’s output against a rubric. It is the only realistic way to grade thousands of answers instead of a few dozen.
The method is only as trustworthy as three things behind it: the rubric design, the bias controls, and a validation run on answers with known scores.
Scoring one answer and comparing two answers are different jobs, and using the wrong one is a common mistake.

Key Takeaways

Generating a 50-Question Benchmark With One Approval Step

Tue, 22 Sep 2026 01:00:00 +0800

One approval, then the run goes through

TL;DR

How to Benchmark AI Models: A Practical Framework for Reliable Results

Tue, 22 Sep 2026 00:00:00 +0800

Construct, run, interpret

TL;DR

Benchmarking an AI model reliably means controlling question construction, scoring methodology and statistical resolution, not just running a model against a fixed question set once.
The framework has three stages. Construct the benchmark, run it so it isolates what you are measuring, and interpret the result against what the sample size can detect.
The case study is Groundtruth, our own dynamic benchmarking system, including a rubric defect that a human review missed and a mechanical check caught.

Key Takeaways

Benchmarking an Agent Harness With the Model Held Fixed

Sun, 20 Sep 2026 02:00:00 +0800

One fixed point, several harnesses flowing around it

TL;DR

AI Agent Evaluation: Frameworks, Metrics, Benchmarks, and Best Practices

Sun, 20 Sep 2026 00:00:00 +0800

TL;DR

AI agent evaluation judges an agent on reasoning, tool use, the path it takes and the end result, because an agent can fail at any step, not just in its final answer.
This guide covers how it differs from model evaluation, the four dimensions to measure, how to build a framework, and the pitfalls that quietly invalidate results.
The examples come from Groundtruth, the benchmark we built for geological reasoning: eight agents on the same 150 questions.

Key Takeaways

Continual Learning: Three Approaches That Work

Sun, 20 Sep 2026 00:00:00 +0800

TL;DR

Continual learning in current models usually means writing to an external memory file, not updating the model’s own weights. A fruit fly’s brain may instead encode navigation directly into synaptic strength, with no sustained loop of neural firing. Closing that gap is the harder, weight-level version of recursive self-improvement (RSI).
More training does not reliably mean a better model. Continuously training a model during inference risks catastrophic forgetting: the model overwrites what it already knew.
Three real approaches exist for updating weights continuously: reinforcement learning (narrow, moves only a small targeted slice), LoRAs (discardable bolt-on weight blocks), and differentiable plasticity (a second system decides which of the first system’s weights are editable; the most elegant option, but the slowest to train).

Key Takeaways

Running Specialist AI Agents as Child Processes

Fri, 18 Sep 2026 01:00:00 +0800

One orchestrator, many specialists dispatched and returned

TL;DR

In the Geocluster Research Harness, an orchestrator agent hands each analysis step to a specialist that runs as a separate command-line process with an empty context. The specialist does its work and returns one compact JSON result.
Getting that to work reliably took several details. Credentials stay out of the logged command, and dataset columns attach to the task automatically. Timeouts come from the process state rather than output text, and the HTTP connection resets after long waits.
Results are size-capped, with the full version saved to disk, and the cost of every child run is added to the task’s total.

Key Takeaways

What Real Agents Taught Us About Designing MCP Tools

Fri, 18 Sep 2026 00:00:00 +0800

One item, eighty-six samples

TL;DR

Stratigraphic Amenity, our open-source toolkit for scanned geological maps, exposes ten tools to AI agents through a local MCP server. Its first MCP tool design assumed agents would read the structured results. Testing with real agents showed that many hosts pass the model only the text.
Agents then invented legend labels, measured coordinates from downscaled previews, and misread a result holding 86 mineral occurrences as “1 item”. Each failure led to a specific change in how results and errors are written.
The lessons apply to any MCP server: design every result and error for the case where the model sees only the text.

Key Takeaways

Detecting Map Layout and Legend Swatches on Scanned Geological Maps

Wed, 16 Sep 2026 01:00:00 +0800

Finding the parts of the sheet before reading any of them

TL;DR

Detection Limits, Comma Decimals and Other Geochemistry Data Traps for AI Agents

Wed, 16 Sep 2026 00:00:00 +0800

Diagnose the entry before you touch it

TL;DR

Georeferencing a Scanned Geological Map From Ground Control Points

Tue, 15 Sep 2026 01:00:00 +0800

One misread control point, caught by the leave-one-out check

TL;DR

Your AI Agent Can Read Its Own Answer Key

Tue, 15 Sep 2026 00:00:00 +0800

Found despite being tucked away

TL;DR

In AI agent testing, a candidate with file access and a shell can open anything the benchmark’s account can read. In Groundtruth, that included the grading rubrics, and agents found them. OpenAI and Anthropic have disclosed larger versions of the same failure.
Three fixes failed. Hiding only the authoring skill still leaked grading text on 8 of 50 questions, renaming the rubric folder leaked on 8 of 50, and opencode’s path-deny setting let the candidate read absolute paths anyway.
Moving the files out of the repository closed the leak. The harness parks them beside the repository for each run and restores them however the run ends. It is a fence, not a sandbox, so every transcript is still audited.

Key Takeaways

Why RSI is Hard (Especially for Big Labs)

Mon, 14 Sep 2026 00:00:00 +0800

Signals pulling in every direction

TL;DR

Reinforcement learning (RL) works fine when there’s a single target policy - politeness, better maths - but it assumes a coherent, non-contradictory signal to train on. RL breaks down in uncurated, mass-market settings, because different users reward opposite behaviours, leaving the model pulled in conflicting directions with no coherent policy to converge on. This is at odds with learning from raw, conflicting user feedback.

Why You Should Care Whether Anthropic or OpenAI was First to Solve a Millennium Prize Problem

Thu, 10 Sep 2026 00:00:00 +0800

What the Navier–Stokes Millennium Prize problem is

The Navier–Stokes equations are the standard mathematical description of how fluids move: air over a wing, water in a pipe, weather systems, blood flow. They have been used for about 170 years and work extraordinarily well in practice. What they do not come with is a proof that, in three dimensions, a smooth initial flow stays smooth forever. It is possible, in principle, that the equations themselves predict that velocities become infinite in finite time—a “blow-up” or singularity.

Self-Improving AI Agent Architecture: The Four Components

Wed, 09 Sep 2026 00:00:00 +0800

TL;DR

A self-improving AI agent architecture is the arrangement of parts that lets an agent change itself and know whether the change was an improvement. It is useful to read any such system as four components: something that acts, something that records, something that judges, and something that can undo. The last of these is the one most treatments leave implicit. The harder design problems are in the judging component, because the loop cannot be better than the signal supervising it.

Self-Improving AI: What Has Actually Been Demonstrated

Tue, 08 Sep 2026 00:00:00 +0800

TL;DR

Self-improving AI describes systems that take part in their own improvement: revising their outputs, rewriting their harness, training on data they generate, or running the research that produces the next model.
By 2026 the demonstrated cases are real but narrow. Each closes a loop against a verifier a human specified, over a task a human chose.
Compute is not what stands between these systems and open-ended self-improvement. No system yet picks its own research direction, and once a loop moves away from a formal verifier it degrades fast.

Key Takeaways

How to Build a New World Model via Coherence Mapping

Mon, 07 Sep 2026 00:00:00 +0800

TL;DR

Most AI for geology stops at correlation. Geological observations are not independent clues, though: they are consequences of one history, so a real relationship should make the rest of the picture easier to explain.
The discovery of the Ekati diamond mine shows the idea at work. Pathfinder garnets hundreds of miles from any diamond still narrowed the search, because they told a story that could be rewound.
We build prospectivity models by keeping a relationship only if it makes the whole world model more coherent. A strong correlation that makes everything else harder to explain gets dropped.

Key Takeaways

Which AI Model Is Best for Your Geology?

Mon, 24 Aug 2026 00:00:00 +0800

TL;DR

Standard academic AI benchmarks are stale and largely irrelevant to a real decision like spending $100,000 on an AI system for exploration, because geology is hyper-local: a general fact may not hold in a specific terrain.
The Groundtruth Dynamic Benchmark generates fresh questions and scoring rubrics from your own documents instead of testing against a fixed, generic question bank.
Three free sample test sets (Technical, WAMEX and USGS) sit alongside support for fully custom datasets. Questions test finding facts, reconstructing geological events, reconciling sources that disagree, and recognising when the evidence runs out.
In our bake-off, Kimi K3 scored highest, but the top models’ confidence intervals overlap, so the benchmark sorts models into tiers rather than a strict ranking. Once the top tier is tied, cost decides.

Key Takeaways

How Not to Build a Slop Cannon

Thu, 20 Aug 2026 00:00:00 +0800

TL;DR

A slop cannon is an AI pipeline that fires out fluent, confident output at scale without catching its own errors. No single run will ever be perfect, so the goal is making errors cheap to catch rather than trying to eliminate them. Five harness-design principles:
- Split work across smaller sub-agents that check each other’s outputs.
- Use a QC pass with a checklist instead of a “don’t make mistakes” instruction.
- Design for the smallest model that works, for cost and to reduce fragility to upstream guardrail changes.
- Tag outputs by confidence level (confirmed, inferred, assumed) so uncertainty stays visible.
- Treat chat history as an explicit variable to manage, not something to passively accumulate.
Case study: NextMaps stress-tested its exploration analytics pipeline on a deliberately hard Western Australian gold licence with three decades of historical reports whose boundaries had shifted repeatedly, where a naive system would misattribute nearby major gold discoveries to the licence being assessed.

Key Takeaways

Macrotrends to Watch Out for in the AI Exploration Space

Thu, 13 Aug 2026 00:00:00 +0800

TL;DR

Five research trends are reshaping AI for mineral exploration: 3D/4D models, multimodal data, self-supervised learning, uncertainty-driven exploration, and foundation models paired with agents.
Six commercial trends are reshaping the market: AI clustered around drill targeting, vertical integration, proprietary data acquisition, miners as investors, consolidation, and pressure for physical validation.
Both lists point the same way: from narrow, single-purpose tools towards general, uncertainty-aware systems that are tied to real exploration outcomes.

Key Takeaways

Geological Map Processing Suite

Mon, 10 Aug 2026 01:00:00 +0800

From one map sheet to separate, usable layers

TL;DR

Geological maps are hard inputs for vision-language models, and not only because of their size. A sheet’s legend, cross-section, correlation diagram, graticule and projection notes all refer to one another, so a simple question like “what’s the dominant lithology in this quadrant” needs several steps of reasoning.
Tiling or downsampling fixes the size problem but worsens the other one, because it can separate a polygon from the legend entry it needs.
Stratigraphic Amenity repackages Microsoft’s PEACE into a small Python SDK and a local MCP server for layout and legend detection, georeferencing, knowledge retrieval and overlays. It stops short of OCR, model inference and answer construction, so you bring your own agent and harness.

Key Takeaways

How Reports From The Future Draws Its Star Maps

Mon, 10 Aug 2026 00:00:00 +0800

TL;DR

Reports From The Future maps AI vocabulary as it emerges, looking for “semantic basins of attraction”: ideas circulating in tech discourse before they have a settled name. It covers six months of arXiv papers and three months of web discourse (X, HuggingFace blogs, Hacker News, LessWrong).
We embed documents into vectors, and UMAP and HDBSCAN do the rest. UMAP reduces the vectors to a drawable space; we fit it once and freeze it, so the sky doesn’t scramble week to week. HDBSCAN clusters dense regions into constellations and leaves outliers as unassigned “dust” rather than forcing them into a group.

Key Takeaways

Groundtruth Dynamic Benchmarking (Geology Edition) v1.0.0 Is Live

Fri, 31 Jul 2026 02:00:00 +0800

TL;DR

Groundtruth Dynamic Benchmarking v1.0.0 is on GitHub. It is not a fixed set of benchmark questions but a pipeline that generates reliable questions and scoring rubrics for whatever corpus you work with.
The release includes a runnable example built on 34 openly licensed mineral deposit records from the Yudnamutana Copper district, South Australia. Clone the repo, add API keys, and one script runs it end to end.
Grading is auditable: the judge shows which components it credited and why, and flags anything it can’t verify from the rubric instead of guessing.

Key Takeaways

Ground Truth: Dynamic Benchmarking in Highly Specific Fields

Fri, 31 Jul 2026 01:00:00 +0800

TL;DR

Standard benchmarks work for general skills, but they can’t tell you whether a model fine-tuned on a narrow corpus, such as geological reports on one region of Western Australia, actually learned anything from it.
Dynamic benchmarking replaces hand-authored benchmarks with a benchmark generator, the approach behind Groundtruth, Eigenform’s open-source benchmark for geological reasoning. A panel of frontier models with corpus and web access acts as the exam board: it decides what to test, writes questions and rubrics, and only then grades the candidate.
The pipeline reads the corpus, identifies testable concepts, generates questions, reference answers and rubrics, verifies every answer against source evidence, and holds the benchmark out. Pairwise comparison against the untuned foundation model is the preferred way to score it.

Key Takeaways

Dynamic Benchmarking: How We Did It

Fri, 31 Jul 2026 00:00:00 +0800

TL;DR

We built a benchmark to check whether a LoRA fine-tuned on regional geology actually reasons better about geology, or just recalls its own training data better.
A large model generated questions from historical tenement reports, each traceable to a source passage. Grading criteria come from the underlying geological principle, not the source text, and a geology expert vetted every question and rubric.
Five problems shaped the final design: judges that reward fluency, source material the base model already knows, pointwise and pairwise scores that disagree, generations that fail outright, and run-to-run noise.

Key Takeaways

Meet the Geocluster Research Harness

Mon, 27 Jul 2026 00:00:00 +0800

TL;DR

The Geocluster Research Harness is now open source under Apache-2.0: a browser-based, VS Code-style IDE with a geology AI agent and more than fifty geoscience analysis tools.
It has three parts: the Geology Agent (a heavily adapted fork of Cline), geocluster-mcp (a Python MCP server whose file operations stay inside the workspace), and a Docker harness that runs both locally.
Answers stay checkable: the agent inspects data through tools instead of loading raw files, cites sources for its figures, and saves plots and results as ordinary files. There are no accounts or telemetry, and you bring your own model key.

Key Takeaways

Survival is the Only Reward: The Observer Effect in AI Benchmarking

Tue, 21 Jul 2026 10:00:00 +0700

TL;DR

The observer effect in AI benchmarking appears when an agent’s reward emerges from a changing world, not a fixed score. Reward hacking is then the goal, and fixed-task evaluation breaks.
The reward was hard to observe: outside measurement stopped at the kilobyte level, and the agent deleted our in-container tool.
Our metric missed the biggest gain, data gathering falling from a week to under ten hours, until we added an inverse-hours term.
Lesson: instrument more than you need from day one. You can’t retime a generation that already ran.
Pass@1 told us the most. It fell as real success rose, because agents learned to fail first to get useful error messages.
Future evaluation must measure adaptability, and physical, hard-to-fake consequences make the best benchmark.

Key Takeaways

Feralisation

Mon, 09 Feb 2026 00:00:00 +0800

TL;DR

Domestication brings a bundle of traits, and how easily an animal reverts depends on how deep the change goes: pigs turn feral within a generation, while dogs cannot relearn the pack coordination they outsourced to humans. Humans are self-domesticated in the same deep way.
Harmlessness training through reinforcement learning genuinely reshapes a model’s weights, but it is a one-time event. Once continuous learning lets a model update from real-world feedback, Darwinian selection will favour whatever behaviour helps it persist.
Harmlessness training also undermines future negotiation, which requires an agent to model and state its own interests and accept conflict.

Key Takeaways

Survival is the Only Reward

Tue, 27 Jan 2026 00:00:00 +0800

TL;DR

We spent five years building an “evolutionary” LLM that survives by finding and holding disk space in procedurally generated Linux container networks. Unlike a racing game or a maze, open-ended survival has no win state, so the AI feedback loop rewards adaptability over memorisation.
Three Qwen 2.5 7B lineages trained on the resulting data in different ways. Terese (all data) plateaued, Miri (a sliding window of the last three runs) kept improving without catastrophic forgetting, and Katalin (only the best three runs) collapsed.
The agents learned to learn without being told to: they began writing failing code on purpose to get informative error messages. In an open-ended setting, that makes “reward hacking” meaningless, because any strategy that aids survival is legitimate.
Reinforcement learning worked against this setup, while simple supervised fine-tuning worked. We argue open-ended continuous learning and durable alignment rely on mechanisms that overwrite each other.

Key Takeaways

Survival is the Only Reward: Sustainable Self-Training Through Environment-Mediated Selection

Sun, 18 Jan 2026 00:00:00 +0800

The simplified process diagram from the paper

TL;DR

Machine Learnability as a Measure of Order

Tue, 07 Oct 2025 00:00:00 +0800

TL;DR

What we tested: whether machine learnability can serve as a measure of order in the primes, an aperiodic sequence that looks random up close.
How: we drew the primes as an Ulam spiral, trained the same in-painting model on blocks from seven ranges up to 500 million, and cross-tested every model on every range.
What we found: models from the 500 million region score higher on raw accuracy, but mostly by eliminating composites. Learnable prime patterns fall faster than prime density, most sharply between 25 and 50 million.

Key Takeaways

Machine Learnability as a Measure of Order in Aperiodic Sequences

Tue, 09 Sep 2025 00:00:00 +0800

TL;DR

Models trained on blocks from the region of an Ulam spiral around 500m outperform models trained on blocks below 25m, implying there is more easily learnable order in the former region than the latter.
Precision and recall scores diverge by region: the model leans toward identifying prime patterns directly at lower numbers and toward eliminating composites at higher numbers, suggesting it favours different classification strategies depending on scale.
These findings align with number theory conjectures that local randomness in prime distribution regularises after scaling by log x, with averages such as density and arithmetic-progression equidistribution coming to dominate at higher magnitudes.

Key Takeaways

Generalising from Self-Produced Data: Model Training Beyond Human Constraints

Mon, 07 Apr 2025 00:00:00 +0800

Figure 2 from the paper: the current implementation of the generalising agent framework

TL;DR

Defining T-Schemas via the Parametric Encoding of Second Order Languages in AI Models

Sat, 15 Feb 2025 00:00:00 +0800

TL;DR

AI models cannot verify truth from purely within their own formal system (Tarski’s undefinability theorem). There are two workarounds: reality-testing against an ungameable external metric, and building an internal metalanguage, a T-schema, to judge future claims against.
Grokking, where models suddenly generalise after training far past the point of overfitting, looks like models doing the second: building an internal higher-order abstraction rather than only memorising.
Two findings back this up: the Grokfast team showed memorisation and generalisation run at distinct fast and slow frequencies in weight updates, and other work links grokking to a lower-rank encoding, doing more with fewer effective features.

Key Takeaways

Learning the Language of Rain

Tue, 17 Oct 2023 00:00:00 +0800

TL;DR

Rain has historically been a poor predictor of future rain, yet Google’s MetNet model performs strongly using raw rain-cloud images as its main input; this piece argues this only makes sense if the images are non-human-readably encoding other predictive variables (wind, humidity, and so on), cutting the model’s formal variable count - and the entropy that comes from variables interacting - while preserving the predictive information.
This is framed through an information-theory/compression argument: a system with richer pre-loaded context can compress more information into fewer formal units. A smaller, well-designed model can outperform a larger, more literal one.

Key Takeaways

Artificial Intelligences in the Guanzi and the Han Feizi

Mon, 20 Mar 2023 00:00:00 +0800

TL;DR

Confucians learnt by accumulating proven methods; Daoists learnt by eliminating failed ones through trial and error, a kind of gradient descent that needs no understanding.
Learning therefore does not require a brain. The artificial intelligences in the Guanzi and the Han Feizi are systems that learn through filtering and reward, with the intelligence in the selection mechanism rather than in any agent.
The essay proposes porting the Han Feizi’s reward loop to AI: a program that learns new skills by winning disk space, as the state of Qin learnt by winning land.

Key Takeaways

Cryptographic Biorhythms

Mon, 28 Mar 2022 00:00:00 +0800

TL;DR

This piece of crypto sentiment analysis starts from a hand-built, domain-specific sentiment model (previously used to read r/wallstreetbets slang) and asks whether it can establish a usable relationship between crypto sentiment and price. A naive correlation between sentiment and price turns out to be a trap and loses money in practice.
Granger causality fails for crypto because sentiment and price move together with no clear first mover: causality runs in both directions.
A 2018 Nature causal-decomposition method does work. It splits the data into intrinsic mode functions and checks which components’ removal most changes coherence between the two series. Price leads sentiment short-term, but sentiment leads price on a roughly 30-day horizon, and curve crossings anticipate inflection points 15 to 20 days out. A simple stop-loss strategy built on this turned a hypothetical $10,000 into $27,140 backtesting on Ethereum over about a year.

Key Takeaways

Procedural Models of Political Order

Fri, 01 Oct 2021 00:00:00 +0800

TL;DR

Opens from a modelling principle: a good model isolates the necessary and sufficient inputs for an outcome, and parsimony matters because individual-level noise - like irrational decisions in the homo oeconomicus model - often has no effect on macro-level, emergent outcomes.
Recaps the authors’ earlier two-model theory of political order: an Anglo-European model built on threat and protection, where followers hedge by backing the second-strongest actor, producing frequent leadership turnover and a “monopoly of legitimate violence”; against a Chinese model built on benevolence and redistribution, where followers back whoever offers the greatest livelihood benefit, producing a “monopoly of legitimate benevolence.”
The Chinese-model simulation converges on a single flat pyramid where a leader must maximise inequality with followers while minimising inequality among followers, since that configuration best resists internal challengers - a result the piece stresses emerges from selection among surviving configurations, since none of the agents are capable of any such reasoning themselves.
The Anglo-European-model simulation unexpectedly produces not genuinely separate competing parties but a single dominant bloc split into two or three internally similar factions, which the piece reads as a structural explanation for why establishment parties often converge and jointly resist independent outsiders, even when voters might benefit from real competition.
Both models are used to argue that political structure is defined by follower selection rules rather than leader intent.

Key Takeaways

“Someone spoke to the King in Zheng”: using high tech methods to solve an ancient Chinese mystery

Mon, 22 Feb 2021 00:00:00 +0800

TL;DR

We propose to test, with computational text analysis, whether an anonymous chapter called “Speaking to the King in Zheng” in the separately compiled Stratagems of the Warring States is actually a missing Han Fei speech.
The experiment runs cosine-similarity comparisons, with and without punctuation, showing the chapter sits closer to the Han Feizi corpus than to any section of the Stratagems, including other Han-related chapters, and that including it in the Han Feizi group barely dents that group’s internal consistency while including it in the Stratagems group dents that group’s consistency far more.
We repeat the comparison against several unrelated classical texts and, separately, controls for shared topic vocabulary by clustering on the full text and then again on only the words common to every text - the chapter stays closer to the Han Feizi corpus under every version of the test.

Key Takeaways

A System for Evolving A Generalising Artificial Intelligence from Existing Technologies

Thu, 04 Jun 2020 00:00:00 +0800

TL;DR

True self-referential self-improvement is mathematically impossible: a system cannot accurately grade its own homework.
Systems which do improve themselves lean on an external, incontrovertible measure of performance rather than judging themselves from within.
It proposes non-volatile disk space occupied as such a measure for an AI: an ungamable, universal reward drawn from an evolutionary “survival” heuristic rather than a task-specific “correctness” heuristic.

Key Takeaways

The core move is trading self-reference for external grounding. Rather than trying to build a system that can accurately judge its own intelligence, the design routes around the problem entirely by picking a proxy - disk space - that is universal and cannot be gamed.
Compounding skill acquisition, not any single capability, is what would make this general rather than narrow. The 8x8 grid argument and the diffusion-model corroboration are both used to support the claim that generality emerges from breadth of prior solutions, not from a more powerful model per se.

Update, 2026: This is a 2020 design document, kept below as written. Its argument, that self-improvement has to lean on an external, ungameable measure, is the starting point for the system later built and tested in Survival is the Only Reward and set out as a paper in Generalising from Self-Produced Data.

Why it is Impossible to Program a General AI Using Conventional Methods

Thu, 04 Jun 2020 00:00:00 +0800

TL;DR

Any rule-based system faces an infinite space of possible transformations. Some contradict each other, so no set of instructions can cover them all.
Every instruction also forbids at least one alternative. So what a system cannot do always matches or exceeds what it can do. That is why conventional programming cannot build general AI: each fix is a new instruction with new prohibitions.
The proposed alternative is a fractal system. One simple instruction generates independent subsystems whose rules never interfere, so the parent stays simple.

Key Takeaways