TL;DR
- Stratigraphic Amenity, our open-source toolkit for scanned geological maps, exposes ten tools to AI agents through a local MCP server. Its first MCP tool design assumed agents would read the structured results. Testing with real agents showed that many hosts pass the model only the text.
- Agents then invented legend labels, measured coordinates from downscaled previews, and misread a result holding 86 mineral occurrences as “1 item”. Each failure led to a specific change in how results and errors are written.
- The lessons apply to any MCP server: design every result and error for the case where the model sees only the text.
Key Takeaways
- Many MCP hosts show the model only a result’s text, never
structuredContent. Anything an agent must not guess (identifiers, bounding boxes, confidence, warnings) is now repeated in an evidence digest appended to the text as well. - Report records, not groups. Agents read a knowledge-query result grouped as “1 item” containing 86 occurrences literally, so summaries now state record counts per provider and in total.
- Say what is missing in words, not with a blank field. A null label with no explanation looks like data. Every uncertain result now carries an explicit label or warning instead:
label_extraction: "not_available",low_confidence, and similar. - Errors carry their own fix, and installation stays server-side. Some hosts show only an error’s message, so the remedy lives in the message itself. The prep tools that install assets take no arguments at all, so a model cannot talk them into fetching something outside the approved list.
Building an MCP server is one thing; building an MCP server real agents can actually use is another. This post collects what testing with real agents changed about our MCP tool design.
How MCP tool results reach the model
The Model Context Protocol lets an AI application start a local server and call its tools. A tool result can carry two things: content, which is text and images, and structuredContent, a JSON object with typed fields. It is natural to put the real data in the structured part and a short summary in the text.
Stratigraphic Amenity is our open-source MCP server for scanned geological maps, the smaller of our two; the Geocluster Research Harness exposes about 50 geoscience tools. Stratigraphic Amenity registers a map, detects its layout, georeferences it, queries earthquake, fault and mineral data for the area, and renders the results as an overlay. We tested it by giving real agents real maps and reading what they did. After a later round with DeepSeek V4, we revised the tool schemas and documentation again.
MCP tool design for what the model actually sees
Many hosts only show the model text
The first finding shaped everything else. Some MCP hosts display a tool’s text and images to the model and never pass structuredContent through. In those hosts, a carefully typed result is invisible to the model.
Without the structured fields, agents filled the gaps. Some invented plausible legend unit names. Others estimated coordinates by looking at the preview image.
The fix was an evidence digest appended to the text of every result. It repeats the fields an agent must not guess, one per line: the map and resource identifiers, each detected region with its bounding box and confidence, whether the detector found a legend region, the georeferencing residuals and bounds, the number of records per knowledge provider, and every warning. Tests check that warnings appear in the text as well as in the structured output.
The rule we took away for MCP tool design is to treat the structured output as the machine record and the text as the part the model is guaranteed to see. Anything that changes what the agent should do belongs in both.
Report records, not items
A knowledge query groups results by provider. Early summaries counted those groups, so a mineral query that returned one group containing 86 occurrences came back summarised as “1 item”. Agents took that literally.
Summaries now report records per provider and in total, in the form Knowledge query found N record(s) across P provider(s): active_faults=…, earthquake_history=…. When a limit truncated the list, they also say how many records came back versus how many the query found. Agents treat a count that sounds small as small.
Say what is missing, in words
The toolkit ships no OCR, so detected legend entries have no labels. An empty string looks like data. Each entry now carries label: null and label_extraction: "not_available", and the summary says the tool did not extract labels.
The same idea runs through the output. The legend count appears as legend_extracted_candidates: N (not a verified map-unit count). Detections below 0.5 confidence carry a low_confidence tag. When the detector finds no legend but does find regions classed as “others”, the warning points at them. Each is a place where an agent previously read more certainty into a result than it contained.
Previews are a different coordinate frame
Tools that return images include a small inline preview, capped at 1,536 pixels on the long edge and about 1 MB. Every bounding box in the result uses the coordinates of the original image, which may be several times larger.
An agent that measures a position on the preview and passes it to georeferencing gets a wrong answer with no error. Results now label the preview’s coordinate_frame as "preview" and the payload’s as "source", and the summary warns that the preview is a different frame from the one the bounding boxes use.
Error handling in MCP tool design
Put the fix inside the error message
Some hosts show the model only an error’s message field, not its details. So errors carry their remedy in the message itself.
When an agent calls map processing before the detector is ready, the error reads, in effect: the detector is not ready in the server environment; call geomap_prepare_detectors after confirming with the user, or report to the operator; outstanding requirements are these. The structured part repeats that with a typed code, a trace_id that matches the server’s log line, a list of recovery_hints, and a cause limited to allowlisted identifiers such as a missing module name. No stack trace or file path reaches the model.
We deliberately keep two knowledge-query errors apart. “No provider matched this name” and “the provider exists but cannot serve this request” need different fixes, so they say different things, and each names what the agent should change.
The agent’s shell is not the server’s environment
An agent that has its own shell access can try to fix a missing model by running the installer itself. That installs into the agent’s environment, with a different data directory and Python path from the server’s, and the server stays broken.
Two changes followed. Recovery hints now say not to run installer commands in the agent’s own shell. And the server offers geomap_prepare_detectors and geomap_prepare_knowledge, which install assets inside the server process. Those tools take no arguments at all, no URL, path or version, so a model cannot talk them into downloading anything outside the approved list: an MCP server security rule enforced in code, not in the prompt. They also cannot install Python packages, and their output says when a remaining requirement is a package only the operator can install.
Check readiness before calling
geomap_list_capabilities reports which parts of the server are usable right now: map registration, map processing, georeferencing, knowledge queries and overlay rendering, plus each of the ten knowledge providers. Its summary reads as instructions, for example “Do not call geomap_process_image until map_processing is ready”.
Readiness is a necessary condition and the documentation says so. The check confirms packages and files exist; it does not test network access or credentials. The detector is the exception, because importing PyTorch and the detection runtime can crash or print to stdout. That check runs in a separate subprocess with a timeout, so a broken native library produces a clean message such as “missing shared library libGL.so.1” instead of taking the server down.
Keeping the MCP server clean
stdout belongs to the protocol
A stdio MCP server speaks JSON-RPC over standard output. One stray print corrupts the stream. The YOLO runtime prints freely, so model loading and prediction run with standard output redirected to standard error, and every log line goes to standard error. It is a small rule that breaks a server outright when forgotten.
Keep the adapter thin
The project’s contributor rules keep the MCP layer free of domain logic. The server validates input and output against JSON schemas, one adapter method per tool translates calls into the Python SDK, and all geology lives in the SDK. As a result, each change above changed how results are presented, without touching how maps are processed.
The server and its agent guide are in the Stratigraphic Amenity repository.
