TL;DR
- Stratigraphic Amenity, our open-source toolkit for scanned geological maps, exposes ten tools to AI agents through a local MCP server. Its first design assumed agents would read the structured results. Testing with real agents showed that many hosts pass the model only the text.
- Agents then invented legend labels, measured coordinates from downscaled previews, and misread a result holding 86 mineral occurrences as “1 item”. Each failure led to a specific change in how results and errors are written.
- The changes generalise: repeat critical fields in text, report record counts, put the fix inside the error message, keep preview images out of measurement, and never let a missing value look like a present one.
Key Takeaways
- Many MCP hosts show the model only a result’s text, never
structuredContent. Anything an agent must not guess - identifiers, bounding boxes, confidence, warnings - is now repeated in an evidence digest appended to the text as well. - Report records, not groups. A knowledge-query result grouped as “1 item” containing 86 occurrences got read literally by agents, so summaries now state record counts per provider and in total.
- Say what is missing in words, not with a blank field. A null label with no explanation looks like data. Every uncertain result now carries an explicit label or warning instead -
label_extraction: "not_available",low_confidence, and similar. - Errors carry their own fix, and installation stays server-side. Some hosts show only an error’s message, so the remedy lives in the message itself, and the prep tools that install assets take no arguments at all, so a model cannot be talked into fetching something outside the approved list.
The Model Context Protocol lets an AI application start a local server and call its tools. A tool result can carry two things: content, which is text and images, and structuredContent, a JSON object with typed fields. It is natural to put the real data in the structured part and a short summary in the text.
Building an MCP server is one thing; building an MCP server real agents can actually use is another. Stratigraphic Amenity is our open-source MCP server for scanned geological maps. It registers a map, detects its layout, georeferences it, queries earthquake, fault and mineral data for the area, and renders the results as an overlay. We tested it by giving real agents real maps and reading what they did. After a later round with DeepSeek V4, the tool schemas and documentation were revised again. This post collects what those rounds changed about our MCP tool design.
Many hosts only show the model text
The first finding shaped everything else. Some MCP hosts display a tool’s text and images to the model and never pass structuredContent through. A carefully typed result is invisible to the model in those hosts.
Without the structured fields, agents filled the gaps. Some invented plausible legend unit names. Others estimated coordinates by looking at the preview image.
The fix was an evidence digest appended to the text of every result. It repeats the fields an agent must not guess, one per line: the map and resource identifiers, each detected region with its bounding box and confidence, whether a legend region was found, the georeferencing residuals and bounds, the number of records per knowledge provider, and every warning. Tests check that warnings appear in the text as well as in the structured output.
The rule we took away is to treat the structured output as the machine record and the text as the part the model is guaranteed to see. Anything that changes what the agent should do belongs in both.
Report records, not items
A knowledge query groups results by provider. Early summaries counted those groups, so a mineral query that returned one group containing 86 occurrences was summarised as “1 item”. Agents took that literally.
Summaries now report records per provider and in total, in the form Knowledge query found N record(s) across P provider(s): active_faults=…, earthquake_history=…, and they add how many were returned versus found when a limit truncated the list. A count that sounds small gets treated as small.
Say what is missing, in words
The toolkit ships no OCR, so detected legend entries have no labels. An empty string looks like data. Each entry now carries label: null and label_extraction: "not_available", and the summary says labels were not extracted.
The same idea runs through the output. The legend count is reported as legend_extracted_candidates: N (not a verified map-unit count). Detections below 0.5 confidence are tagged low_confidence. When no legend is detected but regions classed as “others” exist, the warning points at them. Each is a place where an agent previously read more certainty into a result than it contained.
Previews are a different coordinate frame
Tools that return images include a small inline preview, capped at 1,536 pixels on the long edge and about 1 MB. Every bounding box in the result is in the coordinates of the original image, which may be several times larger.
An agent that measures a position on the preview and passes it to georeferencing gets a wrong answer with no error. Results now label the preview’s coordinate_frame as "preview" and the payload’s as "source", and the summary warns that the preview is a different frame from the one the bounding boxes use.
Put the fix inside the error message
Some hosts show the model only an error’s message field, not its details. So errors carry their remedy in the message itself.
When map processing is called before the detector is ready, the error reads, in effect: the detector is not ready in the server environment; call geomap_prepare_detectors after confirming with the user, or report to the operator; outstanding requirements are these. The structured part repeats that with a typed code, a trace_id that matches the server’s log line, a list of recovery_hints, and a cause limited to allowlisted identifiers such as a missing module name. No stack trace or file path reaches the model.
Two knowledge-query errors are deliberately kept apart. “No provider matched this name” and “the provider exists but cannot serve this request” need different fixes, so they say different things, and each names what the agent should change.
The agent’s shell is not the server’s environment
An agent that has its own shell access can try to fix a missing model by running the installer itself. That installs into the agent’s environment, with a different data directory and Python path from the server’s, and the server stays broken.
Two changes followed. Recovery hints now say not to run installer commands in the agent’s own shell. And the server offers geomap_prepare_detectors and geomap_prepare_knowledge, which install assets inside the server process. Those tools take no arguments at all, no URL, path or version, so a model cannot talk them into downloading anything outside the approved list. They also cannot install Python packages, and their output says when a remaining requirement is a package only the operator can install.
Check readiness before calling
geomap_list_capabilities reports which parts of the server are usable right now: map registration, map processing, georeferencing, knowledge queries and overlay rendering, plus each of the ten knowledge providers. Its summary is written as instructions, for example “Do not call geomap_process_image until map_processing is ready”.
Readiness is a necessary condition and the documentation says so. The check confirms packages and files exist; it does not test network access or credentials. The detector is the exception, because importing PyTorch and the detection runtime can crash or print to stdout. That check runs in a separate subprocess with a timeout, so a broken native library produces a clean message such as “missing shared library libGL.so.1” instead of taking the server down.
stdout belongs to the protocol
A stdio MCP server speaks JSON-RPC over standard output. One stray print corrupts the stream. The YOLO runtime prints freely, so model loading and prediction run with standard output redirected to standard error, and every log line goes to standard error. It is a small rule that breaks a server outright when forgotten.
Keep the adapter thin
The project’s contributor rules keep the MCP layer free of domain logic. The server validates input and output against JSON schemas, one adapter method per tool translates calls into the Python SDK, and all geology lives in the SDK. That made each change above a change to how results are presented, without touching how maps are processed.
The server and its agent guide are in the Stratigraphic Amenity repository. For another MCP server we built for geoscience work, with about 50 tools, see Meet the Geocluster Research Harness. The security side of the same server is covered in a companion piece on trust boundaries for a local MCP server.