Grounding AI in chain data: why tools beat clever prompts

Ask an AI model what the TVL of a pool was last Tuesday and you will get a confident, nicely formatted answer. The question is whether it is true.

Before we used AI on client work, we wanted to know how much to trust it with on-chain questions. So we ran an experiment. We wrote 250 questions of the kind our clients actually ask, with answers we could verify from chain, and tried four setups.

• Prompt onlyThe question, nothing else
• Pasted dataA relevant data export pasted into the prompt
• Tool accessLive, read-only tools for RPC, indexer and prices
• Tools + checksTool access plus automated recalculation of every figure

Results

We scored each answer two ways. Was it wrong? And could a reviewer verify it from the answer alone, with sources cited and numbers traceable?

The jump from “prompt only” to “tool access” dwarfs anything we got from rewording prompts. A model with no data does not know it has no data; it fills the gap with something plausible. Give it a way to look things up, and it looks things up.

Why pasted data is not enough

Pasting an export helps, but it breaks down quickly. Exports go stale, they rarely contain everything a follow-up question needs, and long tables are where small transcription errors creep in. Tools let the model fetch exactly what it needs, when it needs it, and leave a log a reviewer can check.

Design rules we now follow

• Read-only by default. Data tools never sign transactions. Anything that acts goes through a separate, human-approved path.
• Every number has a source. Answers cite the tool call behind each figure.
• Recompute, don’t trust. Key figures are recalculated by plain code before they are shown.
• Say “I don’t know”. If a tool fails, the answer must say so instead of estimating.

These rules underpin every workflow we build for clients. You can see one in action in our LP stress-test walkthrough.