How Do You Know If Your Data Agent Is Any Good?

A field guide to evaluating a data agent: why the answer key is software, the clock is a dependency, and most of the work is debugging the evaluator.

I spent the last few months building an analytics agent: you ask it a question in plain English about a large operational dataset, and it figures out which tables to query, runs the SQL, and answers you. The hard part wasn't building it. The hard part was answering one deceptively simple question that everyone eventually asks about anything with an LLM in it:

Is it actually any good?

This post is what I learned trying to answer that. It's not a tidy success story. It's a tour of every way my measurement turned out to be broken. That, it turns out, is where the real work lives. If you're about to evaluate an agent of your own, I hope it saves you a few of the surprises that cost me weeks.

Why a data agent is uniquely annoying to evaluate

Much normal software has a comforting property: the same input gives the same output, so a test is just assert actual == expected. Agents throw all of that away at once.

• The output is probabilistic. Run the same question twice and you get two differently-worded answers. You can't assert equality.
• It's multi-step and tool-calling. The agent picks tools, constructs queries, reads results, and decides what to do next. "Right answer, wrong reason" is a real and dangerous category.
• It's grounded in live data. The "correct" answer depends on what's in the database right now, which is not a constant.

So you can't only write assert. You need an evaluation: a set of questions, a definition of what "good" means, something to grade against (the ground truth), and a judge that scores the agent's free-text answer against that ground truth. Four moving parts. Every one of them is something you build, which means every one of them can have bugs. That sentence is the whole post.

Over time I stopped thinking of an eval case as a question plus an answer. For a data agent, an eval case is a measurement contract, five things that all need to be pinned before a score means anything:

• What did the user mean? Which reading of the question is being answered.
• Which metric definition and grain applies? Denominator, filters, time window.
• Which data snapshot and clock am I using? Freshness, immutability, version.
• What evidence did the agent retrieve? Tool calls, SQL, returned rows.
• What rule decides whether the answer is acceptable? Judge prompt, rubric, tolerances, accepted alternatives.

Most of my failures came from one of those five contracts being missing, forked, stale, or invisible to the judge. A mismatch between agent and ground truth rarely meant "agent wrong." It meant one of the contracts had moved, and the score couldn't tell me which.

What I decided up front

A few decisions early on saved me from a lot of pain later. If you take nothing else from this, take these:

Treat the eval as a measurement instrument, not a single pass/fail gate. The temptation is to reduce everything to one number and a threshold: "85% or I don't ship." That number feels like control, but it erases all the information you actually need. I scored each answer on several dimensions instead: accuracy (do the numbers match the data), coverage (did it answer the whole question), soundness (does the logic hold), and usefulness (could the user act on it). A single overall pass/fail can't tell you that the agent reasons beautifully but keeps fetching the wrong numbers. The dimensions can, and in my case they told a very specific story. Across the suite, soundness and clarity were consistently strong, while accuracy lagged behind them. One failing question scored well on reasoning but poorly on accuracy: the agent had laid out a perfectly sound analysis on top of the wrong numbers. The collapsed verdict would have said "fail" and sent me hunting through the prompt. The dimensions said, unambiguously, the language is fine; the problem is retrieval. That turned out to be the single most useful sentence the eval ever produced, because it pointed every subsequent fix at the right layer.

To be clear about what I'm not saying: I'm not arguing against pass/fail judgments. Some excellent practitioners deliberately prefer crisp binary judgments per dimension ("is this accurate: yes/no") precisely because nobody knows what to do with a "3 out of 5." I've come around to that view. The thing to avoid isn't binary judgments; it's collapsing many binary judgments into one gate and throwing away which dimension failed. Measure several things, keep them separate, and a low-false-positive check on the dimension you care most about can absolutely block a release.

Never trust a single run, and figure out your noise before you trust any delta. An LLM judge is itself a noisy instrument. Early on, my judge would report "regressions" between runs that were pure variance: the same answer scored 4 one minute and 3 the next. What I did was boring: run the judge several times, take the median, and react to score bands (4–5 = ship, 3 = investigate, below 3 = alert) rather than chasing 0.1 fluctuations. That's one way to tame judge noise, and it worked for me, but it's not the only one. The field also leans on pairwise comparison (rank A vs B instead of scoring each absolutely, which tends to be more reliable than pointwise scores), swapping answer positions to cancel order bias, and measuring the judge's agreement with humans directly. The general principle, whichever mechanism you pick: measure how much your judge disagrees with itself before you believe any difference it reports.

Separate the questions you can grade exactly from the ones you can't. This turned out to be the most important structural decision, and it gets its own section below.

The ground truth is the hard part, and it lies in all the ways your code lies

Here's the thing nobody tells you: the answer key is software too. Mine was generated by a pipeline that pulled numbers from the live database and wrote out one expected answer per question. And it was wrong, repeatedly, in three distinct and instructive ways.

It forked silently

I ended up with two copies of the ground truth, a near-inevitable outcome the moment more than one process can write it. In my case there were two evaluation harnesses built at different times, each with its own copy of the question set, and I'd planned to run them head-to-head and compare scores. Then I actually diffed the two answer keys before trusting the comparison, and they'd already drifted apart: same total number of questions, but a different split between categories. A chunk of questions had been reclassified on one side and not the other. Neither copy carried a version stamp anywhere (no field in the data, no constant in the code), so nothing could even detect the divergence automatically. I'd have run the head-to-head, seen a gap, and read it as "the agent behaves differently" when a real part of it was "the two harnesses were grading different questions."

The fix reframed the whole project for me: the canonical, version-stamped ground truth is the actual asset. The runner, the rubric, and the judge are all replaceable around it. Whoever owns the single versioned answer key owns the eval. If your ground truth doesn't have a version on it, you don't have a baseline; you have two things that happen to disagree.

It drifted on a clock, not on a change

This was my favorite, because it sounds impossible until you see it. The numbers in the answer key stopped matching the database, even though, I was assured, the source data hadn't changed at all. Both things were true.

The structural facts (how many records exist, how many accounts are registered) were rock stable. But every metric that drifted was a time-relative classification: "users active in the last 7 days," "behind schedule," "inactive." The underlying records were staged once, with their timestamps frozen in the past. As the calendar advanced, more and more of those frozen timestamps aged past the 7-day window. One metric, distinct users active in the last week, slid from a healthy nonzero count to zero between runs. Zero. It looked like a catastrophic data-loss bug. It wasn't: every staged user's last-activity timestamp was now more than seven days old, so none of them counted as active-this-week anymore. Nobody had touched the data. The clock moved and the data didn't. (The mirror image showed up too: a "days remaining" countdown that quietly ticked down toward negative across runs.)

The data didn't change. The clock did.

If your ground truth bakes in anything computed relative to "now," it has an expiration date, and nobody will be standing there to read it to you. This is most acute for data agents querying live databases, but the general shape is worth checking for in any eval: a value that looks static is silently a function of time.

My first instinct was the obvious workaround: regenerate the answer key every day so it always matches the live data. That works, and it's also a treadmill. You're re-deriving (and re-reviewing) ground truth constantly, and you can never compare today's eval run against last week's because the target moved underneath you. The treadmill is the clock-drift problem; daily refreshes just let you run on it faster.

So I pushed the fix down a layer, into the data itself. The agent reads from a set of presentation tables in my warehouse; I converted those tables to Apache Iceberg, whose storage format keeps an immutable snapshot of the table on every write and lets you query the table as it existed at a past moment with a SELECT ... FOR TIMESTAMP AS OF <time> SQL query. The elegant part is how little I had to build: each pipeline run was already a full-table overwrite, and a full overwrite in Iceberg automatically creates a new retained snapshot. Time travel wasn't a feature I added; it was a capability already latent in the storage layer that I was previously throwing away. The entire fix was, essentially, stop deleting the snapshots (retain the eval snapshots you depend on), plus the wiring to point the agent and the eval at the same pinned snapshot.

That single change dissolves the treadmill. The ground truth is generated once against a pinned snapshot; the agent is evaluated against that same frozen point in time; "active in the last 7 days" means the same thing for both, forever, because now is no longer ambient. It's a snapshot ID they both agree on. The lesson that generalizes past my stack: when your eval keeps drifting because the world moves, the durable fix usually isn't to chase the world faster. It's to make the thing you measure against immutable, and version-pinning is often cheaper than you'd think because the substrate underneath you may already be keeping the versions for free.

It was never a privileged oracle in the first place

The most humbling discovery. I treated the ground truth as truth, the gold standard the agent's answers were measured against. But when I traced where one of those gold values actually came from (a question about whether a record's contents had changed since assignment), I found it didn't come from a change-log or history table at all. It read a single denormalized boolean column off the same view the agent queries, just aggregated deterministically in Python instead of by an LLM. The "oracle" and the thing it was judging were drinking from the same well.

This came to a head in a conversation with the engineer who'd staged the data. He was adamant the numbers shouldn't have drifted: "I used the exact same source dataset, the raw facts are identical." He was completely right, and the numbers had still moved, and for a while we talked past each other. The resolution was the distinction from the clock-drift story above: the raw facts (counts of things that exist) were byte-for-byte identical, and the derived, time-relative facts had moved because they're computed against "now." Same source, different clock. Two engineers can both be right and still disagree for an afternoon if nobody names which kind of number they're arguing about.

That dissolved the mystique. In this kind of eval, the ground truth wasn't a higher source of truth; it was the same data, computed a different way. (That's not universal: plenty of evals use genuinely held-out, human-verified labels that are a privileged oracle. But for a data agent whose answer key is regenerated from the same warehouse it queries, the "oracle" is just your own deterministic aggregation.) Which is genuinely useful (that's why the question is gradable), but it means a disagreement between agent and ground truth is not, by itself, proof the agent is wrong. It's also not proof the agent is right. It's a signal to investigate and adjudicate: sometimes they read the same rows at slightly different moments and the data plane itself was inconsistent. The discipline is to treat every mismatch as a question (which side is wrong here, and how would I know?) rather than reflexively trusting either one.

That question, which side is wrong, became the core diagnostic, and it's where the judge comes in.

The ambiguous-question problem (and the reframe that fixed it)

Step back from bugs for a second, because there's a deeper issue with the whole "one golden answer per question" model.

A large fraction of real questions (for me it was over a third) are genuinely ambiguous. "What's the completion rate?" has at least three defensible formulas. "How many users were assigned this?" turns out to have two legitimate denominators depending on whether you count distinct users or total assignments. In my data those landed in the same rough range but were meaningfully different, and both were correct; they just answered slightly different readings of the same English sentence. "How many active users?" depends entirely on the window you pick. For a question like that, there is no single right answer, so a single golden value is not just incomplete. It's the wrong kind of object. Grading a legitimately-ambiguous question against one frozen number guarantees you'll mark correct answers wrong.

There's a sharper version of this that bit me repeatedly, which I came to call grain confusion. One question asked for an overall rate; the agent pulled a detail-level table and summed across every segment, reporting a detail-level count several times higher than the org-level rollup the question actually needed. A dramatic inflation. The agent's SQL wasn't wrong exactly; it answered a defensible reading of an underspecified question at the wrong level of aggregation. You can blame the agent, or you can notice the question never pinned the grain. The most durable fix is to pin it, in the question and in a metric definition the agent can read, rather than to add a golden number and hope.

The instinct is to either delete the ambiguous questions (throwing away the most realistic part of your eval) or enumerate every possible answer (impossible). Both are wrong. The reframe that unlocked it for me: stop trying to store a single golden value for a question that has several legitimate answers. Depending on the question, the ground truth becomes:

• a set of acceptable answers ({reading → expected result}), graded on membership: does the agent's answer match one legitimate reading? This is the workhorse, and it's well-trodden: the text-to-SQL and open-domain-QA benchmarks landed here years ago, because single-gold under-reports an agent's real accuracy badly.
• a tolerance band for continuous values: is the answer within range, order-insensitive?
• a rubric or criteria checklist for open-ended "insight" questions, instead of a reference answer at all.

You author two or three interpretations, not unlimited values, so the cost is bounded. (Worth noting the rubric idea is contested: some practitioners I respect argue rubrics and scales just smuggle the ambiguity into the judge, and prefer a few crisp binary checks instead. For my hardest "insight" questions I still found a checklist more honest than a fake golden paragraph, but reasonable people disagree, and acceptance-sets are the part everyone agrees on.) Then you stratify: keep the exactly-gradable questions in a deterministic bucket with single golden answers, put the ambiguous ones in their own bucket graded by their own mechanism, and report the two separately so a soft "judgment-call" score never hides behind a hard "got-the-number-wrong" score.

There's one more category beyond exact answer, acceptable answer set, tolerance, and rubric: questions where the correct behavior is to ask for scope. For ambiguous questions, an agent can be wrong in two opposite ways: it can answer too confidently under an unstated assumption, or it can annoy the user by clarifying something obvious. I now think ambiguity handling deserves its own eval dimension: did the agent answer, answer-with-assumption, clarify, or refuse, and was that the right call? A question like "What's the completion rate?" should be graded differently depending on whether the entity and time window are already in context:

question: "What's the completion rate?"
expected_behavior:
  if entity is in context: answer_with_assumption
  if no entity/time window is in context: clarify
must_disclose:
  - denominator
  - grain
  - time window

The metrics that fall out of this are concrete: wrong-assumption rate (agent picked a reading it shouldn't have), missed-clarification rate (agent should have asked but answered), unnecessary-clarification rate (agent asked instead of answering a clear question), and assumption-disclosure rate (agent answered but failed to state the denominator or grain it used). For a data agent, these matter as much as answer accuracy.

Two judges: LLM-as-judge, and when that fails, agent-as-judge

For most questions, an LLM-as-judge is the only thing that scales: show a model the question, the expected answer, and the agent's answer, and have it score the dimensions. Calibrate it against human ratings until it agrees with people within their own disagreement, run it a few times, take the median. For clean, unambiguous, factual questions, this works well.

But I hit its limit hard, and the limit is worth stating plainly:

The LLM judge can't see what the agent saw.

It scores the agent's answer against a frozen expected answer. It never sees the tool calls the agent made or the rows that came back. So when the live data had drifted (see clock-drift, above), the agent would faithfully report what the database actually returned, and the judge, comparing against a stale golden value, would confidently label a correct answer a hallucination. In the flagged "hallucinations" I sampled and inspected, the judge was wrong more often than the agent.

I want to be careful here, because the failure mode I expected was different. The well-documented problem with LLM judges is unreliability through bias: position bias, verbosity bias, prompt sensitivity. The failure I expected was leniency: missing real errors, getting charmed by fluent answers. I got the opposite, a judge that was too harsh, and I think I know why, and it's worth saying because it's a self-inflicted wound others will repeat. My rubric told the judge to be harsh. I'd instructed it, roughly, that any number not present in the ground truth is a hallucination. That rule is only as good as the ground truth is complete, and mine was thin. It didn't enumerate every legitimate metric or grain the agent might reasonably use, so the moment the agent computed something defensible that I simply hadn't pre-listed, the judge mechanically stamped it a fabrication. So this isn't "LLM judges are inherently harsh." It's "a strict rubric layered over an incomplete answer key manufactures false hallucinations." That's really just the ground-truth problem wearing a different hat. The fix isn't to soften the judge into leniency; it's to stop asking the judge to adjudicate against a list you already know is incomplete.

The first fix is the cheap one, and you should reach for it before anything fancy: give the ordinary judge the retrieved context. Put the agent's tool results into the judge's prompt so it can check the answer against what was actually fetched, not just against the frozen expected answer. That alone recovers most of the gap, and it's most of what the research that quantifies this is really measuring. If you do one thing, do that.

But for the genuinely hard cases (the ambiguous stratum, and anything the first judge flags) I went further, to what I think of as agent-as-judge. Instead of comparing free text to a frozen string, I hand an evaluator agent three things:

1. read access to the live database,
2. the data agent's complete reasoning trace, every tool call it made and every result it got back, and
3. the question and the rubric.

Now the judge isn't asking "does this answer match a string I was given." It's asking the question a human reviewer would ask: "Given what this agent actually looked at, and given what's really in the database right now, is this answer defensible?" It can re-run the query itself. It can see that the agent pulled the right view but the data plane returned a placeholder. It can tell "fabricated" (numbers that appear nowhere in the agent's tool results) apart from "grounded but mismatched" (numbers that are right there in what the agent fetched, and merely disagree with a stale answer key).

That distinction is the whole ballgame, and a concrete case made me a believer. A blind judge had flagged an answer as a hallucination. The verdict said, roughly, that the answer fabricates a completion percentage and supporting counts, "which have no basis in the ground truth." When I opened the agent's trace, every one of those numbers was sitting right there in the rows its query had returned. The agent hadn't invented anything; it had faithfully reported what the warehouse handed it, and the warehouse disagreed with a stale answer key. The blind judge saw a fabrication; a judge that could see the trace saw a data-staleness problem. Same answer, opposite verdict, and only one of them would have sent me debugging the right layer. When I later split my "hallucination" flag into fabricated versus grounded-but-mismatched, most of the flagged cases I inspected turned out to be the latter. I had been calling the data plane's staleness an agent hallucination, at scale, until the judge could finally see what the agent saw.

An evaluator agent with database access is itself a complex, expensive, fallible agent. The "who judges the judge" regress is real, not rhetorical. So I don't run it on everything: the cheap context-fed LLM judge handles the deterministic stratum; the agent-judge is the deliberate, costlier escalation for cases where being able to check reality is worth paying for. It's a tool for hard cases, not a default.

A failure taxonomy worth naming

Once the trace-aware judge existed, a vocabulary emerged. I've found it useful enough to formalize:

A score tells you something changed. This table tells you where to look. Most of the debugging time I saved came from having a label for the failure before I started investigating it.

A score is worthless without a trace

Which leads to the rule I'd tattoo on the next eval I build. The first evaluation tool I used produced beautiful scores and threw away the traces. So I'd get a number ("the agent dropped dramatically this week") and have no way to act on it. Was it a real regression? A judge error? Data drift? You couldn't tell, because the evidence was gone.

When I rebuilt the eval, "preserve the full trace for every single run" was the non-negotiable requirement: the question, every tool call, every raw result, the agent's answer, and all of the judge's raw scoring runs. The payoff was immediate and repeated. The clearest case: one run came back looking like the agent had dropped sharply against the prior baseline, the kind of number that ruins a week. With the traces, it took minutes instead of days. Nearly every "new failure" had the same fingerprint in its tool results: a snapshot timestamp pinned to a sentinel date decades in the past, and an answer where the agent itself said the data looked stale. The agent's behavior was essentially unchanged; the data underneath the eval had rotted between baseline and rerun. Without the traces, that's a frightening regression and a week of bisecting prompts. With them, it's a one-line diagnosis: the snapshot moved, not the agent. A score tells you something changed. Only the trace tells you what to do about it. If your eval emits numbers but not evidence, it can measure your agent but it can never help you improve it. And improvement, not measurement, was always the point.

This is also why the judge's own output goes in the trace, not just its verdict. One run, a question scored as a total failure because the judge "couldn't parse the answer." The raw judge output, preserved, showed why instantly: all three scoring passes had hit a token limit and gotten cut off mid-sentence. The agent was fine; the judge had run out of room. You only catch that if you kept what the judge actually said, not just the number it landed on.

One corollary worth its own line: treat over-refusal as a first-class metric, not an afterthought. It's easy to celebrate an agent that correctly declines out-of-scope questions (mine refused the genuinely-unanswerable ones almost perfectly, which felt great on the dashboard) and never notice it's also declining questions it should answer. I caught one where the agent looked at a question it had every tool to answer, started to, and then bailed with a polite "here's what I can offer instead," a clean miss that the headline pass-rate happily absorbed because a refusal isn't a wrong answer, it's just a missing one. If you only measure wrong answers, a silent, over-cautious agent looks perfect. Count the refusals it shouldn't have made. Pre-launch, the true wrongful-decline rate is unknowable, so I'm treating my current measurement as a floor, not the truth.

The coverage cliff nobody warns you about

A last one, and here I'm explicitly predicting, not reporting, because as I said up top this had not reached broad use yet. My entire eval is built on a hand-authored question set against staged, mock data. That's a guess about what users will ask. My expectation is that the day real usage arrives is not a maintenance event. It's the single largest distribution shift in the product's life, and the moment a carefully-built, carefully-calibrated eval is most exposed to quietly measuring the wrong thing: real questions won't match the hypothetical ones, a judge calibrated on mock answers may not hold on real ones, and the ground truth has to be re-grounded in reality. I could be wrong about the magnitude (I've read accounts of teams whose synthetic eval held up fine post-launch), but I'd rather treat launch as a planned re-calibration than discover on the day that my green dashboard was grading a world that no longer exists. So the plan is to capture real usage from day one, sample it into the eval, and re-validate the judge against it. Ask me in a few months how the prediction held up.

The discipline, in one idea

If there's a single thing I'd hand to someone starting this, it's a posture, not a checklist:

Distrust the ruler as rigorously as the thing you're measuring.

Every artifact in your eval (the questions, the ground truth, the judge, the threshold) is a claim to be verified, not a truth to be trusted. Most of the work of evaluating an agent is debugging the evaluator. Budget for that, and the agent gets easier.

The agent was the easy part. Learning to measure it honestly, with evidence, knowing exactly how my own instruments could lie to me: that was the actual education. I'd do it again, and I'd start by assuming the answer key is wrong.

Details, names, and numbers have been altered; the lessons are real.