The 0.3% Bug That Wasn't

What I learned the embarrassing way about debugging a system that can be wrong and right in the same way.

I spent a few days hardening my analytics agent against a bug it didn't have.

It returned 0.3% where I expected 30%. Not a cosmetic gap. That's a headline metric off by two orders of magnitude.

What made it worse was that the behavior looked stochastic. Across runs against what I thought was the same input, the agent sometimes said 0.3% and sometimes 30%. That looked like a textbook language-model failure: an ambiguously scaled number, and the model flip-flopping on whether it was a fraction or an already-formatted percentage.

So I started hardening the harness. The diagnosis was plausible. The fixes were reasonable. The tests passed. The premise was wrong.

What I couldn't see

Here's the thing I didn't appreciate until later. With ordinary software, the output tells you what the code did. If a function returns 0.3%, you know what happened: the code produced 0.3%. You can work backward from it.

An agent doesn't give you that. The same question sometimes gave me 0.3% and sometimes 30%. One of those was correct and one was a hallucination, and from the outside they were identical. Two numbers on a screen. Nothing about either one told me which was the faithful read of the data and which was the model multiplying by 100 for no reason.

I couldn't debug this by staring at outputs, because the outputs couldn't tell me the truth. I just didn't know that yet, so I kept staring.

The plausible villain

The agent pulled precomputed metrics from a structured store. In this simplified example, the number reached the model as a bare value, with no unit attached:

{ "completion_pct": <number> }

The field name suggested a percentage, but the payload never said whether that number was already on a 0 to 100 scale or a ratio on a 0 to 1 scale waiting to be multiplied by 100. Humans infer that from convention. Models infer it from context, and context can be inconsistent.

I had a ready-made suspect. "The model applied its own scaling heuristic" is a real, known failure mode. I'd seen it before. It was credible. And the credibility is exactly what let me skip the one step that would have ended this in five minutes.

I built my case from a fixture and a formula. The local fixtures held double-digit values, clearly on a 0 to 100 scale. The code that seeded my test fixtures multiplied a ratio by 100 before storing it. So 30% had to be right, and 0.3% had to be the bug.

What I never did was read the number the agent was actually handed. My traces at the time looked roughly like this:

[step 2] TOOL summarize_metrics  status=ok  result=<small payload>
  The cohort's completion rate is 0.3%.

The tool name, a byte count, and the model's interpretation. Not the value the tool returned. The one number that would have settled everything was hidden behind a generic payload-size marker. I inferred it from the fixture code and the test data instead, and they both told me to expect something in the 50s and 60s.

Two fixes that fixed nothing

First I moved the unit from implication into the tool contract. Instead of a bare number, the tool returned a structured envelope:

{
  "value": 54.0,
  "unit": "percent_0_100",
  "display": "54.0%"
}

The unit tests passed. The local fixture value 54.0 became "54.0%" with the right annotation. An agent shouldn't have to guess whether a number is a ratio or a percentage. I kept this change and I'd make it again.

Then I replayed the questions. Every run returned 0.3%.

I read that as a regression. Before my fixes, some runs had been "right." Now every run was "wrong." I was watching the agent get worse, or so I thought.

The second fix was more aggressive. I reordered the envelope so display came first, on the theory that models tend to anchor on the first field in a structured dict. Then I added explicit prose guidance to the domain skill: a percent_0_100 value is already a percentage, render it as-is, do not divide or multiply by 100.

Replayed again. Still 0.3% every time. I was about to start a third intervention when I finally did the thing I should have done first.

The row I never read

I checked the source row directly.

{ "completion_pct": 0.3 }

The value was 0.3. With thousands of assigned records and very few completions, the correct rendering on a 0 to 100 scale was 0.3%. The model wasn't mis-scaling anything. It received 0.3 and faithfully showed 0.3%. The double-digit numbers I expected never existed outside my fixtures.

Same field, same unit, wildly different magnitudes. My test fixtures used double-digit values. The source data, representing very low engagement, produced 0.3. I had quietly substituted the fixture's world for the source data, and then read every correct 0.3% as proof of a scaling bug.

The honest version is messier than "the model was right all along." Some of those early 30% outputs probably were genuine hallucinations, the model multiplying a fractional-looking 0.3 by 100. So the agent was sometimes wrong and sometimes right, and I couldn't tell the two apart because I'd anchored on the wrong magnitude.

Here's the part I missed for days: my fixes actually worked. Before the envelope, some runs hallucinated 30%. Afterward, the agent consistently preserved the source value. I read that as a regression because I had the baseline inverted. Better accuracy looked like total failure.

What changed

Two things, one boring and one harder.

The boring one: I changed my trace format. A tool call used to produce one line:

[step 2] TOOL summarize_metrics  status=ok  result=<small payload>

The tool name, the status, and a byte count. If I'd been able to see what the tool actually returned, this would have ended on day one. After this incident, the trace includes the return payload:

[step 2] TOOL summarize_metrics  status=ok  result=<small payload>
  → { "completion_pct": { "display": "0.3%", "value": 0.3 } }

That's the line I spent four days not having.

The harder one was a habit change. I added a step above my usual debugging routine: before diagnosing the agent, verify that the thing it supposedly got wrong is actually wrong. Not against the fixture. Not against my reconstruction of what the source probably holds. Against the row, from the actual environment, at the actual time.

This sounds obvious written down. It wasn't obvious at 1am with four convincing traces, passing tests, and a theory that explained all of it. What made it hard to see was not carelessness. It was the opposite. Every piece of evidence I'd gathered fit the same story, because I'd gathered all of it under the same unchecked assumption. A good, known failure mode is the most dangerous kind of suspect, because it's credible enough that you stop looking for contradictions.

I also stopped conflating "made it safer" with "fixed the bug." Both my envelope changes were worth keeping. Neither one fixed the incident I thought I was fixing, because that incident didn't exist in the environment I was inspecting. With a nondeterministic system and noisy evals, the difference between "this prevents a plausible failure class" and "this fixed the failure I observed" is easy to blur when you're in the middle of it. I try not to anymore.

The agent didn't learn anything from the 0.3% bug. I did.

Details, names, and numbers have been altered; the lessons are real.