A field guide to evaluating a data agent: why the answer key is software, the clock is a dependency, and most of the work is debugging the evaluator. I spent the last few months building an analytics agent: you ask
What I learned the embarrassing way about debugging a system that can be wrong and right in the same way. I spent a few days hardening my analytics agent against a bug it didn't have. It returned 0.
What a production data agent taught me about treating tool failures as control flow, not exceptions.
What a production data agent taught me about treating tool failures as control flow, not exceptions.
What I learned from a SQL subagent that hallucinated columns five times in a row, and the two-skill system I now run on every harness change.
What I learned from a SQL subagent that hallucinated columns five times in a row, and the two-skill system I now run on every harness change.
This is the first post in a series about building data agents — systems that let people ask questions about business metrics in natural language. I've built more than one of these now, and these are the lessons I