The System Prompt Is the Last Resort

While building a natural-language data-agent harness, I kept feeling the same unhelpful pull. The model was capable. The integrations were ordinary. And yet, every week, the same reflex kept coming back:

"Let me just add one rule to the system prompt to fix this."

"Let me just add one tool that handles this case."

"Let me just add a retry here."

Each fix was individually defensible. None of them held up. Here are the traces that finally taught me why.

The amnesic subagent

I had a SQL-generation subagent. The parent agent received a question, decided SQL was needed, and dispatched a subagent that returned a SQL string. The parent then ran that SQL via an execute_query tool. If the SQL referenced a non-existent column, execute_query returned a structured error envelope naming the offending table and pointing the agent at a schema_lookup tool.

It's a standard architecture. The subagent existed to keep the full table schema out of the main agent's context. It failed in a very specific way.

Here is the simplified anatomy of a single run:

• Turn 1. Subagent writes SQL with a hallucinated column. execute_query returns SCHEMA_MISMATCH.
• Turn 2. Parent re-dispatches the subagent with a "complexity hint" quoting the envelope verbatim. Subagent writes a different hallucinated column. Same envelope back.
• Turn 3. Parent escalates the hint to ALL CAPS. Subagent writes a third hallucinated column.
• …several rounds in, a generic iteration-cap brake fired, the agent apologized, the user got nothing useful, and latency had ballooned.

I watched this same shape across three traces over the month, each captured after I'd added another "fix." The latency curve was the punchline: latency roughly doubled across successive fixes. Each prior fix had made retries more informative — but not fewer. The system was getting more articulate about its own failure.

My reflex was to tighten the subagent's prompt. Add "you must call schema_lookup before writing SQL." More emphasis. CRITICAL. ABSOLUTE.

That instinct was wrong, and I want to be specific about why. The subagent was instantiated fresh on every dispatch — new context window, new system prompt, no access to the prior failure, the parent's complexity hint, the envelope, or the session notes. Each call started amnesic. The parent's only recourse on failure was to re-dispatch a new amnesic subagent. There was no possible amount of prompt tightening that would fix this — the architecture had structurally amputated the very context the subagent needed to converge.

The fix wasn't a rule. It was: delete the subagent. Lift its tools to the parent. The parent now sees the SCHEMA_MISMATCH envelope, calls schema_lookup in the same context, writes new SQL, retries — all inside one continuous reasoning chain. No amnesia by construction.

Net change: one tool boundary removed, the implementation simplified, one prompt rule rewritten rather than expanded, and the retry loop closed deterministically. The "add another prompt rule" instinct would have made the system worse.

This isn't a story about SQL. It's a story about the layer the bug actually lived at — and how easily I'd have patched the wrong one.

Pattern 1: the intervention ladder

After the subagent collapse, I went back through every harness fix I'd shipped that month and asked: which layer did the bug actually live at, and which layer did I patch?

The mismatch rate was uncomfortable. Most "the model misbehaved" reports were getting fixes in the system prompt. Most of them should have been fixed somewhere else.

Here's the ladder I now walk top-down before touching anything:

The asymmetry: earlier layers are more durable, later layers are more flexible but more fragile. A hook fires deterministically. A prompt rule is advisory. A tool schema is enforced by the API. An # IMPORTANT: line is read at the model's discretion.

Most "the model should just know X" bugs are actually a layer-1 verification gap — the model literally cannot tell whether it's right — disguised as a layer-7 prompt problem. The SQL subagent above was a layer-6 boundary bug disguised as a layer-7 prompt problem. The principles aren't novel; they're lifted from Anthropic's Building Effective Agents, Writing Effective Tools for Agents, Claude Code Best Practices, and Cognition's Don't Build Multi-Agents. What was new for me was the discipline of walking the ladder every time, not relying on having internalized it.

The five-question test I now run before adding any rule:

1. Does the failure reproduce in multiple independent evals? Not one trace.
2. Have I walked layers 1–6 and written down why each can't hold the fix?
3. Is the rule short, and have I deleted or merged adjacent guidance?
4. If I removed this rule a week from now, would the agent demonstrably regress?
5. Is the rule global? Conditional rules go in a Skill, not the system prompt.

If any fail, the rule isn't ready.

Pattern 2: Shape A vs Shape B

The subagent collapse needed a sharper test than "rethink the architecture." Here's the one I now apply.

There are two shapes of subagent:

Claude Code's Explore, Plan, and general-purpose subagents are all Shape A. They return a terminal answer. If the answer is wrong, the parent doesn't re-enter the subagent — it changes approach.

My SQL subagent was Shape B by construction. The retry loop's amnesia wasn't a bug in the subagent's prompt; it was the defining property of Shape B.

The single test I now apply to every proposed subagent: can the parent ever re-invoke this subagent because its previous output failed downstream? If yes, it's Shape B — don't build it. Lift its tools to the parent instead.

This rule has prevented several follow-on subagent boundaries that would have been wrong. It's the most portable thing I extracted from the whole experience.

The discipline that makes the patterns stick

The two patterns above are the load-bearing ideas. But patterns alone don't survive contact with a busy week — I needed a system that forced me to use them every time, and produced a record I could read in six months.

That system is two skills (in the Agent Skills sense) and an append-only changelog:

• harness-guidance fires before I touch the harness. It's the ladder, the Shape A/B test, and the five-question test, all wrapped in a checklist that won't let me skip steps.
• harness-changelog fires when the change ships (or when I log an observation without a fix). It writes one entry per change, appends one row to a slim index.

The whole thing is one loop:

signal (a trace or aggregate metric)
   → diagnosis (smoking gun + ranked hypotheses)
      → change (one scoped harness edit)
         → verification (targeted trace + regression suite)
            → log entry (append-only, one row in an index)

Three rules I refuse to break:

1. Append-only. Once an entry is committed, its body is frozen. Corrections happen by writing a new entry, not by editing the old one. The only permitted in-place edit is a single pointer flip — superseded_by: ENTRY-later-structural-fix — on an older entry's frontmatter when something replaces it.

This sounds rigid until you've spent an hour in a six-month-old changelog trying to figure out which of three contradictory rules is the current one. With append-only history, you read the most recent entry that names the topic and you're done. Being able to see what past-you believed is more valuable than a clean retcon.

2. One change per entry. If a session fixes three things, that's three entries. The frontmatter has a files_changed: field whose job is to make this checkable: if the file list spans unrelated components, the entry is wrong-shaped.

3. Every entry names a specific signal. A trace ID with turn numbers, or a metric delta. Never "it felt wrong." If you can't name the signal, the entry is kind: observation, status: investigating until you can.

After the active changelog grew large enough to become hard to navigate, it started doing exactly what the guidance warned about for system prompts — eating context. The fix was the same fix the guidance prescribes: split the monolith into a slim index plus per-entry files. The irony was not lost on me.

HARNESS_CHANGELOG.md                 # preamble + slim index table
harness_changelog/
  entries/
    ENTRY-YYYYMMDD-NN.md             # one file per entry, full body
  traces/
    <conv-id>.txt                    # raw traces, referenced from entries

What an entry looks like

---
id: ENTRY-YYYYMMDD-NN
date: YYYY-MM-DD
status: shipped | reverted | superseded-by-HC-XXXX | proposed-not-shipped | investigating
kind: change | observation | verification
component: system-prompt | tool-definition | tool-loop | hooks | retrieval |
           subagent-dispatch | skill-pack | eval-harness | ...
phenotype: hallucination | loop | silent-deflect | data-gap | instrumentation | null
supersedes: []
superseded_by: null
trace_refs: [conv-1f0b8fa3.txt]
files_changed: [src/agent/tools.py]
commit: <hash or PR link>
---

# ENTRY-YYYYMMDD-NN — short title

**Triggering signal.** Trace ID + turn range, or metric delta.

**Behavior observed.** 2–4 sentences. Quote tool args, stop reason,
turn count, token usage. Specifics, not paraphrase.

**Smoking gun.** The single line of trace/config/prompt that made
the failure inevitable. Or `root-cause-pending`.

**Reasoning considered.** ≥2 hypotheses and why the losers lost.
This is the field that makes the entry useful in six months.

**Change made.** Plain English + `file:line` pointer. Prompt edits
get inline before/after snippets — git diffs rot, snippets don't.

**Result.** Targeted trace pass/fail. Regression suite deltas.
Token/latency/cost delta. "Did anything else regress?" Explicit
"no" is fine; silence is not.

**Regression test pinned.** Path, or justified "none, because…".

**Risk / watch-list.** What might regress later.

**Follow-ups.** Linked entries, open questions, deferred items.

Two details that took iteration to get right:

• phenotype and component are orthogonal. Phenotype names what the agent did wrong (hallucination, loop, silent-deflect, data-gap, instrumentation). Component names where the fix lives. A SQL hallucination fixed in a hook and a SQL hallucination fixed in retrieval share a phenotype but not a component. Splitting the axes lets me ask "show me every loop bug" or "show me every fix in hooks.py" — different questions, different answers.
• Inline prompt snippets, not git diffs. Code lives in version control. Prompts evolve in places version control doesn't reach: config files, runtime injection, skill files. The only durable record of "what the prompt said at time T" is the inlined snippet in the changelog entry.

The entry that justified the whole system

The most useful entry in my changelog isn't one where I shipped a fix. It's a pair.

ENTRY-brake-hook shipped a brake hook — a deterministic safety net that detected the SCHEMA_MISMATCH retry loop and force-stopped the agent on the second occurrence. It worked. Latency capped, agent stopped thrashing, regression tests passed. If I'd been in a hurry that day, I would have shipped it and called it done.

ENTRY-structural-fix shipped four hours later. It deleted the brake's underlying cause by collapsing the SQL subagent — the change I described at the top of this post.

The second entry only got written because the first entry's "Reasoning considered" field — written while the brake felt like the answer — had a paragraph I left honest:

"The brake stops runaway; it doesn't fix the underlying shape. Every first-SCHEMA_MISMATCH still burns a subagent call; every subsequent unknown-data question will hit the same amnesic retry until the brake fires."

Reading that paragraph back made it obvious the brake was a bandage. Without the entry, the brake would have shipped, the symptoms would have gone away, and the structural bug would have stayed. The discipline that forced me to write down the losing hypotheses caught the winning one too.

That's the value proposition in one example. Not "the changelog is useful." More specific: the field that names the alternatives you considered is the field that catches your own bandages — on the next read, when you, or whoever inherits the system, finally has time to look.

Sample skill files

Skeleton structure of the two skills. The full text — checklists, red flags, intervention-ladder rationale, supporting quotes — is more than you want inline.

harness-guidance/SKILL.md — skeleton

---
name: harness-guidance
description: Use BEFORE any change to the agent harness — prompts,
  tools, hooks, context management, subagent dispatch, skills, evals.
  Steers away from reflexive rule-stuffing toward principled
  interventions at the layer that can actually see the constraint.
---

# Harness Guidance

## Core principles (apply in order)
1. Simplicity first — no eval delta, no change.
2. Context is the scarcest resource; protect it before adding to it.
3. ACI (agent-computer interface) deserves as much care as HCI.
4. Make wrong use impossible, not just discouraged.
5. Hooks are deterministic; prompt rules are advisory.
6. Prune before you add.
7. Share full context; fragment only when token math forces it.
8. Intervene at the layer that can see the constraint.

## The intervention ladder
[8-row table: verification → hook → schema → description →
 skill → subagent → prompt → multi-agent]

## Shape A vs Shape B
The only test for a subagent boundary: can the parent ever re-invoke
this subagent because its previous output failed downstream? If yes,
it's Shape B — don't build it. Lift its tools to the parent instead.

## Add-a-rule checklist (all five must hold)
1. Reproduces in multiple independent evals.
2. Layers 1–6 walked and ruled out, in writing.
3. Rule is short, with adjacent guidance deleted or merged.
4. Removing it would demonstrably regress behavior.
5. Rule is global. Conditional → Skill, not prompt.

## Add-a-tool checklist (all four must hold)
1. Maps to a real, distinct subtask current toolset can't express.
2. Unique name, returns only high-signal information.
3. You can delete or merge an overlapping older tool.
4. You ran the description-rewrite pass against a test agent.

## Red flags — stop before patching
[7-row table of "you're thinking" → "what it actually means"]

## Pre-change checklist
1. Name the signal (trace ID or metric delta).
2. Reproduce in an eval before changing anything.
3. Locate the smoking gun.
4. Walk the ladder. Write why each higher layer can't hold the fix.
5. Pair every add with a delete or merge.
6. Apply the smallest change that makes the eval pass.
7. Verify (targeted trace + suite deltas).
8. Hand off to harness-changelog.

harness-changelog/SKILL.md — skeleton

---
name: harness-changelog
description: Use when observing agent behavior from a trace OR when
  making any change to the agent harness. Writes a new entry file
  and appends one row to the index.
---

# Harness Changelog

## The loop
signal → diagnosis → change → verification → entry file + index row.
No step skippable.

## Storage layout
HARNESS_CHANGELOG.md        # preamble + index
harness_changelog/
  entries/HC-YYYYMMDD-NN.md
  traces/<conv-id>.txt

## Append-only invariant
Never edit a committed entry body. Only permitted in-place edit:
a single `superseded_by:` pointer flip.

## Entry frontmatter
[id, date, status, kind, component, phenotype, supersedes,
 superseded_by, trace_refs, files_changed, commit]

## Entry body (≤40 lines of prose)
Triggering signal · Behavior observed · Smoking gun ·
Reasoning considered · Change made · Result ·
Regression test pinned · Risk/watch-list · Follow-ups

## Red flags — stop and ask
- Can't name a specific trace or metric.
- files_changed spans unrelated components → split entries.
- Smoking gun is "the LLM was being weird".
- Body creeping past 40 lines.

## Phenotype enum (closed, deliberately small)
hallucination · loop · silent-deflect · data-gap · instrumentation · null

What to do next

Three things in priority order.

Walk the ladder before you touch the prompt. The system prompt is the seventh of eight layers, and the six layers above it are invisible to you when you're staring at a misbehaving trace. The fact that you can't see them is not evidence they're not where the fix lives. It's evidence you need a checklist.

Apply the Shape A/B test to every subagent you've already built, and every one you're about to. If the parent ever re-invokes the subagent because its previous output failed, the architecture is wrong and no amount of prompt tightening will save it. This is the cheapest single test I know of for an entire class of agent bug.

Write your harness changes down, append-only, one per entry, with a "Reasoning considered" field that names the losing alternatives. Not because the log is useful (it is), but because the field that names what you almost did is the field that catches your own bandages on the next read. Including the bandages you would otherwise have shipped today.

The system prompt is the last resort. Treat it that way.

Details, names, and numbers have been altered; the lessons are real.