I Published the Ladder. Then I Checked Whether I Climb It.

In May I published an intervention ladder for fixing agent behavior: eight layers, ordered from deterministic to prose, with the system prompt dead last. The one-line recap, for anyone who hasn't read it: fix failures at the most deterministic layer that can enforce the constraint — verification loops, hooks, tool schemas, error envelopes — and only reach for prompt rules when nothing below them can see the problem. That post argued the discipline. It did not demonstrate that I follow it.

Writing a discipline down and following it are different things, and I know exactly how different, because I gave myself no room to be vague about it. Every change to the harness goes into an append-only changelog, and every entry is tagged with the layer it touched. Ten weeks later, my compliance with my own ladder is not a self-assessment. It's a query.

This post is the audit.

The numbers

Here's the layer breakdown for the three months I have complete data for. The remainder in each column is eval infrastructure and refactors — changes that don't target agent behavior and don't map to a rung.

Look at April. That is the month I was drafting the ladder post — the month I spent evenings writing sentences like "the system prompt is where fixes go to decay" — and it is also the month with six prompt-rule changes, three times May and June combined. I was writing the sermon and skipping the service.

I don't think this is hypocrisy so much as physics. April was an incident-heavy month. When something is broken in front of you, a prompt rule is a ninety-second fix and a hook is an afternoon. Under pressure, the ninety-second fix wins, and no document you wrote — not even one you're actively writing — changes that. The written discipline did nothing on its own.

What did change behavior was the log. Because every entry is tagged, the violations were visible, and so was their fate: when I filter the changelog for entries later marked reverted or superseded, they cluster hard in April's prose changes. The schema and hook changes mostly stood. The prompt rules mostly didn't. Watching your own quick fixes get struck through, week after week, in a file you can't edit history out of, does something that a principles document doesn't.

By June the ratio had flipped. Deterministic guards were more than half of all changes — eight of fourteen. The single June prompt-rule change shipped with a static regression test attached, which is a sentence I could not have written in April.

When the deterministic layer won

The best advertisement for rung 2 this spring was row-level security. The requirement is simple to state: every SQL query the agent generates must be scoped to the caller's organization, no exceptions, no reliance on the model remembering to do it.

The first implementation was a string rewrite. An injector sat between the agent and the SQL execution tool and appended WHERE org_id = :org_id to the query text with a regex. It passed every test I had. Every test I had was toy SQL.

Real queries broke it in two ways. Queries ending in GROUP BY or ORDER BY got the WHERE clause spliced into the wrong position — syntactically invalid, instant failure. And queries joining two scoped views got the predicate injected without qualification, so the database threw ambiguous-column errors on a column name that appeared in both views.

The second failure produced the worst trace of the spring. The agent cannot see the injector — that is the point of a rail; it has to hold even when the model is wrong or adversarial. So from the agent's side, a query it correctly believed was valid came back with an ambiguity error it could not explain. It did the reasonable thing: rewrote the query. The rewrite hit the same injected predicate and failed the same way. It tried again. Four retries deep, the trace shows an agent methodically fighting its own security rail and losing, burning tokens on a bug it was structurally incapable of observing.

The fix was to stop treating SQL as a string. The injector now parses the query into an AST and wraps every table reference in a scoped subquery — the query the agent wrote stands untouched, and every table it reads from is already filtered by the time the query sees it. Splice position stops being a concept. Ambiguity can't arise, because the scoping happens inside each reference rather than bolted onto the end.

Here's why this belongs on the ladder's low rungs and nowhere else. A prompt rule — "always scope your queries to the caller's org" — could not have fixed this, and not just because models forget. Security that depends on the model remembering a sentence is not security. But the deeper reason is about visibility: the actual constraint lives in the structure of the query — where a predicate can legally attach, which references need qualification — and only a parser can see structure. The model sees text. The regex saw text. The failure was text-shaped tooling applied to a tree-shaped problem, and the fix was picking the layer that operates on trees.

When the ladder inverted

Now the honest one.

The requirement: questions that cross tenant boundaries — "how do we compare to other companies on the platform?" — must get a clean refusal. Not data, not an error. A polite, explained no.

Ladder instinct said rung 2. Detect the ask in a hook, raise a refusal, never let the model near the decision. I built exactly that. It went green in the mock harness and never fired correctly in staging — not once. The agent SDK wrapped exceptions raised from hooks in its own error type before they reached the response path, so instead of the polite decline, users would have seen a raw run-error. Readers of the error-contract post have already met this failure family — a recoverable policy rejection surfacing as a fatal error — but that post was about what the error should have looked like. This is about the decision that post skipped: which layer should have owned the refusal in the first place.

I reverted the hook within a day. The fix that held is the ladder's worst-ranked layer, used deliberately: inject the caller's organization name and ID into the prompt context, and let the model itself recognize a cross-tenant ask and decline it, in its own words, with its own explanation. It has held since.

Two lessons, and I want to be precise about both.

First, mock-green is not production-green. My hook tests exercised my code against my mock of the SDK. The bug lived in the real SDK's exception handling — exactly the surface the mock papered over. The test suite was measuring my agreement with myself.

Second, the ladder is a heuristic, not a law, and this case is what its failure mode looks like. Detecting "is this question asking to compare across tenants?" is a semantic judgment. Semantic judgments are bad ground for deterministic hooks — you either match too little or too much, and both are user-visible. And producing a graceful, natural-language refusal is something no layer below the model could do in that SDK even when detection succeeded; the plumbing turned every below-the-model refusal into a crash. When the upper rungs structurally cannot see the constraint — or cannot express the required behavior — prose is not the lazy option. It is the correct rung. The discipline isn't "never touch the prompt." It's "know why you're at the rung you're at."

The gate got a version bump

The May post also included a five-question gate that any new prompt rule had to pass. One question required the failure to reproduce in at least two evals before a fix could ship.

Ten weeks of changelog entries show that uniform rule failing in both directions at once. For prose changes, it got ignored — "the fix is one sentence, why wait for a second repro" is exactly the April pathology, and the gate didn't stop it. For deterministic safety fixes, it was actively obstructive: when the row-level-security injector produced its first confirmed failure, waiting for a second tenant-isolation incident to satisfy a process checkbox would have been malpractice, so I skipped the gate there too — correctly, but off the books.

So the rule split. Prose changes still require two independent eval reproductions; the friction is the feature, because prose is where decay lives. Deterministic fixes may now ship from a single smoking-gun trace plus a unit test that pins the literal failing shape — the actual query, the actual splice — with a follow-up verification entry owed in the changelog once evals catch up. The process document that carries the gate got a version bump for this. The discipline is versioned like software now, and this was its 1.x release: same contract, corrected behavior.

What the audit actually taught me

The May post ended by telling you to treat the system prompt as the last resort. Ten weeks of my own data say something sharper: you will not treat it that way just because you wrote it down. In the month I was writing that very sentence, I violated it six times.

What moved the numbers wasn't the principle. It was the append-only log that made every violation queryable and every reverted quick fix a permanent, visible strikethrough. A discipline you don't measure is a preference. The log is what turned mine into a practice.