The Review Cadence That Keeps AI Output Honest

Most shops review AI output in bursts: hard for a week after something breaks, barely at all once things look calm. That produces a habit that tracks your mood rather than your risk, and it means each failure gets fixed one at a time instead of at the source. What holds output steady is a rhythm — three loops on three different clocks, each short enough that you will still run it during a bad week.

Three loops, one rhythm

Each loop answers a different question. Collapsing them into one big review is how people end up with a process nobody follows.

• The merge gate asks: is this change’s claim written down and checked, right now?
• The weekly audit asks: does what we shipped last week still do what its claim says, today?
• The monthly read asks: what do the individual misses have in common?

The timeboxes are the design, not a suggestion. A gate that takes an hour gets skipped on a deadline, and a skipped gate leaves nothing behind. A ten-minute gate gets grumbled at and run anyway.

The merge gate: cheap, mechanical, no taste

The gate is not a code review. It reads two things: the claim, and the witness that proves the claim. Taste arguments belong in the weekly audit, where you have the shipped item in front of you and no one is waiting to merge.

The rule that makes the gate work is that the claim is written before the work, while you still know what you meant. A claim written afterward describes what the agent happened to produce. The gate should fail on an empty claim, a missing verification line, and filler text like “looks good” — three cheap greps.

#!/usr/bin/env bash
# .githooks/pre-commit — the same file runs as a CI step
set -euo pipefail

claim=".gate/claim.md"

[[ -s "$claim" ]] || { echo "gate: .gate/claim.md is empty"; exit 1; }
grep -q '^claim: ' "$claim"    || { echo "gate: no claim line"; exit 1; }
grep -q '^verified: ' "$claim" || { echo "gate: no verified line"; exit 1; }

if grep -qiE 'TBD|TODO|looks good|should work' "$claim"; then
  echo "gate: placeholder claim — write what you actually checked"
  exit 1
fi

echo "gate: ok"

The file it checks is small on purpose:

claim: refunds older than 30 days are rejected in checkout
surfaces: checkout handler, refund job
verified: pytest tests/test_checkout.py::test_refund_window passes; order 4411 (34 days) returns 409
residual: partial refunds path not touched

The residual line matters more than it looks. Naming the surface you did not touch is what keeps the weekly audit honest, because it tells the auditor where to poke. If you already run a per-output check before anything ships, this is the same idea with a trigger attached — the handle-and-check rule is the discipline, and the merge is where it becomes non-optional.

The weekly audit: re-check shipped work against its claim

This is the loop most shops lack, and it is the one that catches drift. The gate checks a change against the claim at merge time. The audit checks the shipped thing against the same claim a week later, after adjacent changes have landed.

Two rules make it survivable. First, sample deterministically: hash the item id, take the smallest few. Sampling by anxiety is how you end up auditing only the work you already distrust. Second, re-check the claim, not the diff. If the claim was “refunds over 30 days are rejected,” you send the old order through again. You do not reread the code.

import csv, hashlib, datetime, pathlib

log = pathlib.Path("ops/ship-log.csv")  # date,week,item,claim,witness,surfaces
iso = datetime.date.today().isocalendar()
week = f"{iso[0]}-W{iso[1]:02d}"

with log.open() as f:
    rows = [r for r in csv.DictReader(f) if r["week"] == week]

# deterministic sample: the same week always picks the same items
sample = sorted(rows, key=lambda r: hashlib.sha256(r["item"].encode()).hexdigest())[:6]

print(f"audit {week}: {len(sample)} of {len(rows)} shipped items")
for r in sample:
    print(f"- [ ] {r['item']}  (shipped {r['date']})")
    print(f"      claim:    {r['claim']}")
    print(f"      re-check: {r['witness']}")

Each item gets one of three outcomes: holds, degraded, or false claim. Degraded means it works but the residual risk you noted has grown. False claim means it never did what the claim said, and the witness was too weak — that one goes into the failure log, because a weak witness is a process defect, not an incident.

Watch for the audit that always passes. Six clean weeks in a row usually means the claims are too vague to fail, not that the output is perfect. Change the sample or tighten the claim format before you conclude anything.

The monthly read: find the pattern behind the misses

Once a month, open the whole failure log — every miss from the gate, the audit, and production — and read it in one sitting. Not to fix anything yet. To count.

You are looking for repeated classes, not repeated incidents. Four different features with wrong numbers in a summary is one pattern: no source required in the prompt. Two merges that touched an unexpected surface is one pattern: the claim format has no surface field. The output of the read is exactly one change to the process — the claim template, the sample size, the log fields, or the gate script. Not a new tool, and not five changes.

The last row is where context engineering stops being an abstraction. If the model was never given the source documents, no amount of review fixes the output; you are reviewing a guess. The monthly read is how that shows up as a pattern instead of four separate arguments.

Why cadence beats intensity

Intensity responds to incidents. It is expensive per event, it decays, and it always looks at the failure that already escaped. Cadence is cheap per event and accumulates evidence: the gate produces a claim, the audit produces a verdict, the log produces a pattern.

Cadence also makes skipping visible. A missing ship-log row for Tuesday, or an audit week with no note, is a signal you can see. A skipped burst of hard review leaves no trace at all — which is exactly why nobody notices when it stops happening.

An hour a week is a real cost. So is discovering in month three that half your shipped automations quietly stopped matching their descriptions, with no record of when it started.

What to do next

1. Create .gate/claim.md with the four lines from the example above, and wire the grep script into pre-commit or your CI config today. If the gate cannot be added in under an hour, it is too complicated to survive.
2. Start ops/ship-log.csv with six columns: date,week,item,claim,witness,surfaces. Log every merge for one week. Do nothing else with it.
3. Put the weekly audit on the calendar at a fixed weekday, 45 minutes, six items, hash-sampled. Book the time before you have anything to audit.
4. Book the monthly read as a separate 60-minute block. Bring the log, count classes, and leave with one process change written down.
5. If your shop has not shipped an automation yet, the cadence has nothing to review. Ship With AI is a four-hour practical course that takes a non-coder from zero to a first working AI automation project, priced at $39 USD one-time, available at https://slashmaster6.gumroad.com/l/mgtpcn. Build the first thing, then put the three loops around it.

Get Ship With AI

Ship With AI — $39, one-time payment, instant download. See the full breakdown on the review page.

Related reading

• The Ultimate Guide to AI Automation 2026
• How I Built a 10-Product Digital Business
• Building an AI Content Factory