The Maturity Ladder for Automation Builders: A Four-Question Audit

Most automation advice assumes you are starting from nothing. You are not: you already have a pile of workflows at wildly different quality levels, and the useful question is which one is weakest and what habit would move it up a rung. The four-question audit below places any single workflow on a five-level ladder in about twenty minutes, and the level it lands on tells you where the next habit should go.

Four questions, in order of cost

Can you explain it in five sentences? Not the prompt, the workflow. Trigger, inputs, the decisions it makes, the outputs, and who or what consumes them. If you need the diagram open to answer, you do not have a model of the thing, which means you also cannot tell “behaving oddly” from “correct but unfamiliar.”

Can you debug it without a rebuild? When step three returns garbage, can you re-run step three with the exact input step two produced? Or do you edit a prompt, re-run the whole chain, and inspect the end result? Rebuild-debugging bundles several changes into one observation. You learn that something changed, not which thing. Saving per-step state is the whole trick here, and it is worth reading how to log input, decision and output before you redesign anything.

Is there a test that tells you when it breaks? A test here does not have to be a unit test. It has to fail without a human watching: a saved fixture replayed on demand, a schema assertion on the output, a canary input whose answer you already know. The bar is “fails loudly”, not “covers ninety percent”.

Does it hand its output to something that checks it? Generation and verification should be separate steps with separate instructions. A model asked to write a summary and then grade its own summary inside the same call is not a check; it is one opinion wearing two labels. Put deterministic validation first, schema and required fields and known-bad patterns, and spend a second model call only on the parts a validator cannot judge.

The order matters because the fixes get more expensive as you climb. Question one is a note-taking habit. Question two is structure. Questions three and four require you to build a second thing that runs on its own.

The ladder

Score each workflow on the lowest question it cannot answer. Nobody is at level five on everything.

A workflow can look productive for months at level two. That is the trap: the output is plausible, so nothing forces you to notice that you cannot inspect it.

Fill in one card per workflow

The fastest way to run the audit is to try to write the answers down. Put this next to the workflow itself, not in a wiki nobody opens.

# audit/lead-triage.card.yaml
workflow: lead-triage
owner: you
summary:                       # five sentences, present tense, no adjectives
  - A new lead row in the CRM triggers the run.
  - Inputs are the form payload and the account's plan tier.
  - The workflow classifies intent, then picks an owner from the routing table.
  - Output is a lead record with intent, owner and a confidence field.
  - A support ticket is created for anything below the confidence threshold.
debug:
  rerun_single_step: true      # replay step 3 with step 2's saved input?
  state_saved_at: [classify, route]
test:
  fixture: fixtures/lead-001.json
  assertions: structural       # invariants, not exact strings
verifier:
  separate_step: false         # the finding: nothing checks the classifier
  deterministic_checks: [intent in enum, owner exists, confidence is float]

Fill it in honestly. An empty field is the finding; do not write TBD, because you will never come back to it. A workflow with rerun_single_step: false and separate_step: false sits at level two no matter how good last week’s output looked.

Turning one answer into a test

Once the card says assertions: structural, write the check. String equality on model output rots within days; assertions on shape and invariants survive a model swap. This harness assumes your runner writes a JSON record per step and that the runner is your own script:

#!/usr/bin/env bash
set -euo pipefail

INPUT=fixtures/lead-001.json
OUT="runs/$(date +%Y%m%dT%H%M%S)-lead-triage.json"

./run-workflow.sh --workflow lead-triage --input "$INPUT" --out "$OUT"

jq -e '
  .status == "ok"
  and ([.steps[] | select(.ok == false)] | length) == 0
  and (.steps[] | select(.name == "classify") | .output.intent
       | IN("demo", "pricing", "support", "other"))
  and (.steps[] | select(.name == "route") | .output.owner | length > 0)
' "$OUT" > /dev/null || { echo "FAIL: lead-triage contract"; exit 1; }

echo "ok: lead-triage"

Two things make this worth the twenty minutes. It fails with one line instead of a diff you have to interpret, and it asserts invariants rather than prose. Run it by hand for the first month; schedule it only once the failure message is something you would actually act on at nine in the morning. If the runner cannot emit per-step records at all, that is a level-two problem wearing a test’s clothing.

The questions change when the workflow leaves your laptop

Three shifts happen once a workflow stops being personal.

Someone else runs it: question one has to be written down rather than spoken. The card is no longer an audit, it is onboarding.

Someone else buys it: question three becomes about versions. A buyer will customise the workflow, and your next update must not erase that work, which is a solved problem if you treat the template and the customisation as separate files rather than one merge target. Versioning a template so buyers do not lose their customisations covers the mechanics.

It sits inside a larger stack: a fifth question appears, can you change one layer without rebuilding the rest. That one is worth pinning on the wall separately, because the answer is usually structural rather than procedural, as argued in upgrade one layer of your AI stack without breaking the rest.

What to do next

• Pick the three workflows you touched most this month and write the five sentences for each without opening the editor. Where you stall is your weakest rung.
• For the worst one, save a single real input as a fixture and add structural assertions like the block above. Run it on demand, not on a schedule, until the failure output is actionable.
• Split self-grading. Take any step that both produces and reviews its own output, and give the review its own narrow schema plus deterministic checks that run first.
• Commit one audit/<name>.card.yaml per workflow beside the code, and update it in the same commit as the workflow change. A card that drifts is worse than no card.
• If the weak spot is the definition itself, meaning you can describe the workflow but cannot get it into something inspectable and testable, AI Workflow Builder ($99 USD, one-time) turns plain-English prompts into validated multi-agent workflow definitions you can inspect, version and run. It is available at https://slashmaster6.gumroad.com/l/amwkf.

Get AI Workflow Builder

AI Workflow Builder — $99, one-time payment, instant download. See the full breakdown on the review page.

Related reading

• The Ultimate Guide to AI Automation 2026
• How I Built a 10-Product Digital Business
• Building an AI Content Factory