Four Reusable Debugging Workflows Every Builder Should Keep

A multi-step workflow rarely fails where the error appears. It fails two or three steps earlier, when a function returns a string where the next step expected a list, or when a prompt edit changes a JSON key that a parser downstream silently defaults. The fix is not clever code — it is knowing which step broke, with which input, and what it returned last week. Four small harnesses make that answer cheap enough to get in minutes.

Step echo: print every step’s input and output

Start with the highest-value record you can keep: for each step, the input it received and the output it produced. Most teams log errors and nothing else, which is why the first question in every incident is “can you reproduce it?”

Wrap the step function once, and every step you decorate reports itself:

import functools, hashlib, json, os, time

ECHO = os.getenv("STEP_ECHO", "0") == "1"
SECRET_KEYS = {"api_key", "authorization", "token", "password"}

def _redact(obj):
    if isinstance(obj, dict):
        return {k: ("***" if k.lower() in SECRET_KEYS else _redact(v)) for k, v in obj.items()}
    if isinstance(obj, list):
        return [_redact(v) for v in obj]
    return obj

def _emit(record):
    if ECHO:
        print(json.dumps(record, default=str, ensure_ascii=False), flush=True)

def step(name):
    def wrap(fn):
        @functools.wraps(fn)
        def inner(*args, **kwargs):
            payload = _redact({"args": args, "kwargs": kwargs})
            digest = hashlib.sha256(
                json.dumps(payload, sort_keys=True, default=str).encode()
            ).hexdigest()[:16]
            t0 = time.monotonic()
            try:
                out = fn(*args, **kwargs)
            except Exception as exc:
                _emit({"step": name, "input_digest": digest, "status": "error",
                       "error": repr(exc), "ms": int((time.monotonic() - t0) * 1000)})
                raise
            _emit({"step": name, "input_digest": digest, "status": "ok",
                   "output": _redact(out), "ms": int((time.monotonic() - t0) * 1000)})
            return out
        return inner
    return wrap

Three habits make this usable rather than noisy. Gate it behind an environment variable so it costs nothing when off, and turn it on for any workflow you are still changing. Redact by key name before serialising, not after, because a token leaked into a log file outlives the bug you were chasing. Truncate long outputs and keep a pointer to the full blob instead of printing tens of kilobytes into stdout.

The input digest earns its place later: two runs with the same digest took the same path into a step, so you can group failures instead of reading them one by one. If you want the longer version of this idea, it is the same argument as logging the input, decision and output, applied at the step boundary rather than the workflow boundary.

Dry run against a known-good sample

A golden case is one input you already understand, plus assertions loose enough not to flap. Capture the input, the expected properties of the output, and an explicit list of side effects to suppress:

# fixtures/research_agent/summarise-known-article.yaml
case: summarise-known-article
input:
  doc_path: fixtures/docs/attention-paper.txt
  max_words: 400
assert:
  must_contain: ["multi-head", "positional encoding"]
  must_not_contain: ["as an AI"]
  max_words: 420
side_effects:
  write_files: false
  send_email: false
  create_ticket: false
  http_post: false

Two disciplines matter here. First, assert on structure and required facts, not exact strings: an exact-match fixture on model output breaks on every harmless rewording and trains you to ignore it. Keep exact matching for the deterministic steps — parsing, formatting, routing — where a diff really does mean a bug. Second, make the dry run refuse to touch the outside world. A fixture that still sends the email is not a dry run, and you will stop running it after the first accident.

The dry run is the one harness worth putting in CI, because it answers a question you have before shipping rather than after.

Single-step replay

During an incident you rarely need to re-run the whole workflow. You need to re-run one step with the input it actually received, and compare. The echo trace already holds that input, so replay is mostly plumbing:

# pull the recorded input for one step out of a JSONL trace
jq -c 'select(.step == "fetch_source" and .status == "ok") | .input' \
  traces/run-0941.jsonl | tail -n 1 > /tmp/fetch_source.input.json

# run only that step against the recorded input
python -m jobs.research --only fetch_source --input /tmp/fetch_source.input.json

Keeping --only and --input flags on your runner is the whole investment. The payoff is separating two very different failures: the step changed behaviour, or the step is fine and its input changed. That distinction decides whether you fix a prompt or fix the step before it, and getting it wrong is how people end up rewriting a chain that had one bad handoff.

One caveat for model-backed steps: replay is not bit-for-bit reproducible. Set temperature to zero, pin the recorded model and prompt version, and compare on shape, required facts and refusal behaviour rather than exact wording. If you are still deciding whether a chunk of your pipeline should be a script or a graph at all, this comparison of visual builders and scripts covers where each one starts to hurt during debugging.

Last-known-good vs current

Store the output of every passing golden run, keyed by workflow version. A plain directory is enough:

snapshots/research_agent/
  v12/summarise-known-article.md   # last passing run, the known-good
  v13/summarise-known-article.md   # current candidate

After a prompt or model change, run the golden case, write to the current slot, and read both side by side with diff -u. Reading the diff yourself is the point — an automatic pass or fail hides the sentence that got quietly worse while every assertion still passed. Watch for three things: content that disappeared, hedges that appeared (“it depends”, “generally”), and format drift such as a dropped heading level or a list flattened into a paragraph.

This is also the cheapest guard against the most common upgrade mistake, which is changing one layer and silently breaking another.

How the four fit together

Build them in that order. Step echo makes replay possible, replay makes the diff meaningful, and the diff is what stops a small prompt tweak from shipping a regression.

What to do next

• Add the step decorator to your longest workflow this week and run one real job with STEP_ECHO=1. Do not wait for a failure to find out what your trace looks like.
• Write one golden fixture by hand for the workflow that has cost you the most debugging time. One case is enough to start; add a second only after the first has caught something.
• Add --only and --input flags to your runner so a single step can be replayed from a recorded input.
• Snapshot the golden output of your next passing run into snapshots/<workflow>/<version>/ and diff it after your next prompt change.
• If the workflow definition itself is what keeps getting rewritten, AI Workflow Builder turns plain-English prompts into validated multi-agent workflow definitions you can inspect, version and run. It is a one-time $99 download.

Get AI Workflow Builder

AI Workflow Builder — $99, one-time payment, instant download. See the full breakdown on the review page.

Related reading

• The Ultimate Guide to AI Automation 2026
• How I Built a 10-Product Digital Business
• Building an AI Content Factory