The Agent Reliability Gap: Why Demos Ship, Production Fails

The most common reason a multi-agent system works in a demo and falls over in production is not the model. It is that nobody decided what happens when step four returns a 429, the worker restarts mid-run, or two agents both decide to send the same email. Model quality keeps improving; the plumbing around it does not improve on its own, and the plumbing is what breaks.

Demos are single-path. Production is a graph of failure modes.

A demo runs under four hidden assumptions: every call succeeds, every tool returns well-formed JSON, latency does not matter, and the person reading the output is the person who wrote the prompt. Production removes all four at once. A retrieval API rate-limits you. A tool returns HTTP 200 with an HTML error page in the body. A deploy kills a worker at 60 percent completion. A user double-clicks. A planner retries a step that already had a side effect, and now two branches of the same run are writing to the same record.

None of that is a reasoning problem. A model that reasons twice as well will still double-charge a customer if you never defined an idempotency key. This is why teams that spend every spare hour on prompt and context tuning — a topic worth understanding, see context engineering — often plateau while teams with a mediocre prompt and a disciplined run ledger keep shipping.

So treat the model call as one component inside a distributed system, and apply the rules you would apply to any distributed system.

The four mundane failures

Note what these are: decisions, not libraries. You can close most of this gap with one Postgres table and one YAML file.

Retries and idempotency

Most retry bugs come from treating every failure the same. A 400 from a tool is a bug in your request; retrying it burns money and time. A 429 or a reset connection is transient. Encode that distinction once, in one place.

RETRYABLE = {429, 500, 502, 503, 504}

async def call_tool(name, payload, idempotency_key, max_attempts=4):
    for attempt in range(1, max_attempts + 1):
        try:
            resp = await client.post(
                f"/tools/{name}",
                json=payload,
                headers={
                    "Idempotency-Key": idempotency_key,
                    "traceparent": current_traceparent(),
                },
                timeout=20,
            )
            if resp.status_code in RETRYABLE:
                raise TransientError(resp.status_code)
            resp.raise_for_status()
            return resp.json()
        except (TransientError, httpx.TransportError):
            if attempt == max_attempts:
                raise
            await asyncio.sleep(min(2 ** attempt, 30) + random.random())

The key detail is not the backoff. It is that idempotency_key is derived from the run ID and the step ID, not generated fresh inside the loop. A retry must be the same logical request as the first attempt, or the downstream service cannot deduplicate it. Jitter matters too: without it, every worker that hit the same outage retries in lockstep and hits the recovering service together.

State, ownership and observability: declare it, do not hope for it

State belongs outside the process. Put the run ledger in Postgres or Redis, keyed by run ID, updated after each step, and have a restarted worker resume from the last completed step instead of the beginning. Frameworks name this differently — a checkpointer, a durable execution history — but the requirement is identical: process memory is not durable storage. If that distinction is new to you, the AI automation guide walks through how the orchestration layer sits next to the rest of a stack, and this comparison of workflow tools covers where the orchestration logic should live.

Ownership belongs in the step definition:

- id: draft_reply
  owner: support-agent
  timeout_s: 30
  retries:
    max_attempts: 3
    backoff: exponential
    jitter: true
    on_exhausted: park_for_human
  idempotency_key: "${run_id}:draft_reply"
  inputs: [ticket_body, customer_tier]
  outputs: { draft: string, confidence: number }
  terminal_when: "outputs.draft != null"

Every field answers a question someone will ask at 2am. on_exhausted: park_for_human beats an exception that vanishes into a log. idempotency_key makes replay safe. terminal_when stops a planner from looping forever on a step that will never succeed. And owner means the page goes to a person, not to a channel.

Observability is the cheapest of the four and the one teams skip. One trace per run, one span per step, context propagated with W3C traceparent. Minimum span attributes: run.id, step.id, agent.name, tool.name, retry.attempt, idempotency.key.

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_SERVICE_NAME="agent-orchestrator"
export OTEL_TRACES_SAMPLER="parentbased_traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.2"

That sampler setting is the difference between an observability bill you can defend and one you cannot: parent-based sampling keeps every trace that contains an error while sampling routine successes. Log full prompts only with redaction and sampling — prompt bodies are the most expensive and most sensitive thing you can emit.

What to do next

1. Write the run ledger schema this week: run ID, step ID, status, attempt count, idempotency key, timestamp. Add it before you add another agent.
2. Audit every step that writes to the outside world. If it has no idempotency key, it is not ready for retries.
3. Wire one trace through your orchestrator with run.id and step.id on every span. One end-to-end trace is worth more than a dashboard of averages.
4. Add owner and terminal_when to each step definition, then force a failure: kill the worker mid-run and confirm the run resumes instead of restarting.
5. If assembling the prompt layer, the agent runtime and the deployment tooling separately is what is blocking you, the AI Developer Stack Bundle is a $79 one-time bundle containing a prompt library, agent framework, deployment tooling and tutorials, pre-configured to work together.

Get AI Developer Stack Bundle

AI Developer Stack Bundle — $79, one-time payment, instant download. See the full breakdown on the review page.

Related reading

• The Ultimate Guide to AI Automation 2026
• How I Built a 10-Product Digital Business
• Building an AI Content Factory