Multi-Agent Systems: What Actually Runs in Production

Multi-agent systems are easy to demo and annoying to operate. The distance between a three-agent conversation that produces a nice transcript and a system you’ll let touch production data comes down to four unglamorous areas: orchestration, cost control, evaluation and failure recovery. Below is an honest split of what has landed, what hasn’t, and where the sharp edges still are.

Where multiple agents actually earn their keep

A single model with a decent tool list handles more than people assume. Multi-agent designs pay off in four specific situations:

• Conflicting context. A code-writing agent, a reviewer and a migration planner need different system prompts, different documents and different tools. Stuffing all of it into one context degrades all three. Context discipline matters more than agent count here — see the notes on context engineering.
• Genuinely parallel subtasks. Forty independent vendor pages to extract, then one merge step. Parallelism is a real win; sequential “agent A talks to agent B talks to agent C” usually is not.
• Privilege separation. The agent that proposes a database change should not be the one that can execute it. Splitting roles is a security boundary, not a personality choice.
• Long-running work. Jobs that run for hours benefit from checkpointing and resumable state, which is easier when each step is a separate agent invocation.

What doesn’t pay off: agents debating each other to improve an answer a single well-prompted call already gets right, and org-chart simulations where five LLM calls impersonate a company. Those add latency, cost and failure surface for no measurable gain.

Orchestration: three patterns that survive contact with production

The state machine wins in production far more often than the supervisor, because it’s the only one you can reason about when something breaks at 2am. It also gives you a place to hang budgets, timeouts and approval gates:

# agents/pipeline.yaml
run:
  max_steps: 12
  max_cost_usd: 1.50
  wall_clock_timeout_s: 900
nodes:
  extract:
    model: cheap-tier
    tools: [http_get, parse_html]
    retries: 2
  classify:
    model: mid-tier
    inputs: [extract.output]
  draft:
    model: strong-tier
    inputs: [classify.labels, extract.output]
    context_budget_tokens: 24000
  publish:
    model: none            # deterministic code, not an agent
    requires_approval: true
    side_effect: idempotent

Note that publish is plain code. The best cost optimization in most multi-agent systems is deleting agents. The tool-level plumbing of where these nodes run is a separate decision, covered in the workflow tool comparison.

Cost control is a routing problem

Agent bills grow in four ways: fan-out (a planner spawning eight workers), retries (a loop that keeps “improving”), context re-transmission (every hop re-sends the full transcript), and model tier mismatch (a frontier model reading raw HTML).

Levers that work:

• Budget per run, enforced in the runtime. Not a dashboard alert — a hard stop that kills the run.
• Tiered models by node. Cheap model for extraction and classification, expensive model only where judgment matters.
• Short-circuit before the model call. Rules, regex, cached results, and “is this input even new?” checks. The cheapest call is the one you skip.
• Context compaction at hop boundaries. Pass a typed summary, not the whole transcript.
• Cap handoffs. A counter with a hard failure, otherwise two agents will politely bounce work until you notice the invoice.

If you already have a general automation layer, the cost logic belongs there rather than inside each agent — see the full automation guide for where to draw that line.

Evaluation and failure recovery: what’s still unsolved

Output-level evals are not enough for agents, because the same answer reached by a broken trajectory is a liability. Practical setups check three layers: the final artifact (does the JSON validate, does the draft satisfy the rubric), the trajectory (did it call only allowed tools, stay under the step budget, avoid writes it wasn’t permitted to make), and the cost envelope.

# replay a fixed task set against a new prompt or model
for case in evals/golden/*.json; do
  agent run --config agents/pipeline.yaml --input "$case" --json \
    | tee "runs/$(basename "$case")" | jq -r .run_id
done
jq -e '.status == "ok"' runs/*.json || echo "one or more cases failed"

Twenty cases you understand beat two hundred you generated.

Failure modes worth designing for explicitly: duplicate side effects (a retry sends the email twice), infinite handoff loops, stale state when a tool schema changes, and silent degradation where an agent returns plausible but wrong output after a model swap. The last one is only caught by replay.

def run_node(node, state, run_id):
    key = f"{run_id}:{node.name}"          # idempotency key
    if ledger.seen(key):
        return ledger.result(key)          # resumed run, no duplicate write
    for attempt in range(node.retries + 1):
        try:
            out = node.execute(state)
            ledger.record(key, out)
            return out
        except TransientError:
            continue
        except FatalError:
            break
    raise NodeFailed(node.name)            # caller routes to a human queue

What is still genuinely unsolved: credit assignment across agents (working out which of five steps caused a bad run still means reading logs by hand), state sharing when agents run on different providers, and cost attribution per task when one run spans three vendors. Protocols like MCP cover tool and context plumbing reasonably well; agent-to-agent messaging is younger and still vendor-shaped. The debugging story is closer to distributed systems in 2008 than to modern APM.

My honest maturity read: single agent plus tools is production-ready; a supervised handful of workers on read-mostly tasks is production-ready with guardrails; open-ended swarms negotiating with each other are still a demo waiting on boring tooling.

What to do next

1. Collapse your graph. If one agent with the right tools and scoped context can do the job, delete the other three.
2. Add hard limits. max_steps, a handoff cap and a dollar ceiling per run, enforced by code that fails the run rather than warning you about it.
3. Make side effects idempotent and checkpoint each node’s output to a table or disk, so a resumed run cannot double-write.
4. Write 20 golden tasks with expected artifacts and replay them on every prompt, model or tool change.
5. Log one structured record per node — inputs, outputs, tokens, cost, duration, errors — keyed by run id, so a failure has an owner instead of a vibe. That is also the gap a tool like Cowork Pro fills: a one-time $59 dashboard for organising and orchestrating multiple AI agents on real projects, with task routing and run history.

Get Cowork Pro

Cowork Pro — $59, one-time payment, instant download. See the full breakdown on the review page.

Related reading

• The Ultimate Guide to AI Automation 2026
• How I Built a 10-Product Digital Business
• Building an AI Content Factory