Scope Your First AI Automation Like a Software Project

Most first AI automations fail on scope, not tooling. “Automate my inbox” is a department, not a task, and no model or orchestrator will make an undefined job well-defined. The cheapest hour you spend on the project is the one before you write any prompt.

Scope problems get misdiagnosed as tooling problems

When an automation produces garbage, the reflex is to swap the model, add retrieval, or move to a different workflow tool. Sometimes that helps. More often you have moved the same unbounded job onto a new stack, where it fails for the same reason.

The useful question is not “which tool?” but “what is the unit of work?” A unit of work has a trigger you can name, an input you can point at, an output a specific person reads, and an owner who notices when it is wrong. Miss any one of those and you do not have an automation yet.

Rows one and two are the same business goal at different resolutions. Only one of them can be finished. Row four fails the trigger test outright, so there is nothing to test. Tool selection is a real decision, but it comes after this one — see the tool comparison if you are weighing options.

Write the one-page spec

Six fields, one page. If it does not fit on a page, the task is too big. Split it.

task: triage-unassigned-support-email
trigger: new message appears in shared inbox folder "unassigned"
input:
  source: that folder only
  shape: one email thread, text; ignore attachments
output:
  shape: {label: billing|bug|howto|spam, priority: low|normal|urgent, draft_reply: string}
  destination: saved as a draft on the same thread
  never: send, delete, or archive
owner: you
fallback: leave the thread unassigned and tag it "ai-review"
done_when: see test cases below

Three rules that save rewrites later.

Name the output shape. Free prose is hard to test. A fixed set of keys with a constrained vocabulary lets you diff two runs and see what changed. If the model must return a label from a list, write the list.

Keep actions reversible in v1. Saving a draft is reversible. Sending the reply is not. Your first automation should be able to be wrong without a customer noticing.

Write the input boundary. “That folder only” is a boundary. “Whatever is relevant” is how you end up debugging a pipeline that decided to read your calendar.

If you are still deciding whether the task needs a model at all, the automation framework covers that call before this step.

Pick one painful task, not one impressive task

The first automation should be boring. Painful means frequent, annoying, and low-stakes — the chore you do badly at 4pm. Impressive is the thing you would put in a demo, and it usually has the widest input space.

Score three candidates, 1 to 5:

Take the highest total, but drop anything scoring 2 or below on boundedness or verifiability. A task you cannot check is a task you cannot finish.

Then build the harness before the trigger. If you wire the webhook first, every test costs a real inbound event and you cannot reproduce anything.

mkdir -p fixtures
# Replace pipeline.run with your own entrypoint. Run against fixed files, not live data.
export AUTOMATION_MODE=dry_run
python -m pipeline.run --input fixtures/email_typical.json   --out /tmp/o1.json
python -m pipeline.run --input fixtures/email_empty.json     --out /tmp/o2.json
python -m pipeline.run --input fixtures/email_malformed.json --out /tmp/o3.json
jq -S . /tmp/o1.json   # stable key order makes diffs readable

Three fixtures take ten minutes and turn the rest of the project into a repeatable loop instead of a guessing game.

Define the fallback before the success path

Ask two questions in writing: what happens if the automation never fires, and what happens if it fires and is wrong?

The fallback should be the process you already run today, not a new one you have to maintain. For most first projects the right default is queue-and-notify: the automation does not act, it files the item where a human will see it, and it pings an owner. Silent success is fine. Silent failure is not.

# automation.config.yaml
task: triage-unassigned-support-email
mode: shadow              # shadow -> live -> off
on_error: queue           # queue | notify_owner | fail_closed
max_retries: 1
fallback_owner: "you@example.com"
fallback_path: "manual triage in the shared inbox, exactly as today"
review_queue: "label: ai-review"

mode: shadow means the automation produces output into a review queue and takes no action. Decide in advance how many reviewed runs you need before switching to live, and keep off as a one-word kill switch anyone can flip. Use fail_closed for anything touching money or outbound customer email.

Make failures visible. When a run dies, the log line should name the step, the input id, and the raw model output. The rules for that are in Designing Agent Workflows You Can Actually Debug.

Write the definition of done as test cases

“Works well” is not done. Done is a list you can run.

Minimum set: three typical inputs, two messy ones (a forwarded chain, a request buried at the bottom), one empty, one malformed or oversized, and one where the correct answer is to refuse and hand off. Write the expected output for each before you run it. Cases written after the fact always pass.

Then a stopping rule for the shadow period, using thresholds you set yourself. A common shape: ten consecutive reviewed runs with no hand edits, zero sends, and every failure landing in the review queue with a reason attached. Pick numbers you are willing to defend and put them in the spec.

Finally, write the shutdown instruction in the README: which file to edit, which flag to set, who to tell. The person who needs it will be you, in six weeks, with no memory of how any of it works.

What to do next

1. Write the one-page spec for one task today. If the six fields do not fit on a page, cut the task in half and write it again.
2. Score three candidates on frequency, pain, boundedness and verifiability. Build the highest total that clears 3 on boundedness.
3. Create three fixtures and a local run command before wiring any trigger or webhook.
4. Set mode: shadow and on_error: queue, then write down how many reviewed runs you need before going live.
5. Keep the test cases and the shutdown line in the same file as the spec, so “done” and “off” live together.

If you want the guided version of this sequence, Ship With AI is a four-hour practical course that takes a non-coder from zero to a first working AI automation project, for a one-time $39.

Get Ship With AI

Ship With AI — $39, one-time payment, instant download. See the full breakdown on the review page.

Related reading

• The Ultimate Guide to AI Automation 2026
• How I Built a 10-Product Digital Business
• Building an AI Content Factory