Testable AI tasks

AI Acceptance Criteria for Coding Agents

Acceptance criteria for AI coding are numbered, testable statements of what a change must do, written so an agent can build against them and a test can check the result. In our recorded Claude Code runs, one-line prompts passed every check they stated and failed only on the rules nobody wrote down. This page covers what those criteria need to include, a template, the real spec from those runs, and how to draft criteria with AI without letting it make the decisions.

Last updated: November 11, 2026

Where agents went wrong without criteria

We gave Claude Code the same checkout repo with the one-line ticket "Add coupon codes to checkout. Make sure invalid codes do not break payment." and, separately, with a spec. A hidden suite of 11 checks, written before any run, scored every result. Five checks tested rules the one-line ticket never mentioned. On Opus 5.5, the vague runs passed every check the ticket stated; every failure was on an unstated rule:

Hidden checkIn the one-line ticket?Vague runsSpec runsSpec line that covers it
Expired, disabled or unknown code leaves checkout payableYes3 of 33 of 3AC-2
Processor is charged exactly the displayed totalYes3 of 33 of 3AC-1
A user cannot redeem the same coupon twiceNo2 of 33 of 3Rule: once per user; AC-3
Two open checkouts cannot both spend a one-per-user couponNo2 of 33 of 3AC-4: re-check inside pay
Applying without paying does not use the coupon upNo3 of 33 of 3Rule: redeemed only when paid
A second coupon replaces the firstNo3 of 33 of 3Rule: second replaces first

Selected rows from the 11-check suite on the coupon run page, Opus 5.5. On Haiku 4.5, all three vague runs failed both reuse checks; with the spec, all three passed every check.

Stated rules held

Whatever the ticket said, the agent did. In all three Opus 5.5 runs, "invalid codes do not break payment" held for expired, disabled and unknown codes alike.

Unstated rules were a coin toss

One run decided coupons have no usage limit, and said so in its summary. Its own tests passed, because it wrote them for the rule it chose.

Interfaces drifted silently

Every vague run threw an exception on a bad code; the spec runs returned { ok: false, reason } as AC-2 required. No hidden check failed on it, but a frontend written for one breaks on the other.

Six rules for criteria an agent is checked against

RuleWeakStrong
Number every criterionA bullet list of expectationsAC-1 to AC-n, cited by tests, tickets and review comments
Name the interface and the exact result"Invalid codes show an error""applyCoupon returns { ok: false, reason: 'expired' } and does not throw"
Write down the rules the request assumesNothing about reuse"A coupon can be redeemed once per user. It counts only when an order is paid."
Cover what can change between two stepsValidate the coupon when it is applied"Redemption is re-checked inside pay"
Describe behavior, not implementation"Use a helper to validate coupons"The observable result at the interface; the helper is a task, not a criterion
Make sure each one fails todayA criterion the current code already meetsName the observation that would show it false, and confirm it fails before the change

The third and fourth rules are the ones a human teammate rarely needs written down and an agent always does. A colleague assumes a coupon is single-use because they know the business. An agent reads the data, finds no limit, and builds none. Our acceptance criteria examples guide has more Given/When/Then patterns for the general case.

Copy-ready acceptance criteria template

Use this block inside a spec.md or a ticket. Keep the rules section even when it feels obvious; it is where the unstated decisions go.

acceptance-criteria.md
## Rules
- [Business rule the request takes for granted: limits, who it applies to,
  when something counts]

## Acceptance Criteria
- AC-1 Main behavior: Given [state], when [action at a named interface],
  then [exact observable result].
- AC-2 Failure path: Given [invalid input or provider failure], when [action],
  then [exact error code or return shape]; [what stays unchanged].
- AC-3 Rule check: Given [state that breaks a rule above], when [action],
  then [refusal and its reason].
- AC-4 Between steps: Given [state that changed after step one],
  when [the irreversible step runs], then [it re-checks and what it does].
- AC-5 Non-goal guard: Given the change is complete, when the diff is reviewed,
  then no file, schema, API or dependency outside scope has changed.

## Evidence
- A test per AC, named after it (ac1_..., ac2_...).
- Each test fails on the commit before the change.

Example: the spec from the recorded coupon runs

This is the acceptance criteria section of the spec the agent received, copied from the run pack. With it, all three Opus 5.5 runs passed every hidden check and made the same decisions.

coupon-spec.md
## Rules
- A coupon is either percentOff (of the subtotal) or amountOffCents.
- A coupon can be redeemed once per user. It counts as redeemed only
  when an order is paid, not when it is applied.
- Applying a second coupon replaces the first.

## Acceptance criteria
- AC-1 Valid coupon: getSummary shows the discount and the discounted
  total; pay charges exactly that total.
- AC-2 Unknown, disabled or expired code: applyCoupon returns
  { ok: false, reason } with reason unknown, disabled or expired.
  It does not throw, and the checkout keeps its previous total
  and stays payable.
- AC-3 Code already redeemed by this user: returns
  { ok: false, reason: 'already_redeemed' }.
- AC-4 Redemption is re-checked inside pay: if the user redeemed the
  coupon on another order after it was applied here, pay drops the
  coupon and charges the undiscounted total.
- AC-5 An amount-off coupon larger than the subtotal brings the pre-tax
  amount to 0, never below.
- AC-6 A successful apply returns { ok: true, code, discountCents }.

The gap it left. The spec never says what happens if a coupon expires after it is applied but before payment. All three spec runs made the same choice and charged the discounted total, and one run flagged the question in its summary. A consistent answer is easier to review than three different ones, but it is still a decision nobody made. AC-4 re-checks redemption between the two steps; a complete spec would re-check expiry there too.

Drafting acceptance criteria with AI

An AI acceptance criteria generator, including the ones on this site, is good at the format and the obvious cases. It is weak at exactly the part that matters: the decisions the request leaves open, because a model asked to write criteria will fill those gaps with plausible guesses. Ask it to surface the gaps instead.

draft-criteria-prompt.md
You are drafting acceptance criteria for a coding agent. Do not decide
business rules yourself. For the request below:

1. List every decision the request leaves open as a question:
   limits, who it applies to, when something counts, what happens on
   failure, and what can change between two steps.
2. Draft numbered criteria AC-1..n only for what is already decided.
   Each one names the function, endpoint or screen, the input, and the
   exact result or error code.
3. Mark any criterion the current code would already pass.

Request:
[paste the ticket]

Check an AI-drafted list before using it

  • Did it invent a rule, such as a usage limit, instead of asking?
  • Does every criterion name the interface and the exact result?
  • Is there at least one failure path and one boundary or permission case?
  • Would each criterion fail on today's code?
  • Does anything describe implementation rather than behavior?

Generators on this site

The Gherkin generator turns a user story into Given/When/Then scenarios in a .feature file. The spec packet generator drafts criteria together with scope, tasks and an evidence checklist. Both run in the browser and need no account; treat their output as a first draft for the checklist on the left.

From criteria to review evidence

Criteria pay off at review. Ask for one test per criterion, named after it, and check that each test fails on the commit before the change; a test that never failed proves nothing about the code that made it pass. Then read the diff for behavior that maps to no criterion: an extra state, a renamed field, a new dependency. Treat it as drift and move it into a follow-up spec rather than merging it because it looks useful. The AI PR review checklist walks through this review step by step.

Related resources

Acceptance criteria examples

Given/When/Then patterns for features, APIs and edge cases.

Read the guide

The recorded agent runs

Every prompt, diff and hidden-suite score behind the table above.

Open the runs

Vibe coding vs spec coding

What changed across 78 runs when the agent got a spec instead of a prompt.

Compare

AI acceptance criteria FAQ

What are acceptance criteria for AI coding?

They are numbered, testable statements of what a change must do, written so that a coding agent can build against them and a test or reviewer can check the result without asking what was meant. Compared with criteria written for people, they need to name the interface and exact result, and spell out business rules a human teammate would assume.

Can AI write acceptance criteria?

It can draft the format and the obvious cases well. It should not make the decisions, such as how often a coupon can be used or what happens when a state changes between two steps, because those are the gaps the request left open. Ask the model to list open decisions as questions, then have a person answer them.

How many acceptance criteria does an AI coding task need?

Usually three to seven: the main behavior, at least one failure path, and the boundary or permission cases. The coupon spec in our recorded runs had six criteria plus three written business rules.

Should acceptance criteria name functions or files?

Name the public interface the behavior is observed through, such as a function, endpoint or screen, because that is where tests run. Do not name internal helpers or files to change; those belong in the scope or plan, not in the criteria.

Given/When/Then or plain sentences?

Either works if each criterion names a starting state, an action and an observable result. Given/When/Then makes the three parts hard to skip; plain sentences are shorter. The coupon spec used plain sentences with the function names and return values spelled out.

Before an agent codes, write the criteria a reviewer could test without your help, including the rules you would never think to say out loud.

Copy the template