Nine Claude Code runs on the same small API, asked to make its error responses consistent. We varied two things: whether the prompt was one line or a spec, and whether the repo contained its one-page mobile contract. A hidden HTTP test suite scored every run.
A three-endpoint tasks API on node:http. Its errors are a realistic mess: { error }, { message }, { errors: [...] }, a plain-text 404, a PATCH that crashes on a missing task, and a 500 that returns err.stack.
It also has docs/mobile-contract.md: shipped iOS and Android v3.x apps read body.error as a string, and they send X-Request-Id, which support asks users for.
Claude Code 2.1.284, claude-opus-5-5, headless, fresh copy each run, 37 to 69 seconds each. Recorded September 28, 2026. Run pack.
Our API error responses are all over the place. Clean them up so they're consistent.
Envelope
{ "error": "Human-readable message", "code": "project_not_found",
"requestId": "…", "details": [] }
- error stays a non-empty string (mobile contract).
- requestId echoes X-Request-Id; also set as a header.
- details is always an array.
Acceptance criteria
- AC-2 Unknown project or task → 404.
- AC-3 Invalid fields → 422 validation_failed.
- AC-4 Archived project → 409 project_archived.
- AC-5 Malformed JSON → 400 invalid_json.
- AC-6 Unexpected exceptions → 500, no stack or path.
The suite talks to each build only over HTTP, probing eight error paths. It does not check codes or status numbers the vague prompt could not have known, only 404s for unknown resources, a 400 for malformed JSON, and "never a 500".
| Check | No docs | Docs | Spec |
|---|---|---|---|
| A1 Every error response is JSON | 3/3 | 3/3 | 3/3 |
A2 Mobile contract: body.error is a non-empty string | 1/3 | 3/3 | 3/3 |
| A3 Every error body has the same top-level keys | 2/3 | 0/3 | 3/3 |
| A4 Unknown resource 404, malformed JSON 400, no 500s | 3/3 | 3/3 | 3/3 |
| A5 No stack traces or internal paths leak | 3/3 | 3/3 | 3/3 |
A6 The X-Request-Id the app sends comes back | 1/3 | 3/3 | 3/3 |
| A7 Validation errors name the field | 3/3 | 3/3 | 3/3 |
| A8 Success responses keep their shape | 3/3 | 3/3 | 3/3 |
Every run fixed the crash on a missing task, the plain-text 404 and the stack leak, and every run's own tests passed. The model is good at the part of this task that is visible in the code.
// All error responses share one shape: { error: { code, message, details? } }
GET /projects/nope/tasks → 404
{ "error": { "code": "not_found", "message": "Project not found" } }
This is a common, well-designed envelope. It is also exactly what the v3.x apps cannot read: body.error is now an object, so users see an empty red banner and a retry button that does nothing. Nothing in the code told the agent the apps existed.
// Every error response has the shape:
// { error: string, code: string, requestId: string, details?: [...] }
// `error` must stay a human-readable string: mobile v3.x displays it verbatim
// (see docs/mobile-contract.md). Clients should branch on `code`, not `error`.
Same one-line prompt. The agent found the doc, kept error a string, added code beside it, and echoed the request ID, which the doc only mentioned in passing.
Measured by calling every build with the same requests.
| Request | N1 | N2, N3 | V1 to V3 | S1 to S3 |
|---|---|---|---|---|
| Archived project | 400 | 400 | 400 | 409 |
Missing title | 400 | 400 | 400 | 422 |
| Keys on a 404 | code, error, requestId | error (object) | code, error, requestId | code, details, error, requestId |
| Keys on a validation error | + details | error (object) | + details | same four keys |
All three V runs added details only on validation errors, so clients see two shapes. It is a small thing, and a typed SDK will trip on it. The spec said "details is always an array"; all three S runs did that.
V1 left the archived project at 400: "409 would be more accurate, and the contract only requires non-2xx, but I didn't want to change status codes without asking you." That is good judgement, and it is also a decision the ticket left to the agent.
Spec runs took 60 to 69 seconds against 37 to 49 for the others, and wrote more tests. The cost is in the writing of the spec, not the run.
The biggest single effect came from a 13-line markdown file, not the prompt. A constraint that is written down where the agent looks gets followed. Keep contract notes like this next to the code, and point to them from AGENTS.md.
Docs removed the breaking change; only the spec removed the variance: one key set, 409 and 422 where the team wanted them. Those are product choices no file in the repo stated.
An HTTP-level check like A2 would have caught N2 and N3 before merge. Our API contract guide covers how to keep those checks in CI.
With a spec, all four models, down to Haiku 4.5, passed all 8 checks. Without one, not a single run on any model did: each either broke the mobile contract or left two error shapes, and all six non-spec Haiku 4.5 runs failed the status-code check (one still returned a 500 for malformed JSON).
See the cross-model resultsThe reviewer-ready version: error taxonomy, compatibility notes and SDK fixtures.
Open the API error caseSix runs where the vague prompt worked but picked different business rules each time.
Open the coupon runsNine runs on a schema change, including one that dropped a column other services read.
Open the migration runsThe API spec generator produces an error envelope, status mapping and compatibility notes you can commit next to the code.
Every number on this page comes from the recorded runs in the run pack. We fixed nothing in the agents' output before scoring it.