Harness Engineering vs Spec-Driven Development

Harness engineering is building everything around an AI coding agent except the model: the context it reads, the tools it can call, the constraints it works inside, and the checks that catch its mistakes. Spec-driven development decides what one change must do before the agent starts. The spec is the intent; the harness is what makes the agent follow the intent and proves that it did. Teams that let agents write real code end up needing both, and each one fails in a predictable way without the other.

This page is about software built with AI coding agents. It is not about wiring harnesses in electrical engineering, or about Harness, the CI/CD company.

What is harness engineering?

The shortest definition is an equation: Agent = Model + Harness. Birgitta Böckeler's article on martinfowler.com puts it as "everything in an AI agent except the model itself". For a coding agent, that means the instructions and documents it can read, the tools and permissions it has, the architectural rules it must respect, the tests and linters that run on its output, and the loop that sends failures back to it.

The term took off in February 2026, when OpenAI's Ryan Lopopolo described a five-month experiment in which a small team shipped an internal product with zero lines of manually written code: roughly a million lines and about 1,500 merged pull requests, driven first by three engineers and later by seven. The engineers did not write code. Their job, in the post's words, became "to design environments, specify intent, and build feedback loops that allow Codex agents to do reliable work." The post's summary is four words long: "Humans steer. Agents execute."

Four ideas from that post define the field in practice:

• The repository is the system of record. "From the agent's point of view, anything it can't access in-context while running effectively doesn't exist." Decisions made in Slack or Google Docs are invisible to the agent until they are written into the repo.
• AGENTS.md is a table of contents, not an encyclopedia. A short file, roughly 100 lines, points to deeper sources: design docs, architecture notes, and execution plans checked into the repository.
• Invariants are enforced mechanically. Layering rules and "taste invariants" are checked by custom linters and structural tests, and the lint messages are written as fix instructions for the agent.
• Entropy is cleaned up continuously. Background agent runs scan for drift from agreed principles and open small refactoring pull requests, "like garbage collection."

Anthropic's engineering blog covers the same ground from the long-running-agent side. Its November 2025 post on harnesses for long-running agents pairs an initializer agent with a coding agent that works through a feature list and marks each feature passing only after end-to-end testing. Its March 2026 post on harness design for long-running application development splits the work between a planner that expands a short prompt into a product spec, a generator that builds it, and an evaluator that grades the result against criteria agreed before coding started.

Prompt, context, harness, loop

Harness engineering is often introduced as the next step after prompt engineering and context engineering, and people now search for "loop engineering" too. They are best read as nested layers, each containing the one before it:

A spec is not a fifth layer. It is an input that all four consume: the prompt quotes it, the context includes it, the harness checks against it, and the loop uses its acceptance criteria to decide when the work is done.

Harness engineering vs spec-driven development

The difference is easiest to see in review. Spec-driven development decides what the code is supposed to do before the diff exists: goal, non-goals, API contracts, edge cases, acceptance criteria, and the evidence a reviewer will ask for. Harness engineering decides how that intent is enforced every time, for every change: what the agent reads first, which files it may touch, which checks run on its output, and what happens when one fails.

Böckeler's taxonomy shows exactly where they meet. She splits a harness into guides, feedforward controls that "anticipate the agent's behaviour and aim to steer it before it acts", and sensors, feedback controls that "observe after the agent acts and help it self-correct". Each can be computational (tests, linters, type checkers) or inferential (AI code review). She also groups harnesses by what they regulate: maintainability, architecture fitness, and behaviour, and notes that the behaviour harness is the least mature and leans heavily on specifications. A functional spec is listed there as a feedforward guide.

So the spec is a guide inside the harness, and the most important one for behaviour, because it is the only guide that changes with every feature. The OpenAI post makes the same point from the human side: once agents write the code, people "prioritize work, translate user feedback into acceptance criteria, and validate outcomes." That translation is spec work.

Where they meet: every spec section becomes a check

The practical move is to make each part of the spec produce a harness control, so that writing the spec also tells the harness what to check. Here is the mapping we use:

The two rows that catch the most agent mistakes, allowed files and acceptance criteria, can be enforced with one short script. It reads the spec, fails when the diff leaves the allowed list, and fails when an acceptance criterion has no test that mentions it. We tested it against a sample repository with bash 3.2, the version macOS ships:

#!/usr/bin/env bash
# Usage: scripts/spec-gate.sh <change-name> [base-ref]
# Fails if the diff leaves the spec's allowed files, or an AC has no test.
set -euo pipefail
spec="specs/active/$1/spec.md"
base="${2:-origin/main}"
status=0

allowed=$(sed -n '/^## Allowed files/,/^## /p' "$spec" | sed -n 's/^- //p')
while read -r file; do
  grep -qxF "$file" <<<"$allowed" || { echo "out of scope: $file"; status=1; }
done < <(git diff --name-only "$base"...HEAD)

for ac in $(grep -o 'AC-[0-9]\+' "$spec" | sort -u); do
  grep -rqw -- "$ac" tests/ || { echo "no test mentions $ac"; status=1; }
done

exit "$status"

It expects the spec to carry an ## Allowed files list of exact paths and acceptance criteria numbered AC-1, AC-2, and so on. The -w flag matters: without it, a test named after AC-10 would satisfy AC-1. Wired into a pull request check, with the branch name used as the change folder name:

name: spec-gate
on: pull_request
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: scripts/spec-gate.sh "$GITHUB_HEAD_REF" "origin/$GITHUB_BASE_REF"

This is deliberately small. It does not prove the tests are good; it proves the agent stayed inside the boundary and did not skip a criterion. The test-evidence gates article covers the next step, checking that each test actually exercises the behaviour it names.

Worked harness case: refund retry endpoint

Take one change: "retry refunds when the payment gateway times out." The spec for it is short:

## Acceptance
- AC-1: a gateway timeout retries once with the same idempotency key
- AC-2: a second timeout leaves the refund pending, no duplicate refund_id
- AC-3: every attempt writes an audit event

## Allowed files
- src/billing/refund-worker.ts
- tests/refund-worker.test.ts
- specs/active/refund-retry/evidence.md

Without a harness, an agent given this spec will usually write a plausible retry loop and a test that mocks the gateway to succeed on the second call. The test passes, AC-2 is never exercised, and the agent adds a helper to src/billing/ledger.ts because it seemed tidy. Review catches some of that, late.

With a harness, the same run hits three sensors before a human looks: the scope gate flags ledger.ts; the AC check reports that no test mentions AC-2; and the gateway mock, which is part of the test harness rather than something each test improvises, has a ready-made "times out twice" state, so writing the missing test costs the agent one line. The spec said what mattered. The harness made the agent's shortcuts visible while they were still cheap to fix.

Why a test harness is separate engineering work

One reason the two terms get confused is that "test harness" is much older than "agent harness". A test harness is the layer between assertions and the system under test. In an agent harness it is the behaviour sensor, and it is its own piece of engineering, not a side effect of writing tests. A test that passes locally and fails in CI because the database was in a different state is not a flaky test; it is a test running without a harness.

The last row is the one agents need most and humans skip most. Anthropic's long-running-agent harness has its initializer agent write an init.sh for exactly this reason: every later session has to be able to start the app and test it without rediscovering how. For the contract layer in depth, see Contract Testing Plan: From OpenAPI to CI.

How each fails without the other

A spec without a harness is a document nobody enforces. The agent reads it, mostly follows it, and drifts at the edges: an extra field, a refactor outside scope, a criterion quietly dropped. Every one of those is found in human review, which is the most expensive place to find it and the place that scales worst as agents open more pull requests. A note in OpenSpec's community schema catalog says it plainly: "OpenSpec only checks that artifacts exist." That is true of any spec tool until a harness checks the code against the artifact.

A harness without a spec is a fast, green pipeline pointed at the wrong target. Linters pass, structure is clean, tests are thorough, and the feature does something nobody asked for, because nothing in the environment said what the feature was. Anthropic's March post found a related trap on the evaluation side: "When asked to evaluate work they've produced, agents tend to respond by confidently praising the work." Its fix was a separate evaluator, grading against criteria agreed before coding started. Criteria agreed before coding is a spec by another name.

The OpenAI post adds a caveat worth keeping: the agent behaviour it describes "depends heavily on the specific structure and tooling of this repository and should not be assumed to generalize without similar investment." A harness is built, not installed.

A starter harness for a spec-first team

You do not need a million-line experiment to start. For a team that already writes specs, this is roughly one week of work, in order of return:

1. Make AGENTS.md a map. Keep it short and point to where specs, architecture notes, and commands live. What belongs in it and what belongs in a spec is covered in AGENTS.md for Spec-First Teams.
2. Give every spec numbered criteria and an allowed-files list. The spec packet generator produces both.
3. Add the spec gate above to CI. It is the cheapest sensor with the highest catch rate.
4. Build one real test harness layer. Pick whichever of fixtures, a mock server, or environment bootstrap your agents stumble on most.
5. Add an evidence section to the PR template. Test names per criterion, logs, screenshots. It gives the human reviewer, the inferential sensor, something concrete to check.
6. Turn repeat failures into new controls. When the same agent mistake shows up twice, add a guide (a line in AGENTS.md or the spec template) or a sensor (a lint or test). This is the steering loop Böckeler describes, and it is what keeps the harness growing in the right direction.

If you are choosing tools for the spec side, OpenSpec vs Superpowers vs Spec Kit compares the main options, including how to run a planning tool with an execution-discipline tool, which is a harness decision as much as a spec decision.

References

• OpenAI: Harness engineering: leveraging Codex in an agent-first world (Ryan Lopopolo, February 2026)
• martinfowler.com: Harness engineering for coding agent users (Birgitta Böckeler, April 2026)
• Anthropic: Effective harnesses for long-running agents (November 2025)
• Anthropic: Harness design for long-running application development (March 2026)
• Anthropic: Effective context engineering for AI agents
• Addy Osmani: Loop Engineering (June 2026)

Quotes and figures checked against the linked sources on 2026-09-24. The spec gate script was run against a sample repository before publishing.