What a trace report is
A trace report links three things for every promise in the spec: the acceptance criterion, the automated test that checks it, and the result of the latest run. It is generated by CI, not written by hand, so it cannot drift from what actually ran. We deliver one with every milestone and every release.
Traceability is an old idea from regulated engineering, where every requirement must be shown to be tested. We use it for a simpler reason: it answers the question a founder actually has. Not “how many tests passed”, but “did we get what we agreed”.
A sample report
A report opens with a summary, then lists every criterion. The columns stay the same across releases, so two reports can be compared line by line. This sample uses a neutral member checkout and an example release.
| Criterion | Description | Test | Result |
|---|---|---|---|
| AC-1 | Spending-limit step shown before payment methods on first checkout | checkout.spec.ts › AC-1 |
Passed |
| AC-2 | Skipping the limit requires an explicit “no limit” choice, which is logged | checkout.spec.ts › AC-2, audit-log.test.ts › AC-2 |
Passed |
| AC-3 | Raising a limit takes effect after 24 hours; lowering is immediate | limits.test.ts › AC-3, account.spec.ts › AC-3 |
Passed |
| AC-4 | Payment error keeps entries and states that no charge was made | checkout.spec.ts › AC-4 |
Passed |
| AC-5 | No third-party request before consent | consent.spec.ts › AC-5 |
Passed |
| AC-6 | Limit options operable by keyboard with a visible focus indicator | a11y.spec.ts › AC-6 |
Passed |
| AC-7 | Limit saved confirmation announced to screen readers | a11y.spec.ts › AC-7 |
Moved to v1.3 by approved delta D-12 |
Release 1.2 · spec checkout@1.2 · commit 4e1a9c0 · run on 2026-11-20
Criteria: 7 · Passed: 6 · Failed: 0 · Deferred by approved delta: 1
Quality gates: security ✓ · privacy ✓ · accessibility ✓ · performance budget ✓
A deferred criterion is never hidden. It appears with the approved delta that moved it, so the report always accounts for the whole spec.
How the report is produced
The report is assembled from the test run. Each test cites the criterion ID it covers in its name. After CI runs, a script reads the spec’s criteria and the test results, and joins them on the ID. A criterion with no matching test shows up as uncovered, and that blocks the release.
The join is deliberately simple. Spec criteria are listed in the spec file with stable IDs. Test names start with those IDs. Results come from the standard reporters Playwright and Vitest already produce. Because nothing depends on a proprietary tool, the client’s own team can run and extend the report after handover.
// Each test cites the criterion it proves; the report joins on the ID.
test('AC-4 payment error keeps entries and states no charge', async ({ page }) => {
// ...
});
What it tells a founder
A trace report tells you, in plain rows, which agreed behaviours work in this release. You do not need to read Gherkin or code. If every row shows a passing result or an approved deferral, the milestone met its spec. If not, the report names exactly which promise is unmet.
That makes acceptance concrete. In our Build engagements, a milestone is delivered with its trace report, and acceptance is judged against it. A rejection cites a failed criterion, not a general impression. A disagreement about whether something is done becomes a question about one row.
What it tells an engineer
For a technical reviewer, the report is a map from requirements to code. It shows where each behaviour is tested, which tests cover more than one criterion, and which criteria rely on a single check. Gaps in coverage are visible before they become defects in production.
It also makes regressions obvious. A criterion that passed in release 1.1 and fails in 1.2 is a regression by definition, and the diff between two reports shows it immediately. Combined with the quality gates, this means a change cannot quietly break an earlier promise.
Reports over time
Each report is stored with the release it describes. Over a Retainer, the series of reports becomes an audit trail: what the spec said at each version, and what was proven at each release. When a new team takes over, or a security reviewer asks how a rule is enforced, the answer is a link to a report.
| Question | Where the report answers it |
|---|---|
| Did this release meet its spec? | Summary line: passed, failed, deferred |
| Is rule X enforced? | The row for its criterion, and the test that checks it |
| What changed since the last release? | The diff between two reports |
| Why is a criterion missing? | The approved delta that deferred or removed it |
When a criterion fails
A failed criterion is fixed or formally moved before release. Fixing it is the default. Moving it requires a written delta that the client approves, naming the criterion and the new target version. Nothing ships with a silent failure.
During the warranty period after a build, the same rule applies. A defect is a failure against an accepted criterion, and its fix is shown as that row turning green again. New behaviour is a new criterion, written as a delta and priced before work starts.
What a trace report is not
A trace report is not a coverage percentage. Line coverage says how much code ran during tests. It says nothing about whether the agreed behaviour works. A report can show every criterion passing with modest line coverage, or fail a criterion with very high coverage. We report against the spec, because the spec is what was agreed.
Use it on your own project
The trace report template has the columns and the summary block we use. It works with any test runner that can report test names. Start by giving every criterion in your spec a stable ID, and naming each test after the criterion it proves.