Production readiness
The big picture: a feature that passes every test can still fail its users in production, and nobody finds out because nothing was built to notice. This page is the checklist of operational questions a user-visible feature answers before it ships. It is a reference, not a gate: three places in the harness ask the questions, and this page is where the answers are defined.
Where it is asked:
architectasks, for a user-visible feature: what is the SLO, and how will we know it is breached?/dotplanninglists production readiness as an asset class in its feature-to-asset map, so a workstream that ships a user-visible feature is checked for it.completeness-checkverifies that a user-visible feature names its observability (the signals, the SLO, the alert or dashboard) in the issue, ADR, or MR description.
A feature that names “none, because …” with a reason passes, and so does a small change that is not behind a flag (name the rollback as “revert”). A feature that names nothing does not. Docs, chore, and internal-refactor branches are out of scope.
This page is vendor-neutral on purpose; map each item to your stack’s tools.
Structured logging and trace IDs
Section titled “Structured logging and trace IDs”- Log one JSON object per line with a fixed core:
timestamp,level,service,message,trace_id, and, when known,request_idand an opaqueuser_id. - Generate the trace ID at the edge (or accept a well-formed inbound W3C
traceparent), propagate it to every downstream call and background job, and return it to the client in a response header so a user report can quote it. - Never log secrets, tokens, or personal data; see Secure code in
AGENTS.md. - Log a state change and every failure with enough context to act on it. Do not log per-item inside hot loops.
Feature question: which log line, with which fields, proves this feature worked for one request, and which proves it failed?
Golden signals
Section titled “Golden signals”Per user-facing surface, name the four signals and where each is measured:
| Signal | Question | Example for an API endpoint |
|---|---|---|
| Latency | How long do successful and failed requests take? | p50 / p95 / p99 of request duration, split by outcome |
| Traffic | How much demand is there? | requests per second per route |
| Errors | What fraction fail, including wrong-but-200 results? | 5xx rate plus an explicit business-failure counter |
| Saturation | How full is the scarcest resource? | queue depth, connection-pool use, disk, memory |
SLOs and error budgets
Section titled “SLOs and error budgets”- An SLI is a measured ratio of good events to valid events (for example, requests under 500 ms and not 5xx, over all requests).
- An SLO is the target for that ratio over a window (for example, 99.5% over 28 days). Pick it from what users tolerate, not from what the system happens to do today.
- The error budget is
1 - SLO. When it is spent, reliability work outranks new feature work until the window recovers. - Alert on burn rate (the budget being spent faster than the window allows), not on a single bad minute.
Feature question: what is the SLI, the target, the window, and which alert fires when the budget is burning? A feature with no way to be measured has no SLO; say so and name what would be needed to measure it.
Health and readiness probes
Section titled “Health and readiness probes”- Liveness answers “is the process wedged?” It checks the process only, never a dependency. A dependency outage that fails liveness restarts every replica and turns a partial outage into a total one.
- Readiness answers “can this instance serve traffic right now?” It may check required dependencies, with a short timeout, and must go unready (not crash) when one is down.
- Startup gives slow boots (migrations, cache warm-up) room without loosening liveness.
- Probe endpoints are unauthenticated, cheap, and return no internal detail.
- Verify the built artifact serves real traffic, not only that the probe is green; see Served-assets smoke test.
Rollout and rollback with feature flags
Section titled “Rollout and rollback with feature flags”- Ship dark: merge behind a flag that defaults off, enable for staff, then a small percentage, then everyone. Each step has an exit criterion tied to the SLO above.
- Rollback is a flag flip, not a deploy. Name the flag, its default, its owner, and the removal date when the feature is created; an unowned flag is debt.
- Schema changes follow expand then contract: add nullable or defaulted columns first, deploy code that tolerates both shapes, and remove the old shape in a later release. Rolling back code must never require rolling back data.
- Test the rollback once before you need it.
Data retention and deletion
Section titled “Data retention and deletion”- State per data class (user content, logs, traces, backups, analytics) how long it is kept and why. “Forever” is a decision, not a default.
- Deletion has a defined meaning: soft delete (recoverable for N days) versus hard delete, and it reaches every copy, including search indexes, caches, exports, logs, and backups (which age out on their own schedule; state it).
- A user’s deletion request and an account closure each have a runbook step, an owner, and an evidence record that the deletion happened.
Backup and restore drills
Section titled “Backup and restore drills”- A backup nobody has restored is a hope. Schedule a restore drill (quarterly is a common starting point) into a scratch environment and verify row counts and one application-level read.
- Record the RPO (how much data you can lose) and RTO (how long you can be down); the drill measures whether the real numbers meet them.
- Keep at least one copy in a separate failure domain and account from the primary.
- Log each drill: date, backup used, time to restore, what broke, the fix issue.
Product incident runbook template
Section titled “Product incident runbook template”Copy this into the incident issue when one opens.
## Incident: <short symptom, user-visible words>
- Severity: <SEV1 total outage | SEV2 major degradation | SEV3 minor>- Started: <UTC time> Detected by: <alert | user report | engineer>- Incident lead: <name> Communications: <name>- Trace IDs / sample requests: <ids>
### ImpactWho is affected, what they see, and whether data is at risk.
### Timeline (UTC)- hh:mm <event>
### MitigationFirst action is to stop the harm (flag off, rollback, scale, fail over), not to findthe cause. Record what was done and when.
### Communication- Status page / customer notice posted: <time, link>- Next update due: <time>
### Resolution and follow-up- Root cause, contributing factors, and why detection took as long as it did.- Was an error budget spent? How much?- Follow-up issues (each one open and linked): #...After a harness or process incident, the incident-postmortem skill writes the lesson
into durable memory.