Skip to content

Production readiness

The big picture: a feature that passes every test can still fail its users in production, and nobody finds out because nothing was built to notice. This page is the checklist of operational questions a user-visible feature answers before it ships. It is a reference, not a gate: three places in the harness ask the questions, and this page is where the answers are defined.

Where it is asked:

  • architect asks, for a user-visible feature: what is the SLO, and how will we know it is breached?
  • /dotplanning lists production readiness as an asset class in its feature-to-asset map, so a workstream that ships a user-visible feature is checked for it.
  • completeness-check verifies that a user-visible feature names its observability (the signals, the SLO, the alert or dashboard) in the issue, ADR, or MR description.

A feature that names “none, because …” with a reason passes, and so does a small change that is not behind a flag (name the rollback as “revert”). A feature that names nothing does not. Docs, chore, and internal-refactor branches are out of scope.

This page is vendor-neutral on purpose; map each item to your stack’s tools.

  • Log one JSON object per line with a fixed core: timestamp, level, service, message, trace_id, and, when known, request_id and an opaque user_id.
  • Generate the trace ID at the edge (or accept a well-formed inbound W3C traceparent), propagate it to every downstream call and background job, and return it to the client in a response header so a user report can quote it.
  • Never log secrets, tokens, or personal data; see Secure code in AGENTS.md.
  • Log a state change and every failure with enough context to act on it. Do not log per-item inside hot loops.

Feature question: which log line, with which fields, proves this feature worked for one request, and which proves it failed?

Per user-facing surface, name the four signals and where each is measured:

Signal Question Example for an API endpoint
Latency How long do successful and failed requests take? p50 / p95 / p99 of request duration, split by outcome
Traffic How much demand is there? requests per second per route
Errors What fraction fail, including wrong-but-200 results? 5xx rate plus an explicit business-failure counter
Saturation How full is the scarcest resource? queue depth, connection-pool use, disk, memory
  • An SLI is a measured ratio of good events to valid events (for example, requests under 500 ms and not 5xx, over all requests).
  • An SLO is the target for that ratio over a window (for example, 99.5% over 28 days). Pick it from what users tolerate, not from what the system happens to do today.
  • The error budget is 1 - SLO. When it is spent, reliability work outranks new feature work until the window recovers.
  • Alert on burn rate (the budget being spent faster than the window allows), not on a single bad minute.

Feature question: what is the SLI, the target, the window, and which alert fires when the budget is burning? A feature with no way to be measured has no SLO; say so and name what would be needed to measure it.

  • Liveness answers “is the process wedged?” It checks the process only, never a dependency. A dependency outage that fails liveness restarts every replica and turns a partial outage into a total one.
  • Readiness answers “can this instance serve traffic right now?” It may check required dependencies, with a short timeout, and must go unready (not crash) when one is down.
  • Startup gives slow boots (migrations, cache warm-up) room without loosening liveness.
  • Probe endpoints are unauthenticated, cheap, and return no internal detail.
  • Verify the built artifact serves real traffic, not only that the probe is green; see Served-assets smoke test.
  • Ship dark: merge behind a flag that defaults off, enable for staff, then a small percentage, then everyone. Each step has an exit criterion tied to the SLO above.
  • Rollback is a flag flip, not a deploy. Name the flag, its default, its owner, and the removal date when the feature is created; an unowned flag is debt.
  • Schema changes follow expand then contract: add nullable or defaulted columns first, deploy code that tolerates both shapes, and remove the old shape in a later release. Rolling back code must never require rolling back data.
  • Test the rollback once before you need it.
  • State per data class (user content, logs, traces, backups, analytics) how long it is kept and why. “Forever” is a decision, not a default.
  • Deletion has a defined meaning: soft delete (recoverable for N days) versus hard delete, and it reaches every copy, including search indexes, caches, exports, logs, and backups (which age out on their own schedule; state it).
  • A user’s deletion request and an account closure each have a runbook step, an owner, and an evidence record that the deletion happened.
  • A backup nobody has restored is a hope. Schedule a restore drill (quarterly is a common starting point) into a scratch environment and verify row counts and one application-level read.
  • Record the RPO (how much data you can lose) and RTO (how long you can be down); the drill measures whether the real numbers meet them.
  • Keep at least one copy in a separate failure domain and account from the primary.
  • Log each drill: date, backup used, time to restore, what broke, the fix issue.

Copy this into the incident issue when one opens.

## Incident: <short symptom, user-visible words>
- Severity: <SEV1 total outage | SEV2 major degradation | SEV3 minor>
- Started: <UTC time> Detected by: <alert | user report | engineer>
- Incident lead: <name> Communications: <name>
- Trace IDs / sample requests: <ids>
### Impact
Who is affected, what they see, and whether data is at risk.
### Timeline (UTC)
- hh:mm <event>
### Mitigation
First action is to stop the harm (flag off, rollback, scale, fail over), not to find
the cause. Record what was done and when.
### Communication
- Status page / customer notice posted: <time, link>
- Next update due: <time>
### Resolution and follow-up
- Root cause, contributing factors, and why detection took as long as it did.
- Was an error budget spent? How much?
- Follow-up issues (each one open and linked): #...

After a harness or process incident, the incident-postmortem skill writes the lesson into durable memory.