Runbook

What to do at 3am. Written for someone tired and unfamiliar.

Format.md
Length76 lines
Includesnone

The source

76 lines of Markdown, with nothing to install. Copy it, or open the template inside Gnomon and render it as it is.

# Runbook: <Service / Feature>

- **Owner team:** <team>
- **On-call rota:** <link to PagerDuty / Opsgenie schedule>
- **Last updated:** YYYY-MM-DD

## Purpose

What this service does, in one paragraph. Why it matters if it's down.

## Architecture Refresher

Quick diagram + 3–4 bullets so a fresh on-call can orient in under a minute.
Link to the full HLD for details.

- Entry points: …
- Critical dependencies: …
- Datastores: …

## Dashboards

- **Service health:** <link>
- **Latency / errors:** <link>
- **Infra:** <link>

## Alerts

| Alert                          | Severity | Meaning                              | First response                                   |
|--------------------------------|----------|--------------------------------------|--------------------------------------------------|
| `HighErrorRate`                | P1       | 5xx rate > 5% for 5m                 | Check recent deploys; consider rollback          |
| `HighLatency`                  | P2       | p95 > 1s for 10m                     | Check downstream; check DB connections           |
| `QueueDepthGrowing`            | P2       | Backlog growing for 15m              | Scale workers; check for poison messages         |
| `HealthCheckFailing`           | P1       | Liveness failing on > 1 instance     | Check logs; restart pod; escalate if persistent  |

## Common Incidents

### Symptom: 5xx spike after deploy
1. Check the most recent deploy in <CI link>.
2. If correlated, roll back: `<command>`.
3. Post in #<channel>; open incident.

### Symptom: queue backlog growing
1. Check worker count and CPU: <dashboard link>.
2. If healthy, scale: `<command>`.
3. If unhealthy, check DLQ for poison messages.

### Symptom: latency rising, errors flat
1. Check downstream service status pages.
2. Check DB slow query log.
3. Check connection pool saturation.

## Useful Commands

```bash
# Tail logs
…

# Scale workers
…

# Drain a node
…
```

## Escalation

| Level | Who                     | When                                          |
|-------|-------------------------|-----------------------------------------------|
| L1    | Primary on-call         | All P1/P2 alerts                              |
| L2    | Secondary on-call       | No ack within 10 min, or L1 requests help     |
| L3    | Team lead / EM          | P1 not mitigated within 30 min                |
| L4    | Director / incident cmd | P1 user-visible > 1h, or data loss suspected  |

## Post-Incident

- File a [post-mortem](./post-mortem.md) within 5 business days for any P1.

Render this offline

This template ships in Gnomon and renders on your machine, with no account and no diagram uploads. The browser editor is free and needs no install.

Get Gnomon Open the browser editor

Others in Architecture document templates

ADR — Architecture Decision RecordContext, decision, consequences. The alternatives section is the valuable one.
RFC — Request for CommentsFor proposing a change and inviting disagreement before building.
HLD — High-Level DesignHigh-level design: the shape, the constraints, the risks.
Tech Spec / LLDLow-level design. What an implementer needs that the HLD does not say.
NFR CatalogueThe requirements that get discovered late and expensively.
Discovery / Spike ReportFindings from a time-boxed investigation, including the dead ends.
Post-Mortem / Incident ReviewBlameless incident review. Timeline first, conclusions second.
Solution One-PagerA solution summary for people who will not read the HLD.