Runbook

What to do at 3am. Written for someone tired and unfamiliar.

The source

76 lines of Markdown, with nothing to install. Copy it, or open the template inside Gnomon and render it as it is.

# Runbook: <Service / Feature>

- **Owner team:** <team>
- **On-call rota:** <link to PagerDuty / Opsgenie schedule>
- **Last updated:** YYYY-MM-DD

## Purpose

What this service does, in one paragraph. Why it matters if it's down.

## Architecture Refresher

Quick diagram + 3–4 bullets so a fresh on-call can orient in under a minute.
Link to the full HLD for details.

- Entry points: …
- Critical dependencies: …
- Datastores: …

## Dashboards

- **Service health:** <link>
- **Latency / errors:** <link>
- **Infra:** <link>

## Alerts

| Alert                          | Severity | Meaning                              | First response                                   |
|--------------------------------|----------|--------------------------------------|--------------------------------------------------|
| `HighErrorRate`                | P1       | 5xx rate > 5% for 5m                 | Check recent deploys; consider rollback          |
| `HighLatency`                  | P2       | p95 > 1s for 10m                     | Check downstream; check DB connections           |
| `QueueDepthGrowing`            | P2       | Backlog growing for 15m              | Scale workers; check for poison messages         |
| `HealthCheckFailing`           | P1       | Liveness failing on > 1 instance     | Check logs; restart pod; escalate if persistent  |

## Common Incidents

### Symptom: 5xx spike after deploy
1. Check the most recent deploy in <CI link>.
2. If correlated, roll back: `<command>`.
3. Post in #<channel>; open incident.

### Symptom: queue backlog growing
1. Check worker count and CPU: <dashboard link>.
2. If healthy, scale: `<command>`.
3. If unhealthy, check DLQ for poison messages.

### Symptom: latency rising, errors flat
1. Check downstream service status pages.
2. Check DB slow query log.
3. Check connection pool saturation.

## Useful Commands

```bash
# Tail logs
…

# Scale workers
…

# Drain a node
…
```

## Escalation

| Level | Who                     | When                                          |
|-------|-------------------------|-----------------------------------------------|
| L1    | Primary on-call         | All P1/P2 alerts                              |
| L2    | Secondary on-call       | No ack within 10 min, or L1 requests help     |
| L3    | Team lead / EM          | P1 not mitigated within 30 min                |
| L4    | Director / incident cmd | P1 user-visible > 1h, or data loss suspected  |

## Post-Incident

- File a [post-mortem](./post-mortem.md) within 5 business days for any P1.

Render this offline

This template ships in Gnomon and renders on your machine, with no account and nothing sent to a server. The browser editor is free and needs no install.

Get GnomonOpen the browser editor

Others in Architecture document templates