Case 17

Incident Runbook Automation

Incidents open a living playbook: alert-linked diagnostics, escalation, and safe actions — calm steps instead of blank-chat heroics.

Open on GitHub ↗ Discuss this pattern All cases
01

How it works

Flip the playbook — not a blank chat.

Incident Runbook Automation by Andrey Lesnikov — every alert arrives with diagnostics, escalation, and a human gate for risk.

Stacked runbook pages linked from alert to diagnostics and human-gated action
Playbook metaphor — calm steps beat heroic improvisation.
Animated playbook pages flipping from alert through diagnose to safe act
Lifecycle — Alert → Diagnose → Escalate → Act safely
  1. 01

    Attach the playbook to the alert

    Paging without a next step is how nights go long. The runbook is already linked.

  2. 02

    Run diagnostics first

    Copy-pasteable checks confirm the symptom before anyone touches production levers.

  3. 03

    Escalate with a path

    Ownership and next humans are written — not discovered in a group DM.

  4. 04

    Act behind a human gate

    Safe remediation is guided; high-risk actions still need a person who can say no.

02

Evidence

From problem to proof.

How the system is framed, operated, and proven — not a slide deck summary.

01
When to use

On-call still jumps between dashboards, logs, and wiki pages with no dry-run path for remediation.

02
When not

You need proved restore drills or continuous self-healing loops — prefer Kanister or Self-Healing Infrastructure.

03
See also

Self-Healing Infrastructure and SLO-Driven Monitoring feed calmer response. Kanister Backup & Restore proves recovery when the runbook says restore.

Live signal Last commit · · ec87f6c github-actions(deps): bump github/codeql-action/upload-sarif from 4.38.3 to 4.39.0 GitHub ↗
Evidence · EVIDENCE.md ↗Checks · passingv2.2.0 · 11 Oct 2026
Open in repo
/docs
Constraints

On-call pressure, incomplete symptoms, permissions, dry-run safety, and post-incident learning.

03

Related

Nearby systems.

Case 18

Self-Healing Infrastructure

  • Detect
  • Heal
  • Verify

Transient infrastructure failures can become user-facing incidents when recovery depends on manual detection

Common failures recover with guardrails

Read case ↗ GitHub ↗
Case 03 · Flagship

Kanister Backup & Restore

  • Kanister
  • Postgres
  • Object store

A green backup job is not proof you can recover — artifacts, secrets, schema, and RTO still fail quietly

Restore proved before the incident

Read case ↗ GitHub ↗
Available for meaningful infrastructure conversations

Build systems that stay calm under pressure.

Available for conversations

Talk infrastructure

Book a working session

Meet ↗

Evidence first

For deep dives

Explore the systems

All cases ↗