01
Known failure modes still wait for humans — remediation is tribal and untested under pressure.
Case 18
Self-Healing Infrastructure
An immune loop with a human brake: detect, bounded heal, verify — and stop automatic remediation when the blast radius demands approval.
01
How it works
Heal the common case — brake the dangerous one.
Self-Healing Infrastructure by Andrey Lesnikov — auto-remediation with proof, plus a hard stop for high-risk actions.
-
01
Detect with a clear signal
Health and failure modes that matter — not every flake becomes a heal storm.
-
02
Heal inside bounds
Remediation is limited and reversible — power without a blank check.
-
03
Verify the result
The loop only closes when the signal recovers — hope is not a status.
-
04
Hit the human brake
High-risk actions stop for approval. Auto-heal is strength; the brake is wisdom.
02
Evidence
From problem to proof.
How the system is framed, operated, and proven — not a slide deck summary.
02
You need pre-deploy blast-radius rehearsal rather than runtime healing — Architecture Rehearsal owns that loop.
03
Incident Runbook Automation and SLO burn rates decide when to heal. Kanister proves recoverability when healing is not enough.
04
v2.2.0 ships bounded remediation: per-namespace budget in a sliding window, per-object cooldown, and approval annotations for dangerous node actions (reboot / drain). Defaults are fail-safe (PR #65): DRY_RUN defaults to true so a fresh install never mutates the cluster; live remediation requires DRY_RUN=false and a non-empty REMEDIATION_NAMESPACES allowlist containing the target namespace (otherwise allowlist_empty / namespace_not_allowed). In CI a crash-looping deployment was remediated 20.1 s after fault injection, and with REMEDIATION_BUDGET_PER_NAMESPACE=2 the brake engaged (budget_exhausted) on the next attempt (run 38062111574, docs/results.md). CI wall clock on the pipeline cluster, not a large-cluster SLO.
Live signal
github-actions(deps): bump github/codeql-action/upload-sarif from 4.38.3 to 4.39.0
GitHub ↗
Open in repo
Constraints
False positives, blast radius, rollback safety, observability confirmation, and human override.
03
Related
Nearby systems.
Incident Runbook Automation
Incidents are slower when context, dashboards, logs, and recovery steps live in different places
Repeatable response → lower MTTR
Kanister Backup & Restore
A green backup job is not proof you can recover — artifacts, secrets, schema, and RTO still fail quietly
Restore proved before the incident
SLO-Driven Monitoring
Dashboards can look healthy while users experience latency, errors, or degraded workflows
Alerts from SLO burn, not chart noise
Available for meaningful infrastructure conversations