Prove It Before Production
Evidence-Driven Infrastructure Engineering for Kubernetes and AI
How do you know that your infrastructure change is actually safe?
The question
The question the book answers.
Most platform books show how to assemble Kubernetes, GitOps, portals and pipelines. This book asks the harder question: when terraform plan is clean, CI is green and GitOps reports Synced, what evidence do you actually have?
The model
The Verifiable Infrastructure Model.
Observe → Reconstruct → Simulate → Decide → Prove. Principle: Unknown ≠ Safe.
Learning
What you will learn.
- Separate successful execution from evidence of safety.
- Reconstruct live dependency and ownership graphs.
- Rehearse changes, including recovery and rollback, before apply.
- Emit explicit verdicts ALLOW, BLOCK or REQUIRE_APPROVAL.
- Fail closed when proof is missing (Unknown ≠ Safe), including when the evidence system itself is down.
- Treat a green canary, a signed image and a Synced app as signals, not as proof that recovery still works.
- Apply the same contracts to AI agents, inference budgets and GPU admission.
Audience
Who it is for.
Senior, Staff and Principal Platform, SRE and Infrastructure engineers.
Contents
Table of contents.
Part VI is a retrospective reading of public incident reports. It does not claim the model would have prevented any incident.
Front matter
- —Preface
- —How to read
Part I · The Problem
- 01Production Is a Terrible Test Environment
- 02Infrastructure Lies
- 03Reconstruct Reality
Part II · The Evidence Machinery
- 04The Evidence Contract
- 05Rehearse Before You Change
- 06Measure the Blast Radius
- 07Fail Closed
Part III · Recovery, Rollback, and Evidence-Plane Failure
- 08Prove That Recovery Works
- 09Prove That Rollback Works
- 10When the Evidence System Fails
Part IV · AI Agents on the Control Plane
- 11AI Agents Change the Threat Model
- 12Agent Security Contracts
- 13Inference Budget as Infrastructure Policy
- 14GPU Admission and Multi-Tenant AI
Part V · Name the Model, Then Implement It
- 15The Verifiable Infrastructure
- 16Risk Classes
- 17Adoption Maturity
- 18When the Evidence System Lies
- 19What This Looks Like in a Real Platform
- 20Build It on Monday
- 21One Change, End to End
Part VI · Public Incidents Through the Model
- 22Reading Public Incidents
- 23GitLab.com, January 2017: The Backups Nobody Had Restored
- 24Knight Capital, August 2012: Seven Servers out of Eight
- 25Amazon S3, February 2017: One Input, Two Subsystems
- 26CrowdStrike, July 2024: When the Validator Is the Evidence
- 27Facebook, October 2021: The Audit Tool and the Way Back
- 28Google Cloud, June 2025: Global Data, Regional Rollout
- 29Replit Agent, July 2025: A Code Freeze Is Not a Control
Appendices
- AAcme Evidence Walkthroughs
- BGlossary
- CReferences
- DCompanion Projects
- EReproducible Walkthroughs
- —About the Author
Companion
Verify it yourself.
The companion repository justrunme/acme-cloud-platform is public under Apache-2.0. It walks one fictional estate — Acme Cloud Platform — through blocked upgrades, proved restores, agent gates and evidence-plane outages. Evidence packages, the agent-gate corpus and policy fixtures are machine-readable and validated in CI so readers can re-run the checks.
Acme is fictional. Its logs, timings and quotas are illustrative.
On this site
Ideas from the book on this site.
- Inference Budget Guard Chapter 13 — also related to the AI runtime platform case.
- AI runtime execution plane Chapter 13 companion on this site.
- Agent Security Contract Chapters 11 and 12.
- Live Evidence Grid Chapter 4 and Appendix A — evidence packages on this site.
- Architecture Rehearsal Chapters 7 and 10 — fail-closed gates and evidence-plane failure themes.
- GPU admission on Kueue Design discussion in chapter 14 — the book describes a design to validate on your own scheduler, not a finished GPU Admission implementation claimed by the author.
Author
About the author.
FAQ
Questions.
Who is this book for?
Senior, Staff and Principal Platform, SRE and Infrastructure engineers.
What is the Verifiable Infrastructure Model?
Observe → Reconstruct → Simulate → Decide → Prove. The principle is Unknown ≠ Safe. Verdicts are ALLOW, BLOCK or REQUIRE_APPROVAL.
Is Acme Cloud Platform a real company?
No. Acme is a fictional estate. Its logs, timings and quotas are illustrative. The companion repository walks that estate through blocked upgrades, proved restores, agent gates and evidence-plane outages under Apache-2.0.
Where can I buy it?
The Kindle edition is published on Amazon. Use the Get the Kindle edition on Amazon link on this page.
Get the book
Read it on Kindle.
English Kindle edition, published on Amazon on 4 November 2026.