Building Software
Engineering Fundamentals for the Agent Era
Contents Section 7, Operations
Backups that were never restored and turn out to be empty, partial or corrupt when they are needed.
Two 'redundant' instances that share one database, one availability zone or one credential.
A health check that reports healthy while the application can't reach its database.
An incident closed the moment the symptom stops, before the cause is known, so it comes back the next week at the same hour.
Designing systems that survive component failures, recovering data and service when they don't, and learning from every incident.
An agent set up nightly database backups to a storage bucket in the same account and region as the database, plus a health check endpoint that returns 200 without touching the database. Find the failure scenarios in which both mechanisms report success while the service or its data is gone.
Watch
The author of Release It! shows how integration points, missing timeouts and chain reactions turn one failure into an outage, and how circuit breakers, bulkheads and load shedding contain it.
Cook explains why complex systems always run partly broken and why 'human error' is a starting point for an investigation rather than a root cause, which is the basis of blameless postmortems.
A step-by-step retelling of the 2017 GitLab outage, in which several backup and replication mechanisms that looked healthy turned out to be empty or broken when a restore was needed.
Catalogs production failure modes (cascading failures, blocked threads, unbounded result sets) and the stability patterns that keep one broken dependency from taking down the whole product.
Covers backup strategy, restore testing, recovery objectives and replication from the operator's side, including why a backup nobody has restored cannot be trusted.
Google's chapters on designing for recovery, disaster planning, crisis management and recovery aftermath treat outages and compromises as one discipline, including shared-credential and blast-radius failure domains.
Reference
GitLab's own postmortem, with the timeline, the backup and replication methods that failed silently, and the systemic fixes that followed, which also makes it a model for writing one.
Reference
Explains the trade-off between shallow liveness checks and deep dependency checks, and how to keep a health check from reporting healthy while the service can't reach its database, or from turning one dependency failure into a fleet-wide outage.