Preparing for recovery
February 2026 note · Text revised September 30, 2026.
A failure tests both a system and the conditions in which a team can intervene. Preparing for recovery means making room for both dimensions from the design stage.
I try to define the situations we want to handle: an interrupted service, an unavailable dependency, an incomplete process, or information that has become uncertain. Useful guarantees depend on the need, the acceptable cost, and the available resources.
Providing guidance for action
When intervention is needed, the team should be able to find out what happened, what remains reliable, and which actions are possible. Understandable errors, observable state, and an accessible procedure can help inform that decision.
An up-to-date runbook is part of this preparation. Recovery can combine automatic mechanisms, guided steps, and human decisions, depending on the consequences of the action and the context.
Practicing recovery and learning
A recovery exercise makes it possible to observe the expected behavior in a given situation: what restarts, which data is available, how long it takes, and where intervention is needed. It also helps identify missing information and refine procedures.
The difficulties encountered provide material for improving the system with the people who operate it. My aim is to reduce repetitive tasks and uncertainty while giving teams the means to choose and act.