Running the Six-Phase Lifecycle

An incident is not a moment; it is a lifecycle. Six phases, in order, each with an owner and an exit condition: prepare, detect, contain, eradicate, recover, and learn. Teams that improvise inside an incident lose. Teams that execute a rehearsed lifecycle win. The work that determines the outcome happens before the alarm, in preparation.

Prepare is the phase nobody funds and everybody needs. Preparation is the plan, the roles, the authorities, the tools, and the muscle memory. Define the incident-commander role and make it a function, not a person -- whoever holds it runs the response and owns the decisions, and the rank of the room does not override them. Pre-authorize containment actions so that at 3 a.m. an analyst can isolate a segment or revoke a credential without hunting for a signature on a form. Stage the tooling: out-of-band communications that survive the corporate network being compromised, forensic capture that works on both servers and controllers, and a single source of truth for the incident timeline. Write playbooks for the incidents you can foresee -- weight exfiltration, ransomware in the orchestration plane, a control-system intrusion, a vendor-account compromise -- and keep them short enough to actually run. Preparation is the interest you earn during the crisis.

Detect is the handoff from the operations center. The center's detections and hunts surface the signal; the response process decides whether it is an incident and at what severity. Severity is a decision, not a feeling, so define it on two axes: impact -- could this stop training, corrupt models, or endanger the physical plant -- and confidence. A high-impact, high-confidence signal declares immediately. Declaration starts the clock, opens the timeline, and pulls in the commander. Under-declaring is the more common and more expensive error; the intrusion that sat as a medium for two weeks is the one that becomes the breach report.

Contain is about stopping the bleeding without destroying the evidence or the plant. There are two containment horizons. Short-term containment isolates the threat now -- quarantine the host, cut the segment, disable the account -- accepting some disruption to halt the spread. Long-term containment stabilizes operations while you plan eradication -- rebuilding clean systems in parallel, tightening segmentation, watching the adversary in a controlled way to learn scope. In AI infrastructure containment carries a physical constraint the IT world never faces: you cannot simply power off the cooling loop or the power train to contain a control-system intrusion without risking equipment or safety. Containment decisions on the operational-technology side must be made with a control-systems engineer and, where safety is in play, with the authority to stop work.

Eradicate removes the adversary and the access. Find every foothold -- the persistence mechanism, the backdoor account, the tampered firmware, the poisoned image in the registry -- and remove all of them, because eradicating nine of ten footholds is eradicating none. This is where scoping discipline pays off: if the investigation understated the blast radius, eradication is partial and the adversary returns through the door you missed. Rebuild from known-good sources. Rotate every credential and key the adversary could have touched. Re-establish trust in the model artifacts before anything trained during the incident window is served.

Recover restores service deliberately, not hopefully. Bring systems back on a plan, verify integrity as they return, and monitor intensively for the adversary's re-entry -- the highest-probability window for a second intrusion is the days after recovery, when the team is exhausted and watching less. Define, in advance, what recovered means for each critical service, and who has the authority to declare it. Recovery on an AI campus means confirming not just that inference is serving, but that the power and cooling that sustain it are stable and under legitimate control.

Learn closes the loop. Within days, run a blameless post-incident review: reconstruct the timeline, find the root cause, and identify what let the incident happen and what let it spread. Every finding becomes an action with an owner and a date -- a new detection, a closed visibility gap, a rewritten playbook. The lifecycle is circular. Learn feeds prepare, and the next incident is met by a better-prepared team. Run the phases in order, respect the physical constraints that make AI infrastructure different, and the incident becomes an event you manage rather than a disaster that manages you.