managing-incidents_skill

Overview

This skill guides incident response from detection through post-mortem using SRE principles, severity classification, on-call management, blameless culture, and structured communication. It provides concrete workflows, role definitions, runbook standards, and metrics to reduce MTTR and improve reliability. Use it to design or improve incident processes, escalation policies, and post-incident learning loops.

How this skill works

The skill defines a standard incident lifecycle (Detection → Triage → Declaration → Mitigation → Resolution → Post-Mortem) and maps decision points for declaring, escalating, and closing incidents. It prescribes role assignments (Incident Commander, Communications Lead, SMEs, Scribe), severity levels (SEV0–SEV3) with response expectations, runbook structure, communication cadences, and tooling recommendations. It also includes templates, scripts, and an interactive severity classifier to operationalize practices.

When to use it

• Setting up or revising incident response processes for a team or product
• Designing on-call rotations, escalation matrices, and handoff procedures
• Creating or hardening runbooks and playbooks for common failure scenarios
• Conducting blameless post-mortems and tracking corrective action items
• Implementing incident communication protocols and status pages

Best practices

• Declare early and often — coordination beats delayed certainty
• Mitigate customer impact first (rollback/disable/failover), root cause later
• Enforce a clear IC who coordinates but does not do hands-on debugging
• Keep post-mortems blameless, timeboxed, and follow up action items in sprints
• Design actionable alerts, audit them regularly, and limit night pages

Example use cases

• Create a SEV classification and escalation policy for a new service
• Build five runbooks (DB failover, cache issues, DDOS mitigation, deploy rollback, auth outages)
• Run a tabletop drill and refine communication cadence and handoff checklist
• Integrate PagerDuty/Opsgenie with Slack and automate status page updates
• Run post-mortems within 48 hours and convert findings into tracked sprint tasks