incident-response_skill

Overview

This skill guides a structured, repeatable approach to investigating and production incidents, from initial triage to post-incident review. It focuses on fast impact reduction, clear data collection, and a blameless root cause analysis to prevent recurrence. Use it to coordinate responders, gather evidence, and drive timely mitigations during outages or degradations.

How this skill works

Start with a rapid triage checklist to confirm the incident, identify affected services, and set severity. Collect key telemetry (metrics, logs, traces), construct a timeline, and correlate recent changes. Apply quick mitigation options to stop the bleeding, then perform a root cause analysis only after stability is restored. Complete resolution, handoff, and a blameless post-incident review with concrete action items.

When to use it

• Alerts firing or monitoring thresholds breached in production
• User-reported outages, errors, or service degradation
• On-call escalation or paged incidents
• Spike in error rates, latency, or resource exhaustion
• After a deployment or config change that coincides with failures

Best practices

• Confirm incidents quickly and avoid chasing false positives
• Prioritize mitigation (restore service) before deep debugging
• Collect and preserve evidence: dashboards, logs, traces, commands
• Use severity levels to drive communication cadence and staffing
• Keep updates concise and regular for stakeholders during high-severity incidents

Example use cases

• SEV1: Total service outage after a production deployment — use rollback or feature-flag disable and run a post-incident review
• SEV2: Significant latency and 5xx spike in a critical API — gather traces, scale or restart pods, and monitor recovery
• SEV3: Partial degradation affecting a region — failover, verify dependency health, and schedule follow-up remediation
• Investigating recurring errors tied to a recent config change — build a timeline and correlate change audits
• On-call handoff mid-incident — use the handoff template to summarize status, actions, and next steps