mend_skill

Overview

This skill automates remediation for known failure patterns by matching incident symptoms to a catalog of tested fixes, executing actions within safety tiers, and verifying recovery through staged checks. It handles operational fixes only (restarts, config adjustments, scaling, rollbacks) and never changes application business logic. Safety gates, rollback plans, and logging are enforced for every remediation. The agent also extracts new patterns from postmortems to improve the catalog over time.

How this skill works

Mend receives diagnoses from Triage or alerts from Beacon, extracts symptoms, and performs a pattern lookup to determine a match and safety tier. For permitted tiers it executes runbook steps with guardrails: validate prerequisites, checkpoint state, run actions, and perform a four-stage verification loop (immediate health, 30s smoke, 5min SLO, 10min recovery). If verification fails, Mend triggers an automated rollback and escalates as needed. All actions are logged and reported to Radar, Beacon, Builder, Gear, and Triage depending on outcome.

When to use it

• When an incident matches a documented remediation pattern (high-confidence, operational fix).
• To auto-restart services, clear caches, scale resources, or apply reversible config fixes.
• When fast, safe recovery is required and the action falls within T1–T2 safety tiers.
• To execute Triage-provided runbooks with enforced guardrails and rollback checkpoints.
• When you want post-incident pattern extraction and catalog updates from postmortems.

Best practices

• Ensure Triage provides clear diagnosis and runbooks before invoking automated remediation.
• Verify pattern match confidence ≥ 50% and prefer AUTO-REMEDIATE for ≥ 90% matches.
• Classify every action into safety tiers (T1–T4) and require explicit approval for T3.
• Use the staged verification loop and abort on unexpected side effects or failed checks.
• Include a rollback step and checkpoint state after each runbook step for rapid recovery.

Example use cases

• Beacon detects increased error rate for a service; Mend matches INFRA pattern and restarts pods, then verifies SLO recovery.
• Triage provides a runbook for cache corruption; Mend runs cache flush steps, performs smoke tests, and logs outcome.
• Automatic horizontal scaling when resource thresholds are reached and the action is classified T2 (notify-and-fix).
• Failed remediation triggers Gear rollback: Mend reverts infra changes and escalates remaining issues to Builder/Triage.
• Postmortem identifies a new symptom cluster; Mend extracts a new pattern and adds it to the catalog.