observability-design_skill

Overview

This skill provides practical patterns and guidance for designing observability systems: monitoring strategies, distributed tracing, SLI/SLO design, alerting patterns, and dashboard best practices. It focuses on production readiness across the three pillars of observability—metrics, logs, and traces—so teams can detect, diagnose, and learn from incidents. Use it to design measurable reliability targets and effective operational procedures.

How this skill works

The skill inspects service behavior and telemetry needs, then recommends instrumentation and measurement approaches for metrics, logs, and traces. It defines SLI/SLO error-budget workflows, alerting strategies (symptom-based and burn-rate alerts), and dashboard hierarchies for triage and deep-dive. It also prescribes runbook and incident-response practices to turn alerts into reliable actions and learning opportunities.

When to use it

• Designing monitoring infrastructure for a new or refactored service
• Defining SLIs, SLOs, and error budgets to set reliability targets
• Implementing distributed tracing to understand request causality
• Creating alert rules that reduce noise and point to action
• Building dashboards for operations, engineering, and business stakeholders

Best practices

• Correlate metrics, logs, and traces using shared identifiers (trace_id, request_id)
• Instrument at service boundaries and collect user-centric SLIs (availability, latency, error rate)
• Prefer symptom-based alerts and use multi-window burn-rate detection for fast and slow failures
• Structure logs (JSON) with required fields and retention policies tied to value
• Design dashboards to answer specific questions and include deploy/incident context
• Run regular game days, test alerts in staging, and document SLOs and runbooks

Example use cases

• Set a 30-day SLO for request latency (p95 < 200ms) with a defined error budget and deployment policy
• Create multi-window alerts: fast burn (1 hour) and slow burn (3 days) to capture severe and gradual degradations
• Build a service-health dashboard showing SLO status, error budget, and top-level business metrics for on-call triage
• Instrument distributed tracing to identify a latency hotspot across microservice spans during peak load
• Establish an alert review cadence to prune noisy alerts and update runbooks after postmortems