metrics-monitoring_skill

Overview

This skill implements application metrics (RED, USE, Four Golden Signals), builds monitoring dashboards, and configures alerting strategies so teams detect and resolve issues before users are impacted. It provides instrumentation patterns, collection choices (Prometheus, StatsD, CloudWatch, DataDog), alert rules, and dashboard templates for production services. The output is practical code snippets, alert rules, and best-practice guidance for reliable observability.

How this skill works

The skill instruments services with counters, gauges, histograms, and summaries to capture rate, errors, duration, utilization, and saturation. It wires those metrics into a collection backend (e.g., Prometheus) and deploys dashboards and alert rules (Grafana + Prometheus alerts or equivalent). It also includes patterns like automatic decorators, background resource polling, and example alert expressions for error rate, latency, and resource saturation.

When to use it

• Deploying services to production
• Detecting incidents and enabling fast response
• Tracking SLAs/SLOs and business KPIs
• Capacity planning and performance tuning
• Establishing on-call alerting and runbooks

Best practices

• Use RED for request-driven services and USE for resources
• Record percentiles (p95/p99) for latency rather than averages
• Limit label cardinality; avoid user IDs and timestamps
• Alert on symptoms (error rate, latency, saturation), not on root causes
• Define SLOs and alert on SLO burn or violations
• Create runbooks and test alerts regularly to avoid alert fatigue

Example use cases

• Instrument a Flask API with Prometheus metrics and expose /metrics for scraping
• Create Grafana dashboards: request rate, error rate, p95 latency, and connection pool gauges
• Define Prometheus alert rules: high error rate, P95 latency breach, CPU > 80%, DB pool near limit
• Use a decorator to auto-instrument important functions (calls, errors, duration)
• Track business metrics (orders, revenue) alongside technical metrics to correlate customer impact