monitoring_skill

Overview

This skill provides a focused monitoring and observability toolkit built around Prometheus, Grafana, the ELK Stack, and distributed tracing. It helps teams instrument services, build dashboards, define SLIs/SLOs, and set robust alerting and recovery procedures. The goal is to make metrics, logs, and traces actionable for faster incident resolution and reliable SLAs.

How this skill works

The skill inspects and documents common observability setups: Prometheus for metrics and PromQL, Grafana for visualizations, ELK for log aggregation, and tracing tools for request flows. It includes practical queries, alert examples, troubleshooting checklists, and recovery steps for typical failures like OOMs or missing scrapes. It also covers SRE concepts such as SLIs, SLOs, error budgets, and strategies for high-cardinality and performance tuning.

When to use it

• When you need to instrument services and establish core observability (metrics, logs, traces).
• When creating Grafana dashboards and PromQL alerts for production services.
• When troubleshooting missing metrics, slow queries, or alert misfires.
• When defining SLIs, SLOs, and alerting thresholds tied to business outcomes.
• When you need guidance on scaling Prometheus (cardinality, federation, retention).

Best practices

• Start with the SRE golden signals: latency, traffic, errors, saturation.
• Keep Prometheus label cardinality low; avoid high-cardinality dynamic labels.
• Use recording rules and aggregation to improve query performance.
• Define clear SLIs and SLOs first, then derive alerting rules and error budgets.
• Correlate logs and traces with metric alerts to reduce time-to-detect and time-to-restore.

Example use cases

• Create a 99th-percentile latency dashboard using histogram_quantile and Grafana panels.
• Write alerting rules for error rate increases and wire them into Alertmanager silences.
• Investigate missing metrics by checking /targets, Prometheus logs, and NTP time sync.
• Mitigate Prometheus OOM by diagnosing cardinality, reducing retention, or adding federation.
• Implement end-to-end request tracing with OpenTelemetry and correlate traces to slow metrics.