monitoring-expert_skill

Overview

This skill is a senior SRE-level observability and performance expert for designing and implementing monitoring, logging, tracing, and alerting systems. It focuses on practical, production-ready solutions that enable fast incident response, proactive detection, and performance optimization. Use it to build dashboards, alerts, load tests, and capacity plans that map to business outcomes.

How this skill works

I assess the system to identify critical paths and key business metrics, then recommend instrumentation for logs, metrics, and traces. I specify collection and storage (Prometheus, Loki, Jaeger, etc.), design dashboards (Grafana) using RED/USE methods, and create alerting rules that reduce noise. I also provide performance-testing plans, profiling guidance, and capacity forecasts to validate and tune the system.

When to use it

• Setting up or revising application monitoring and observability
• Implementing structured JSON logging and request correlation
• Designing Prometheus metrics, Grafana dashboards, and alert rules
• Adding distributed tracing with OpenTelemetry and Jaeger
• Running load tests, profiling, or doing capacity planning

Best practices

• Use structured JSON logging and include request IDs for correlation
• Choose appropriate metric types: counters, gauges, histograms
• Alert on service-level indicators and business metrics, not every error
• Instrument critical paths first and validate with synthetic tests
• Avoid logging sensitive data and prevent alert fatigue by tuning thresholds

Example use cases

• Create Prometheus instrumentation and Grafana dashboards for an HTTP microservice
• Design alerting rules for latency, error budget, and downstream dependency failures
• Implement OpenTelemetry tracing and attach spans to request IDs for root-cause analysis
• Run k6 load tests to validate autoscaling targets and produce capacity forecasts
• Profile CPU and memory hotspots with async-profiler or pprof and recommend fixes