monitoring-observability_skill

Overview

This skill provides production-ready monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse LLM tracing, and drift detection. It consolidates RED-method metrics, alerting rules, LLM cost and trace instrumentation, and statistical drift detection into actionable rules and templates. Use it to add reliable health signals, cost visibility, and automatic drift alerts to AI services.

How this skill works

The skill supplies modular rule files that load on demand for four categories: infrastructure monitoring, LLM observability, drift detection, and silent-failure detection. It includes Prometheus metric patterns (counters, histograms, cardinality guidance), Grafana dashboard and SLO templates, Langfuse tracing instrumentation (observe decorator and OTEL spans), and statistical drift checks (PSI, KS, EWMA) with alerting recommendations. Templates and scripts accelerate integration with FastAPI, TypeScript services, and agent runtimes.

When to use it

• When adding RED-method metrics, latency histograms, and error counters to a service.
• When you need end-to-end LLM tracing, token-level cost tracking, or evaluator scoring with Langfuse.
• When you must detect quality or distribution drift in production models and trigger alerts.
• When preventing silent failures like skipped tools, token spikes, or looped agent behavior.
• When building dashboards, SLOs, and escalation-aware alerting to reduce noise and fatigue.

Best practices

• Instrument Rate, Errors, Duration (RED) for all user-facing endpoints and background tasks.
• Limit metric label cardinality and prefer histograms for latency; use counters for idempotent events.
• Use Langfuse @observe and OTEL spans for traceable LLM calls and annotate traces with user/session tags.
• Choose PSI for large-sample drift detection and KS for small-sample sensitivity; combine with EWMA for trends.
• Adopt dynamic thresholds (percentiles) and four severity levels to reduce alert fatigue and enable escalation.

Example use cases

• Add Prometheus counters/histograms to a FastAPI service and export dashboards to Grafana for service SLOs.
• Instrument an agent's LLM calls with Langfuse to correlate cost, latency, and response quality per session.
• Run daily PSI checks between baseline and production outputs and trigger PagerDuty for PSI >= 0.25.
• Detect silent failures by comparing expected vs actual tool calls in traces and alert on repeated skips.
• Create a cost-monitoring alert for token spend per model and notify when 95th-percentile spend exceeds budget.