observability-monitoring_skill

Overview

This skill provides a practical framework for implementing observability across services using structured logging, metrics, distributed tracing, and alerting strategies. It bundles concrete patterns, telemetry templates, and checklist-driven guidance to get production monitoring working reliably. The focus is on actionable implementations in TypeScript ecosystems.

How this skill works

The skill inspects and codifies best practices for logs, metrics, and traces: structured JSON logging with context, RED metrics and Prometheus histogram buckets, OpenTelemetry tracing with auto-instrumentation and manual spans, and alerting rules for key incidents. It includes health check patterns and templates for wiring these concerns into Node/Express services and Kubernetes probes.

When to use it

• Setting up or standardizing application monitoring for new or existing services
• Adding structured logging and request correlation to improve diagnostics
• Instrumenting services with RED metrics and Prometheus-compatible histograms
• Enabling distributed tracing with OpenTelemetry for cross-service latency analysis
• Defining alerting rules and severity levels for production incidents
• Creating health checks and readiness/liveness probes for Kubernetes deployments

Best practices

• Log as structured JSON with contextual fields (userId, requestId, duration_ms) rather than string interpolation
• Capture RED metrics (Rate, Errors, Duration) plus key business metrics and use appropriate Prometheus buckets
• Auto-instrument standard libraries and add manual spans around business-critical operations
• Attach correlation IDs to requests and propagate them through logs, metrics, and traces
• Tune alerts by severity to reduce noise: start with critical service-down and high error-rate rules, then iterate

Example use cases

• Implement a Winston-based structured logger and request middleware that emits JSON logs with correlation IDs
• Configure Prometheus metrics: request counter, error counter, and latency histogram with recommended buckets
• Integrate OpenTelemetry to auto-instrument Express, PostgreSQL, and Redis, and add manual spans for payment processing
• Create Prometheus alerting rules for ServiceDown, HighErrorRate, and HighLatency with escalation severities
• Add Kubernetes liveness, readiness, and startup probes and expose a health endpoint that reports dependency statuses