observability_skill

Overview

This skill shows how to instrument Go services using OpenTelemetry for distributed tracing, Prometheus for metrics, and Go's slog for structured logging. It provides concrete middleware patterns, exporter setup, metric types, and context-aware logging examples to correlate traces, metrics, and logs. The material emphasizes production-ready concerns like health checks and graceful shutdown.

How this skill works

It configures an OpenTelemetry TracerProvider with a Jaeger exporter and provides examples for creating and propagating spans in application code and HTTP middleware. It defines Prometheus metric types (counters, gauges, histograms, summaries) and a middleware to record request duration, status, and active connections. It also shows using slog to emit JSON-structured logs that include trace and span identifiers, plus health and shutdown handlers for safe lifecycle management.

When to use it

• You are instrumenting microservices for production observability.
• You need distributed tracing across services (request flow and latency hotspots).
• You want Prometheus metrics exposed via /metrics for scraping and dashboards.
• You need structured, correlated logs that include trace IDs and span IDs.
• You require readiness/liveness probes and graceful shutdown for Kubernetes deployments.

Best practices

• Always propagate context when starting spans; prefer automatic middleware and manual spans for business logic.
• Use probability sampling in high-traffic production; avoid AlwaysSample except for debugging.
• Label metrics consistently (method, path, status) and avoid high-cardinality labels on frequently changing values.
• Attach trace_id and span_id to logs using context-aware loggers to enable full triage across pillars.
• Ensure exporters and background processors are cleanly shut down on SIGTERM to avoid losing telemetry data.

Example use cases

• Instrument an HTTP API with otelhttp middleware and Prometheus metrics to measure latency and error rates.
• Add span creation in business functions (e.g., ProcessOrder) to trace downstream calls and database queries.
• Emit JSON logs via slog that include trace_id/span_id so logs can be filtered by trace in observability tools.
• Expose /metrics and /health endpoints for Prometheus scraping and Kubernetes probes during rolling updates.
• Use histograms for request duration buckets and summaries for database query quantiles to drive SLOs.