production-debugging_skill

Overview

This skill helps debug production issues in Kubernetes microservice environments. It guides systematic log analysis, end-to-end request tracing, and identification of common bug patterns like schema mismatches, wrong endpoints, timezone errors, and async database issues. Use it to quickly narrow root causes and verify fixes before redeploying.

How this skill works

The skill inspects pod and sidecar logs, pod status, events, and traces requests across services using kubectl and in-cluster HTTP checks. It provides targeted commands to shell into pods, query Dapr subscriptions, publish test events, and compare observed errors to known patterns. It also offers a reproducible checklist and remediation suggestions to validate fixes and prevent regressions.

When to use it

• Investigating HTTP 500 errors in production
• Feature works locally but fails in production
• Tracing requests across microservices and sidecars
• Finding silent failures with no obvious errors
• Diagnosing service-to-service integration or Dapr pub/sub issues

Best practices

• Reproduce and document the exact symptom before changing production state
• Start from the failing service and expand outward tracing the full request path
• Check both main app and sidecar logs (e.g., daprd) and pod events for lifecycle issues
• Use correlation IDs and structured logs to follow requests across services
• Verify fixes locally or in a staging cluster before deploying to production

Example use cases

• User-facing page returns 500; trace frontend → API → downstream service to locate the failing component
• Scheduled jobs firing at wrong time; inspect logs and compare stored timestamps to UTC to find timezone handling bugs
• Missing notifications: follow API → pub/sub → notification service and inspect raw message format for CloudEvent wrapping issues
• Intermittent AttributeError after a deploy: check model/schema differences and confirm deployed code matches migrations
• Dapr integration failing: check daprd logs, subscriptions, and publish a test event from inside the cluster