nik-kale/sre-skills
Overview
This skill provides systematic debugging workflows for common Kubernetes problems like pod failures, resource exhaustion, image pull errors, and networking issues. It gives a concise set of diagnostic commands, targeted troubleshooting steps, and practical fixes to accelerate incident resolution. Use it to triage CrashLoopBackOff, OOMKilled, ImagePullBackOff, readiness failures, scheduling problems, and service connectivity issues.
How this skill works
The skill walks through a layered diagnostic approach: cluster-level checks, pod inspection, logs and exit codes, probe validation, and resource analysis. It maps common symptoms to likely causes and prescribes concrete kubectl commands and configuration fixes (resources, probes, imagePullSecrets, node affinity/taints). It also includes quick commands for interactive debugging and network/DNS tests from within the cluster.
When to use it
- Pod stuck in CrashLoopBackOff or continuously restarting
- Pods terminated with OOMKilled or unexpected exit codes
- ImagePullBackOff or ErrImagePull errors
- Pod stuck Pending or unschedulable due to resource/taint/affinity issues
- Service unreachable, DNS failures, or endpoints not matching selectors
- Cluster resource pressure or PVC/PV provisioning problems
Best practices
- Start with kubectl get pods -A and kubectl get events -n <ns> to capture recent failures
- Examine current and previous container logs (kubectl logs --previous) and describe output for events and exit codes
- Validate readiness/liveness probes and increase initialDelay/timeout before changing app code
- Use resource requests/limits conservatively and profile memory/cpu with kubectl top before raising limits
- Create and reference imagePullSecrets for private registries; verify image tags and registry reachability
- Debug networking by exec-ing into a debug pod and testing DNS and service endpoints
Example use cases
- Triage a CrashLoopBackOff: check kubectl logs, previous logs, and container Last State to find startup errors or OOMs
- Resolve ImagePullBackOff: verify image name/tag, check imagePullSecrets, and confirm registry connectivity
- Fix Pending pods: inspect node capacity, taints, and pod affinity/requests to decide whether to add nodes or adjust requests
- Diagnose service reachability: confirm service selector matches pod labels, check endpoints, and test from a debug pod
- Investigate OOMKilled: run kubectl top, increase memory limits if appropriate, and profile application memory usage
FAQ
Start with kubectl get pods -n <ns>, kubectl describe pod <pod> -n <ns>, and kubectl logs <pod> -n <ns> (add --previous if restarting). Then check events with kubectl get events -n <ns> --sort-by='.lastTimestamp'.
How do I know if an OOMKilled was caused by the app or limits?
Check the pod describe output for OOM messages and the container exit code 137. Use kubectl top pod <pod> -n <ns> --containers to observe peak memory. If usage exceeds the limit, increase limits or profile the app for leaks.
2 skills
This skill helps you debug Kubernetes issues quickly by guiding you through pod failures, resource constraints, and networking problems with actionable steps.
This skill guides systematic production incident investigation from triage through root cause analysis, improving outage response and post-incident learning.
