☸️ Kubernetes Debugging & Troubleshooting Guide

A comprehensive guide for diagnosing and troubleshooting Kubernetes cluster and workload issues.

📋 Table of Contents

Cluster Information
Pods & Containers
Deployments & ReplicaSets
Services & Networking
Persistent Volumes & Storage
Resource Management
Logs & Events
Node Debugging
Network Troubleshooting
Performance & Monitoring
Common Issues

🌐 Cluster Information

Start every investigation by confirming you’re on the right cluster and that the control plane is healthy:

kubectl config current-context        # right cluster?
kubectl get --raw='/readyz?verbose'   # API server health
kubectl get nodes -o wide             # all nodes Ready?
kubectl get pods -n kube-system       # control-plane / CNI pods healthy?

Full kubectl command references for cluster info, contexts, and node inspection live in OPSDECK as interactive command builders — this guide keeps only the diagnostic flow.

🐳 Pods & Containers

Pod Triage

The three commands that answer most “why is my pod broken” questions:

kubectl get pods -o wide              # phase, restarts, node placement
kubectl describe pod <POD_NAME>       # events at the bottom name the real cause
kubectl get pod <POD_NAME> -o yaml    # full spec + status conditions

The events from describe surface scheduling failures (Pending), image pull errors, probe failures, and OOM kills — read them before anything else.

Listing/filtering variants (label selectors, custom columns, watch mode, all-namespaces) are available as an interactive kubectl builder in OPSDECK.

Container Logs

# Get logs from pod
kubectl logs <POD_NAME>

# Get logs from specific container in pod
kubectl logs <POD_NAME> -c <CONTAINER_NAME>

# Follow logs (tail -f)
kubectl logs -f <POD_NAME>

# Get previous container logs (after crash)
kubectl logs <POD_NAME> --previous

# Get logs with timestamps
kubectl logs <POD_NAME> --timestamps

# Get last N lines
kubectl logs <POD_NAME> --tail=100

# Get logs since time
kubectl logs <POD_NAME> --since=1h
kubectl logs <POD_NAME> --since=2023-01-01T00:00:00Z

# Get logs from all pods with label
kubectl logs -l app=nginx --all-containers=true

Execute Commands in Containers

# Execute command in pod
kubectl exec <POD_NAME> -- <COMMAND>

# Interactive shell
kubectl exec -it <POD_NAME> -- /bin/bash
kubectl exec -it <POD_NAME> -- /bin/sh

# Execute in specific container
kubectl exec -it <POD_NAME> -c <CONTAINER_NAME> -- /bin/bash

# Run command with namespace
kubectl exec -n <NAMESPACE> -it <POD_NAME> -- /bin/bash

# Copy files to/from pod
kubectl cp <POD_NAME>:/path/to/file ./local/path
kubectl cp ./local/file <POD_NAME>:/path/to/destination

Pod Debugging Tools

# Run debug container in pod
kubectl debug <POD_NAME> -it --image=busybox

# Debug with ephemeral container (K8s 1.23+)
kubectl debug <POD_NAME> -it --image=busybox --target=<CONTAINER_NAME>

# Create debug pod that shares namespaces
kubectl debug <POD_NAME> -it --image=nicolaka/netshoot --share-processes --copy-to=debug-pod

# Port forward to pod
kubectl port-forward <POD_NAME> 8080:80

# Port forward to service
kubectl port-forward svc/<SERVICE_NAME> 8080:80

🚀 Deployments & ReplicaSets

Debugging a Stuck Rollout

A rollout that never completes almost always means new pods can’t become Ready:

kubectl rollout status deployment/<DEPLOYMENT_NAME>   # where is it stuck?
kubectl get rs -o wide                                # is the new ReplicaSet scaling up?
kubectl describe deployment <DEPLOYMENT_NAME>         # conditions: ProgressDeadlineExceeded?
kubectl describe pod <NEW_POD_NAME>                   # why aren't new pods Ready?

Then either fix the new pods (image, probes, resources) or roll back:

kubectl rollout undo deployment/<DEPLOYMENT_NAME>
kubectl rollout history deployment/<DEPLOYMENT_NAME>  # find a known-good revision

Scale, autoscale, set-image, pause/resume, and restart command builders for Deployments live in OPSDECK — build the exact command interactively instead of memorizing flags.

🌐 Services & Networking

Service Debugging Flow

A Service that doesn’t route traffic is almost always an empty endpoints list — the selector doesn’t match any Ready pod:

kubectl get endpoints <SERVICE_NAME>     # empty? selector/readiness problem
kubectl describe svc <SERVICE_NAME>      # note the Selector line
kubectl get pods -l <SELECTOR> -o wide   # do matching pods exist and show READY?
kubectl describe ingress <INGRESS_NAME>  # for ingress: backend service + port correct?
kubectl get netpol                       # a NetworkPolicy silently dropping traffic?

Service/Ingress/NetworkPolicy listing and inspection command references are covered interactively in OPSDECK.

DNS Debugging

# Test DNS resolution from pod
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup kubernetes.default

# Test service DNS
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup <SERVICE_NAME>

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns

💾 Persistent Volumes & Storage

Volume Management

# Get persistent volumes
kubectl get pv

# Get persistent volume claims
kubectl get pvc

# Describe PV
kubectl describe pv <PV_NAME>

# Describe PVC
kubectl describe pvc <PVC_NAME>

# Get storage classes
kubectl get storageclass
kubectl get sc

# Describe storage class
kubectl describe sc <STORAGECLASS_NAME>

Volume Troubleshooting

# Check PVC binding status
kubectl get pvc -o wide

# Check which pod is using PVC
kubectl get pods -o json | jq '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="<PVC_NAME>") | .metadata.name'

# Check volume mount in pod
kubectl describe pod <POD_NAME> | grep -A 5 "Mounts:"

📊 Resource Management

Resource Pressure Triage

kubectl top nodes                                   # node-level saturation
kubectl top pods -A --sort-by=memory | head -15     # who is eating the cluster
kubectl describe quota -A                           # quota exhausted? (pods stuck Pending with "exceeded quota")

Quota errors appear in the ReplicaSet’s events (kubectl describe rs), not the Deployment’s — a classic hiding spot.

Full `kubectl top` / quota / limitrange command variants are available in OPSDECK.

Check Resource Requests/Limits

# Get pod resources
kubectl get pods -o custom-columns=NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,CPU_LIM:.spec.containers[*].resources.limits.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,MEM_LIM:.spec.containers[*].resources.limits.memory

# Describe node capacity
kubectl describe node <NODE_NAME> | grep -A 5 "Allocated resources"

📋 Logs & Events

Event Monitoring

# Get all events
kubectl get events

# Get events sorted by timestamp
kubectl get events --sort-by='.lastTimestamp'

# Get events for specific namespace
kubectl get events -n <NAMESPACE>

# Watch events
kubectl get events -w

# Get events for specific resource
kubectl get events --field-selector involvedObject.name=<POD_NAME>

# Get warning events only
kubectl get events --field-selector type=Warning

# Get events in last hour
kubectl get events --field-selector type!=Normal | grep "([0-9]|[1-5][0-9])m"

Audit Logs

# Check API server audit logs (if enabled)
kubectl logs -n kube-system kube-apiserver-<NODE_NAME>

🖥️ Node Debugging

Node Status

# Get node status
kubectl get nodes

# Describe node
kubectl describe node <NODE_NAME>

# Get node conditions
kubectl get nodes -o custom-columns=NAME:.metadata.name,STATUS:.status.conditions[?(@.type=="Ready")].status

# Check node taints
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

# Check node labels
kubectl get nodes --show-labels

Node Troubleshooting

# Cordon node (mark unschedulable)
kubectl cordon <NODE_NAME>

# Uncordon node
kubectl uncordon <NODE_NAME>

# Drain node (evict pods)
kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data

# Check pods on specific node
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<NODE_NAME>

SSH to Node (if accessible)

# SSH to node
ssh <NODE_IP>

# Check kubelet status
systemctl status kubelet

# Check kubelet logs
journalctl -u kubelet -f

# Check container runtime
systemctl status containerd
systemctl status docker

# Check node resources
df -h
free -h
top

🔍 Network Troubleshooting

Network Testing

# Run network debug pod
kubectl run netshoot --rm -i --tty --image nicolaka/netshoot -- /bin/bash

# Test connectivity to service
kubectl run -it --rm debug --image=busybox --restart=Never -- wget -O- http://<SERVICE_NAME>:<PORT>

# Test DNS
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup <SERVICE_NAME>

# Curl test
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- curl http://<SERVICE_NAME>:<PORT>

# Test TCP connection
kubectl run -it --rm debug --image=busybox --restart=Never -- telnet <SERVICE_NAME> <PORT>

Check Network Plugin

# Check CNI pods (Calico example)
kubectl get pods -n kube-system | grep calico

# Check CNI pods (Flannel example)
kubectl get pods -n kube-system | grep flannel

# Check CNI pods (Weave example)
kubectl get pods -n kube-system | grep weave

📈 Performance & Monitoring

Metrics Server

# Check metrics server
kubectl get deployment metrics-server -n kube-system

# Get metrics server logs
kubectl logs -n kube-system -l k8s-app=metrics-server

Cluster Performance

# Check API server performance
kubectl get --raw /metrics

# Check controller manager
kubectl get pods -n kube-system | grep controller-manager

# Check scheduler
kubectl get pods -n kube-system | grep scheduler

🚨 Common Issues

Issue: Pods Stuck in Pending State

Diagnose:

# Check pod events
kubectl describe pod <POD_NAME>

# Check node resources
kubectl top nodes
kubectl describe node <NODE_NAME>

Common Causes:

Insufficient CPU/Memory on nodes
No nodes match pod’s node selector
PVC not bound
Image pull errors

Solutions:

# Scale cluster or reduce pod resources
# Check node selectors and taints
# Verify PVC status
kubectl get pvc

Issue: CrashLoopBackOff

Diagnose:

# Check pod logs
kubectl logs <POD_NAME> --previous

# Check events
kubectl describe pod <POD_NAME>

# Check liveness/readiness probes
kubectl get pod <POD_NAME> -o yaml | grep -A 10 "livenessProbe\|readinessProbe"

Common Causes:

Application crashes on startup
Failed health checks
Missing dependencies
Configuration errors

Issue: ImagePullBackOff

Diagnose:

# Check events
kubectl describe pod <POD_NAME>

# Check image pull secrets
kubectl get secrets

Solutions:

# Verify image name and tag
# Check image registry authentication
# Create/update image pull secret
kubectl create secret docker-registry regcred --docker-server=<REGISTRY> --docker-username=<USER> --docker-password=<PASSWORD>

Issue: Service Not Accessible

Diagnose:

# Check service endpoints
kubectl get endpoints <SERVICE_NAME>

# Check service selector matches pods
kubectl get pods -l <SELECTOR>

# Test from within cluster
kubectl run -it --rm debug --image=busybox --restart=Never -- wget -O- http://<SERVICE_NAME>:<PORT>

Solutions:

Verify pod labels match service selector
Check pod readiness
Verify service port configuration

Issue: Node Not Ready

Diagnose:

# Check node status
kubectl describe node <NODE_NAME>

# Check kubelet logs (SSH to node)
journalctl -u kubelet -f

Common Causes:

Kubelet not running
Network plugin issues
Disk pressure
Memory pressure

Issue: High Memory/CPU Usage

Diagnose:

# Check pod resources
kubectl top pods --all-namespaces --sort-by=memory
kubectl top pods --all-namespaces --sort-by=cpu

# Check node resources
kubectl top nodes

Solutions:

Scale horizontally with HPA
Increase resource limits
Optimize application
Add more nodes

🛠️ Advanced Debugging

API Server Debugging

# Increase verbosity of kubectl
kubectl get pods -v=8

# Check API server logs
kubectl logs -n kube-system kube-apiserver-<NODE_NAME>

# Check API resources
kubectl api-resources

# Check API versions
kubectl api-versions

RBAC Debugging

# Check if user can perform action
kubectl auth can-i create pods
kubectl auth can-i create pods --as=user@example.com

# List user permissions
kubectl auth can-i --list

# Check role bindings
kubectl get rolebindings
kubectl get clusterrolebindings

# Describe role
kubectl describe role <ROLE_NAME>
kubectl describe clusterrole <CLUSTERROLE_NAME>

Check Certificates

# Check certificate expiration
kubectl get csr

# Check certificate on node (SSH)
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout

📝 Best Practices

Use namespaces to organize resources and apply resource quotas
Set resource requests and limits for all containers
Implement health checks (liveness, readiness, startup probes)
Use labels consistently for better organization and selection
Monitor cluster metrics with Prometheus and Grafana
Enable audit logging for security and compliance
Regular backups of etcd and critical resources
Use GitOps for declarative cluster management
Implement network policies for pod-to-pod communication control
Keep cluster updated with latest stable versions

🔗 Useful Tools

kubectl - Official Kubernetes CLI
k9s - Terminal UI for Kubernetes
lens - Kubernetes IDE
kubectx/kubens - Switch contexts and namespaces easily
stern - Multi-pod log tailing
kubectl-debug - Debug running pods
kube-capacity - Resource capacity analysis
popeye - Cluster sanitizer

📚 Additional Commands

Quick Debugging Pod

# Create a debug pod
kubectl run debug --rm -i --tty --image=nicolaka/netshoot -- /bin/bash

Shell aliases and general kubectl cheat-sheet material moved to OPSDECK, which generates commands interactively.

⚠️ Security Notes

Never expose cluster credentials in logs or configs
Use RBAC for access control
Enable Pod Security Policies/Pod Security Standards
Scan container images for vulnerabilities
Use network policies to restrict traffic
Regularly rotate certificates and secrets
Enable audit logging
Keep cluster components updated