☸️ Kubernetes Debugging & Troubleshooting Guide

A comprehensive guide for diagnosing and troubleshooting Kubernetes cluster and workload issues.

📋 Table of Contents

Cluster Information
Pods & Containers
Deployments & ReplicaSets
Services & Networking
Persistent Volumes & Storage
Resource Management
Logs & Events
Node Debugging
Network Troubleshooting
Performance & Monitoring
Common Issues

🌐 Cluster Information

Check Cluster Status

# Get cluster info
kubectl cluster-info

# Get cluster version
kubectl version --short

# Get all nodes
kubectl get nodes

# Detailed node information
kubectl get nodes -o wide

# Describe specific node
kubectl describe node <NODE_NAME>

Check Cluster Components

# Check component status
kubectl get componentstatuses

# Check API server health
kubectl get --raw='/readyz?verbose'

# Check all system pods
kubectl get pods -n kube-system

# Check all namespaces
kubectl get namespaces

Context Management

# List all contexts
kubectl config get-contexts

# Current context
kubectl config current-context

# Switch context
kubectl config use-context <CONTEXT_NAME>

# View kubeconfig
kubectl config view

🐳 Pods & Containers

List and Inspect Pods

# Get all pods in current namespace
kubectl get pods

# Get pods in all namespaces
kubectl get pods --all-namespaces
kubectl get pods -A

# Get pods with more details
kubectl get pods -o wide

# Watch pods in real-time
kubectl get pods -w

# Get pods with labels
kubectl get pods --show-labels

# Filter pods by label
kubectl get pods -l app=nginx
kubectl get pods -l environment=production

Pod Details and Debugging

# Describe pod (events, conditions, containers)
kubectl describe pod <POD_NAME>

# Describe pod in specific namespace
kubectl describe pod <POD_NAME> -n <NAMESPACE>

# Get pod YAML
kubectl get pod <POD_NAME> -o yaml

# Get pod JSON
kubectl get pod <POD_NAME> -o json

# Check pod status and restart count
kubectl get pods -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount

Container Logs

# Get logs from pod
kubectl logs <POD_NAME>

# Get logs from specific container in pod
kubectl logs <POD_NAME> -c <CONTAINER_NAME>

# Follow logs (tail -f)
kubectl logs -f <POD_NAME>

# Get previous container logs (after crash)
kubectl logs <POD_NAME> --previous

# Get logs with timestamps
kubectl logs <POD_NAME> --timestamps

# Get last N lines
kubectl logs <POD_NAME> --tail=100

# Get logs since time
kubectl logs <POD_NAME> --since=1h
kubectl logs <POD_NAME> --since=2023-01-01T00:00:00Z

# Get logs from all pods with label
kubectl logs -l app=nginx --all-containers=true

Execute Commands in Containers

# Execute command in pod
kubectl exec <POD_NAME> -- <COMMAND>

# Interactive shell
kubectl exec -it <POD_NAME> -- /bin/bash
kubectl exec -it <POD_NAME> -- /bin/sh

# Execute in specific container
kubectl exec -it <POD_NAME> -c <CONTAINER_NAME> -- /bin/bash

# Run command with namespace
kubectl exec -n <NAMESPACE> -it <POD_NAME> -- /bin/bash

# Copy files to/from pod
kubectl cp <POD_NAME>:/path/to/file ./local/path
kubectl cp ./local/file <POD_NAME>:/path/to/destination

Pod Debugging Tools

# Run debug container in pod
kubectl debug <POD_NAME> -it --image=busybox

# Debug with ephemeral container (K8s 1.23+)
kubectl debug <POD_NAME> -it --image=busybox --target=<CONTAINER_NAME>

# Create debug pod that shares namespaces
kubectl debug <POD_NAME> -it --image=nicolaka/netshoot --share-processes --copy-to=debug-pod

# Port forward to pod
kubectl port-forward <POD_NAME> 8080:80

# Port forward to service
kubectl port-forward svc/<SERVICE_NAME> 8080:80

🚀 Deployments & ReplicaSets

Deployment Management

# Get all deployments
kubectl get deployments
kubectl get deploy

# Describe deployment
kubectl describe deployment <DEPLOYMENT_NAME>

# Get deployment YAML
kubectl get deployment <DEPLOYMENT_NAME> -o yaml

# Check deployment status
kubectl rollout status deployment/<DEPLOYMENT_NAME>

# Get deployment history
kubectl rollout history deployment/<DEPLOYMENT_NAME>

# Get specific revision details
kubectl rollout history deployment/<DEPLOYMENT_NAME> --revision=2

Scaling and Updates

# Scale deployment
kubectl scale deployment <DEPLOYMENT_NAME> --replicas=5

# Autoscaling
kubectl autoscale deployment <DEPLOYMENT_NAME> --min=2 --max=10 --cpu-percent=80

# Check HPA (Horizontal Pod Autoscaler)
kubectl get hpa

# Update deployment image
kubectl set image deployment/<DEPLOYMENT_NAME> <CONTAINER_NAME>=<NEW_IMAGE>

# Edit deployment
kubectl edit deployment <DEPLOYMENT_NAME>

# Rollback deployment
kubectl rollout undo deployment/<DEPLOYMENT_NAME>

# Rollback to specific revision
kubectl rollout undo deployment/<DEPLOYMENT_NAME> --to-revision=2

# Pause/Resume rollout
kubectl rollout pause deployment/<DEPLOYMENT_NAME>
kubectl rollout resume deployment/<DEPLOYMENT_NAME>

# Restart deployment (rolling restart)
kubectl rollout restart deployment/<DEPLOYMENT_NAME>

ReplicaSets

# Get all replicasets
kubectl get replicasets
kubectl get rs

# Describe replicaset
kubectl describe rs <REPLICASET_NAME>

# Get replicaset with deployment info
kubectl get rs -o wide

🌐 Services & Networking

Service Information

# Get all services
kubectl get services
kubectl get svc

# Get services with endpoints
kubectl get svc -o wide

# Describe service
kubectl describe svc <SERVICE_NAME>

# Get service endpoints
kubectl get endpoints <SERVICE_NAME>

# Get all endpoints
kubectl get endpoints

Ingress

# Get all ingresses
kubectl get ingress
kubectl get ing

# Describe ingress
kubectl describe ingress <INGRESS_NAME>

# Get ingress with addresses
kubectl get ingress -o wide

Network Policies

# Get network policies
kubectl get networkpolicies
kubectl get netpol

# Describe network policy
kubectl describe netpol <POLICY_NAME>

DNS Debugging

# Test DNS resolution from pod
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup kubernetes.default

# Test service DNS
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup <SERVICE_NAME>

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns

💾 Persistent Volumes & Storage

Volume Management

# Get persistent volumes
kubectl get pv

# Get persistent volume claims
kubectl get pvc

# Describe PV
kubectl describe pv <PV_NAME>

# Describe PVC
kubectl describe pvc <PVC_NAME>

# Get storage classes
kubectl get storageclass
kubectl get sc

# Describe storage class
kubectl describe sc <STORAGECLASS_NAME>

Volume Troubleshooting

# Check PVC binding status
kubectl get pvc -o wide

# Check which pod is using PVC
kubectl get pods -o json | jq '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="<PVC_NAME>") | .metadata.name'

# Check volume mount in pod
kubectl describe pod <POD_NAME> | grep -A 5 "Mounts:"

📊 Resource Management

Resource Quotas and Limits

# Get resource quotas
kubectl get resourcequota
kubectl get quota

# Describe quota
kubectl describe quota <QUOTA_NAME>

# Get limit ranges
kubectl get limitrange
kubectl get limits

# Describe limit range
kubectl describe limits <LIMITRANGE_NAME>

Resource Usage

# Get node resource usage
kubectl top nodes

# Get pod resource usage
kubectl top pods

# Get pod resource usage in all namespaces
kubectl top pods --all-namespaces

# Get pod resource usage with containers
kubectl top pods --containers

# Sort by CPU or memory
kubectl top pods --sort-by=cpu
kubectl top pods --sort-by=memory

Check Resource Requests/Limits

# Get pod resources
kubectl get pods -o custom-columns=NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,CPU_LIM:.spec.containers[*].resources.limits.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,MEM_LIM:.spec.containers[*].resources.limits.memory

# Describe node capacity
kubectl describe node <NODE_NAME> | grep -A 5 "Allocated resources"

📋 Logs & Events

Event Monitoring

# Get all events
kubectl get events

# Get events sorted by timestamp
kubectl get events --sort-by='.lastTimestamp'

# Get events for specific namespace
kubectl get events -n <NAMESPACE>

# Watch events
kubectl get events -w

# Get events for specific resource
kubectl get events --field-selector involvedObject.name=<POD_NAME>

# Get warning events only
kubectl get events --field-selector type=Warning

# Get events in last hour
kubectl get events --field-selector type!=Normal | grep "([0-9]|[1-5][0-9])m"

Audit Logs

# Check API server audit logs (if enabled)
kubectl logs -n kube-system kube-apiserver-<NODE_NAME>

🖥️ Node Debugging

Node Status

# Get node status
kubectl get nodes

# Describe node
kubectl describe node <NODE_NAME>

# Get node conditions
kubectl get nodes -o custom-columns=NAME:.metadata.name,STATUS:.status.conditions[?(@.type=="Ready")].status

# Check node taints
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

# Check node labels
kubectl get nodes --show-labels

Node Troubleshooting

# Cordon node (mark unschedulable)
kubectl cordon <NODE_NAME>

# Uncordon node
kubectl uncordon <NODE_NAME>

# Drain node (evict pods)
kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data

# Check pods on specific node
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<NODE_NAME>

SSH to Node (if accessible)

# SSH to node
ssh <NODE_IP>

# Check kubelet status
systemctl status kubelet

# Check kubelet logs
journalctl -u kubelet -f

# Check container runtime
systemctl status containerd
systemctl status docker

# Check node resources
df -h
free -h
top

🔍 Network Troubleshooting

Network Testing

# Run network debug pod
kubectl run netshoot --rm -i --tty --image nicolaka/netshoot -- /bin/bash

# Test connectivity to service
kubectl run -it --rm debug --image=busybox --restart=Never -- wget -O- http://<SERVICE_NAME>:<PORT>

# Test DNS
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup <SERVICE_NAME>

# Curl test
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- curl http://<SERVICE_NAME>:<PORT>

# Test TCP connection
kubectl run -it --rm debug --image=busybox --restart=Never -- telnet <SERVICE_NAME> <PORT>

Check Network Plugin

# Check CNI pods (Calico example)
kubectl get pods -n kube-system | grep calico

# Check CNI pods (Flannel example)
kubectl get pods -n kube-system | grep flannel

# Check CNI pods (Weave example)
kubectl get pods -n kube-system | grep weave

📈 Performance & Monitoring

Metrics Server

# Check metrics server
kubectl get deployment metrics-server -n kube-system

# Get metrics server logs
kubectl logs -n kube-system -l k8s-app=metrics-server

Cluster Performance

# Check API server performance
kubectl get --raw /metrics

# Check controller manager
kubectl get pods -n kube-system | grep controller-manager

# Check scheduler
kubectl get pods -n kube-system | grep scheduler

🚨 Common Issues

Issue: Pods Stuck in Pending State

Diagnose:

# Check pod events
kubectl describe pod <POD_NAME>

# Check node resources
kubectl top nodes
kubectl describe node <NODE_NAME>

Common Causes:

Insufficient CPU/Memory on nodes
No nodes match pod’s node selector
PVC not bound
Image pull errors

Solutions:

# Scale cluster or reduce pod resources
# Check node selectors and taints
# Verify PVC status
kubectl get pvc

Issue: CrashLoopBackOff

Diagnose:

# Check pod logs
kubectl logs <POD_NAME> --previous

# Check events
kubectl describe pod <POD_NAME>

# Check liveness/readiness probes
kubectl get pod <POD_NAME> -o yaml | grep -A 10 "livenessProbe\|readinessProbe"

Common Causes:

Application crashes on startup
Failed health checks
Missing dependencies
Configuration errors

Issue: ImagePullBackOff

Diagnose:

# Check events
kubectl describe pod <POD_NAME>

# Check image pull secrets
kubectl get secrets

Solutions:

# Verify image name and tag
# Check image registry authentication
# Create/update image pull secret
kubectl create secret docker-registry regcred --docker-server=<REGISTRY> --docker-username=<USER> --docker-password=<PASSWORD>

Issue: Service Not Accessible

Diagnose:

# Check service endpoints
kubectl get endpoints <SERVICE_NAME>

# Check service selector matches pods
kubectl get pods -l <SELECTOR>

# Test from within cluster
kubectl run -it --rm debug --image=busybox --restart=Never -- wget -O- http://<SERVICE_NAME>:<PORT>

Solutions:

Verify pod labels match service selector
Check pod readiness
Verify service port configuration

Issue: Node Not Ready

Diagnose:

# Check node status
kubectl describe node <NODE_NAME>

# Check kubelet logs (SSH to node)
journalctl -u kubelet -f

Common Causes:

Kubelet not running
Network plugin issues
Disk pressure
Memory pressure

Issue: High Memory/CPU Usage

Diagnose:

# Check pod resources
kubectl top pods --all-namespaces --sort-by=memory
kubectl top pods --all-namespaces --sort-by=cpu

# Check node resources
kubectl top nodes

Solutions:

Scale horizontally with HPA
Increase resource limits
Optimize application
Add more nodes

🛠️ Advanced Debugging

API Server Debugging

# Increase verbosity of kubectl
kubectl get pods -v=8

# Check API server logs
kubectl logs -n kube-system kube-apiserver-<NODE_NAME>

# Check API resources
kubectl api-resources

# Check API versions
kubectl api-versions

RBAC Debugging

# Check if user can perform action
kubectl auth can-i create pods
kubectl auth can-i create pods --as=user@example.com

# List user permissions
kubectl auth can-i --list

# Check role bindings
kubectl get rolebindings
kubectl get clusterrolebindings

# Describe role
kubectl describe role <ROLE_NAME>
kubectl describe clusterrole <CLUSTERROLE_NAME>

Check Certificates

# Check certificate expiration
kubectl get csr

# Check certificate on node (SSH)
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout

📝 Best Practices

Use namespaces to organize resources and apply resource quotas
Set resource requests and limits for all containers
Implement health checks (liveness, readiness, startup probes)
Use labels consistently for better organization and selection
Monitor cluster metrics with Prometheus and Grafana
Enable audit logging for security and compliance
Regular backups of etcd and critical resources
Use GitOps for declarative cluster management
Implement network policies for pod-to-pod communication control
Keep cluster updated with latest stable versions

🔗 Useful Tools

kubectl - Official Kubernetes CLI
k9s - Terminal UI for Kubernetes
lens - Kubernetes IDE
kubectx/kubens - Switch contexts and namespaces easily
stern - Multi-pod log tailing
kubectl-debug - Debug running pods
kube-capacity - Resource capacity analysis
popeye - Cluster sanitizer

📚 Additional Commands

Useful Aliases

# Add to ~/.bashrc or ~/.zshrc
alias k='kubectl'
alias kgp='kubectl get pods'
alias kgd='kubectl get deployments'
alias kgs='kubectl get services'
alias kgn='kubectl get nodes'
alias kdp='kubectl describe pod'
alias kl='kubectl logs'
alias kex='kubectl exec -it'

# Get all resources
alias kga='kubectl get all'

# Watch pods
alias kgpw='kubectl get pods -w'

Quick Debugging Pod

# Create a debug pod
kubectl run debug --rm -i --tty --image=nicolaka/netshoot -- /bin/bash

⚠️ Security Notes

Never expose cluster credentials in logs or configs
Use RBAC for access control
Enable Pod Security Policies/Pod Security Standards
Scan container images for vulnerabilities
Use network policies to restrict traffic
Regularly rotate certificates and secrets
Enable audit logging
Keep cluster components updated