πŸŸ₯ Redis Debugging & Troubleshooting Guide

← Back to Home Kafka β†’ NGINX β†’ PostgreSQL β†’

A comprehensive guide for diagnosing and troubleshooting Redis performance, memory, persistence, and replication issues.

πŸ“‹ Table of Contents


🩺 Server Health & Info

Basic Health Checks

# Ping the server
redis-cli ping

# Full server info
redis-cli info

# Specific info sections
redis-cli info server
redis-cli info memory
redis-cli info stats
redis-cli info replication
redis-cli info persistence
redis-cli info clients

# Server uptime and version
redis-cli info server | grep -E 'redis_version|uptime_in_days'

# Real-time stats (ops/sec, memory, connections)
redis-cli --stat

Key Metrics to Watch

# Hit ratio β€” below ~0.9 for a cache workload means keys evict/expire too fast
redis-cli info stats | grep -E 'keyspace_hits|keyspace_misses'

# Evicted and expired keys
redis-cli info stats | grep -E 'evicted_keys|expired_keys'

# Rejected connections (maxclients reached)
redis-cli info stats | grep rejected_connections

# Instantaneous ops per second
redis-cli info stats | grep instantaneous_ops_per_sec

🧠 Memory Analysis

Memory Overview

# Human-readable memory report with recommendations
redis-cli memory doctor

# Detailed memory stats
redis-cli memory stats

# Key memory numbers
redis-cli info memory | grep -E 'used_memory_human|used_memory_rss_human|used_memory_peak_human|maxmemory_human|mem_fragmentation_ratio|maxmemory_policy'

Interpreting mem_fragmentation_ratio:

Value Meaning
~1.0 – 1.5 Healthy
> 1.5 Fragmentation β€” RSS much larger than logical usage; consider activedefrag yes or a restart
< 1.0 Redis is swapping β€” usually a severe problem, add RAM or lower maxmemory

Finding Big Keys

# Sample the keyspace for the biggest key per type (safe, uses SCAN)
redis-cli --bigkeys

# Sample for keys with most memory usage (Redis 6+)
redis-cli --memkeys

# Memory used by one key
redis-cli memory usage <KEY_NAME>

# Memory usage sampling nested elements
redis-cli memory usage <KEY_NAME> samples 0

Eviction Policy

# Check current policy
redis-cli config get maxmemory-policy
redis-cli config get maxmemory

# Set policy at runtime (persist in redis.conf too)
redis-cli config set maxmemory-policy allkeys-lru

Policy cheat sheet: noeviction (writes fail when full β€” default), allkeys-lru (classic cache), volatile-lru (only keys with TTL), allkeys-lfu (frequency-based, better for skewed access).


🐌 Slow Queries & Latency

Slowlog

# Show slow commands (default threshold 10ms)
redis-cli slowlog get 25

# How many slow entries recorded
redis-cli slowlog len

# Reset the slowlog
redis-cli slowlog reset

# Lower the threshold to 5ms (microseconds!)
redis-cli config set slowlog-log-slower-than 5000

Latency Monitoring

# Built-in latency doctor (needs latency-monitor-threshold set)
redis-cli config set latency-monitor-threshold 100
redis-cli latency doctor

# Latency history per event
redis-cli latency history command

# Measure round-trip latency from a client
redis-cli --latency
redis-cli --latency-history

# Intrinsic latency of the host itself (run ON the server)
redis-cli --intrinsic-latency 30

Live Command Stream

# Watch every command in real time β€” EXPENSIVE, use briefly in production
redis-cli monitor

# Per-command call counts and average time
redis-cli info commandstats

πŸ”‘ Keyspace Inspection

# Keys per database
redis-cli info keyspace

# NEVER run KEYS * in production β€” O(N), blocks the server
# Use SCAN instead (cursor-based, non-blocking)
redis-cli --scan --pattern 'session:*'

# Count matching keys without listing them
redis-cli --scan --pattern 'cache:*' | wc -l

# Inspect a key
redis-cli type <KEY_NAME>
redis-cli ttl <KEY_NAME>
redis-cli object encoding <KEY_NAME>

# How stale is a key (seconds since last access)
redis-cli object idletime <KEY_NAME>

πŸ’Ύ Persistence (RDB/AOF)

Check Persistence Status

# Persistence overview
redis-cli info persistence

# Critical fields
redis-cli info persistence | grep -E 'rdb_last_bgsave_status|rdb_last_save_time|aof_enabled|aof_last_write_status|aof_last_bgrewrite_status'

# When was the last successful save
redis-cli lastsave

RDB Snapshots

# Trigger a background snapshot
redis-cli bgsave

# Check snapshot config
redis-cli config get save
redis-cli config get dir
redis-cli config get dbfilename

# Verify an RDB file offline
redis-check-rdb /var/lib/redis/dump.rdb

AOF

# Check AOF status and rewrite
redis-cli config get appendonly
redis-cli bgrewriteaof

# Repair a truncated/corrupt AOF file (server must be stopped)
redis-check-aof --fix /var/lib/redis/appendonly.aof

If rdb_last_bgsave_status is err and stop-writes-on-bgsave-error is yes (the default), Redis rejects all writes with MISCONF. Fix the disk/fork problem β€” don’t just flip the flag.


πŸ” Replication

Replication Status

# On the primary: role, connected replicas, replication offsets
redis-cli info replication

# Check replica lag: compare master_repl_offset (primary)
# with slave_repl_offset (replica)
redis-cli -h <REPLICA_HOST> info replication | grep -E 'master_link_status|slave_repl_offset|master_last_io_seconds_ago'

Diagnosing Replication Problems

# Replica can't sync? Check backlog size β€” too small forces full resyncs
redis-cli config get repl-backlog-size

# Count of full/partial syncs β€” rising sync_full means repeated full resyncs
redis-cli info stats | grep -E 'sync_full|sync_partial_ok|sync_partial_err'

# Watch primary log for fork/COW memory during full sync
tail -f /var/log/redis/redis-server.log

Common causes of broken replication: network flaps + small repl-backlog-size (full resync loops), replica slower disk during RDB load, client-output-buffer-limit replica too small on the primary (replica disconnected mid-sync β€” raise it).


πŸ•ΈοΈ Cluster & Sentinel

Cluster

# Cluster health β€” state must be "ok", all 16384 slots assigned
redis-cli cluster info

# Node list, roles, slot ranges
redis-cli cluster nodes

# End-to-end check with slot coverage
redis-cli --cluster check <HOST>:6379

# Find which node owns a key's slot
redis-cli cluster keyslot <KEY_NAME>

# Fix slot coverage problems
redis-cli --cluster fix <HOST>:6379

Sentinel

# Ask sentinel about the primary
redis-cli -p 26379 sentinel master <MASTER_NAME>
redis-cli -p 26379 sentinel get-master-addr-by-name <MASTER_NAME>

# List replicas and other sentinels
redis-cli -p 26379 sentinel replicas <MASTER_NAME>
redis-cli -p 26379 sentinel sentinels <MASTER_NAME>

# Force a failover (testing)
redis-cli -p 26379 sentinel failover <MASTER_NAME>

πŸ”Œ Client Connections

# Connected client count vs limit
redis-cli info clients
redis-cli config get maxclients

# List every client with age, idle time, last command
redis-cli client list

# Find idle connections (idle > 300s)
redis-cli client list | awk '{for(i=1;i<=NF;i++) if($i ~ /^idle=/){split($i,a,"="); if(a[2]>300) print $0}}'

# Kill a misbehaving client
redis-cli client kill id <CLIENT_ID>
redis-cli client kill addr <IP>:<PORT>

# Check output buffer pressure (omem = output buffer memory)
redis-cli client list | grep -o 'omem=[0-9]*' | sort -t= -k2 -rn | head

🚨 Common Issues

Issue: OOM / β€œcommand not allowed when used memory > maxmemory”

Diagnose:

redis-cli info memory | grep -E 'used_memory_human|maxmemory_human|maxmemory_policy'
redis-cli info stats | grep evicted_keys
redis-cli --bigkeys

Common Causes:

Solutions:

Issue: MISCONF β€” Redis refuses writes

Diagnose:

redis-cli info persistence | grep rdb_last_bgsave_status
df -h                       # disk full?
dmesg | grep -i oom         # fork killed?
sysctl vm.overcommit_memory # should be 1

Solutions:

Issue: Latency Spikes

Diagnose:

redis-cli slowlog get 25
redis-cli latency doctor
redis-cli info commandstats | sort -t= -k3 -rn | head

Common Causes:

Solutions:

Issue: Mass Key Expiry Storm

Diagnose:

redis-cli info stats | grep expired_keys   # sudden spike

Solutions:

Issue: Replica Keeps Doing Full Resyncs

Diagnose:

redis-cli info stats | grep sync_full
redis-cli config get repl-backlog-size

Solutions:


πŸ“ Best Practices

  1. Set maxmemory and an explicit eviction policy β€” never rely on defaults for cache workloads
  2. Always use SCAN family, never KEYS, in application code
  3. Put TTLs on cache keys and jitter them
  4. Keep keys small β€” a hash of 100 fields beats 100 string keys, but a 50MB hash beats nothing
  5. Disable THP on Redis hosts and set vm.overcommit_memory=1
  6. Monitor mem_fragmentation_ratio, hit ratio, and rejected_connections
  7. Use UNLINK and FLUSHDB ASYNC for large deletions
  8. Test failover (Sentinel/Cluster) before you need it
  9. Require AUTH / ACLs and bind to private interfaces β€” an open Redis port is a shell for an attacker
  10. Size the replication backlog for real-world network blips

πŸ”— Useful Tools


⚠️ Security Notes


← Back to Home Kafka β†’ NGINX β†’ Kubernetes β†’