🟪 Apache Kafka Debugging & Troubleshooting Guide

A comprehensive guide for diagnosing and troubleshooting Kafka brokers, topics, consumer groups, and replication.

📋 Table of Contents

Cluster & Broker Health
Topics & Partitions
Consumer Groups & Lag
Producers
Replication & ISR
Message Inspection
Broker Logs & JVM
Common Issues

🩺 Cluster & Broker Health

# List brokers and cluster metadata (Kafka 3+, KRaft or ZK)
kafka-broker-api-versions.sh --bootstrap-server <BROKER>:9092 | head -1

# Describe the cluster (controller, broker ids, racks)
kafka-metadata-quorum.sh --bootstrap-server <BROKER>:9092 describe --status   # KRaft only

# Quick reachability test
kafka-topics.sh --bootstrap-server <BROKER>:9092 --list > /dev/null && echo OK

# ZooKeeper-era: which broker is controller
zookeeper-shell.sh <ZK_HOST>:2181 get /controller

Key Broker Metrics (JMX)

Metric	Healthy	Meaning when bad
`UnderReplicatedPartitions`	0	Replicas falling behind — broker down or overloaded
`OfflinePartitionsCount`	0	Partitions with NO leader — data unavailable
`ActiveControllerCount`	exactly 1 (cluster-wide)	0 or 2 = controller election problem
`RequestHandlerAvgIdlePercent`	> 0.3	Broker threads saturated
`ISRShrinksPerSec`	~0	Followers repeatedly dropping out of ISR

# Scrape one JMX metric without a full monitoring stack
kafka-run-class.sh kafka.tools.JmxTool \
  --object-name kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions \
  --jmx-url service:jmx:rmi:///jndi/rmi://<BROKER>:9999/jmxrmi --one-time true

📚 Topics & Partitions

# List all topics
kafka-topics.sh --bootstrap-server <BROKER>:9092 --list

# Describe a topic: leaders, replicas, ISR per partition
kafka-topics.sh --bootstrap-server <BROKER>:9092 --describe --topic <TOPIC>

# Find problem partitions cluster-wide
kafka-topics.sh --bootstrap-server <BROKER>:9092 --describe --under-replicated-partitions
kafka-topics.sh --bootstrap-server <BROKER>:9092 --describe --unavailable-partitions
kafka-topics.sh --bootstrap-server <BROKER>:9092 --describe --under-min-isr-partitions

# Topic configuration (retention, min.insync.replicas, …)
kafka-configs.sh --bootstrap-server <BROKER>:9092 --describe --entity-type topics --entity-name <TOPIC>

# Change retention at runtime
kafka-configs.sh --bootstrap-server <BROKER>:9092 --alter --entity-type topics \
  --entity-name <TOPIC> --add-config retention.ms=86400000

# Earliest/latest offsets per partition (topic size in offsets)
kafka-get-offsets.sh --bootstrap-server <BROKER>:9092 --topic <TOPIC> --time -1   # latest
kafka-get-offsets.sh --bootstrap-server <BROKER>:9092 --topic <TOPIC> --time -2   # earliest

👥 Consumer Groups & Lag

Lag — the gap between the log end offset and the group’s committed offset — is the single most important consumer-side signal.

# List consumer groups
kafka-consumer-groups.sh --bootstrap-server <BROKER>:9092 --list

# Describe group: per-partition CURRENT-OFFSET, LOG-END-OFFSET, LAG, owner
kafka-consumer-groups.sh --bootstrap-server <BROKER>:9092 --describe --group <GROUP>

# Members and their partition assignments
kafka-consumer-groups.sh --bootstrap-server <BROKER>:9092 --describe --group <GROUP> --members --verbose

# Group state (Stable / PreparingRebalance / Empty / Dead)
kafka-consumer-groups.sh --bootstrap-server <BROKER>:9092 --describe --group <GROUP> --state

Resetting Offsets

# ALWAYS dry-run first (default prints the plan; --execute applies)
kafka-consumer-groups.sh --bootstrap-server <BROKER>:9092 --group <GROUP> \
  --topic <TOPIC> --reset-offsets --to-earliest

# Common reset targets
#   --to-earliest | --to-latest | --to-datetime 2026-07-01T00:00:00.000
#   --shift-by -1000 | --to-offset 12345
kafka-consumer-groups.sh --bootstrap-server <BROKER>:9092 --group <GROUP> \
  --topic <TOPIC> --reset-offsets --to-datetime 2026-07-01T00:00:00.000 --execute

Offsets can only be reset while the group has no active members — stop the consumers first.

Rebalance Storms

Symptoms: group flips between PreparingRebalance and Stable, consumers process nothing, duplicate handling spikes.

Usual causes:

max.poll.interval.ms exceeded — one batch takes too long to process; lower max.poll.records or raise the interval
session.timeout.ms too low for GC pauses / network blips
Consumers without stable group.instance.id churning during deploys (use static membership + CooperativeStickyAssignor)

📤 Producers

# Smoke-test produce
kafka-console-producer.sh --bootstrap-server <BROKER>:9092 --topic <TOPIC>

# Produce with acks/all and key separator for compacted topics
kafka-console-producer.sh --bootstrap-server <BROKER>:9092 --topic <TOPIC> \
  --property parse.key=true --property key.separator=: \
  --request-required-acks all

# Throughput/latency benchmark
kafka-producer-perf-test.sh --topic <TOPIC> --num-records 100000 \
  --record-size 1024 --throughput -1 \
  --producer-props bootstrap.servers=<BROKER>:9092 acks=all

Producer error decoder:

Error	Root cause
`TimeoutException: Failed to update metadata`	Wrong `bootstrap.servers`, or broker’s `advertised.listeners` unreachable from the client
`NotEnoughReplicasException`	In-sync replicas < `min.insync.replicas` with `acks=all`
`RecordTooLargeException`	Record > `max.request.size` (client) or `message.max.bytes` (broker/topic)
`TimeoutException: … batch expired`	Broker slow / partition leaderless; check `linger.ms`, `delivery.timeout.ms`, broker health

🔁 Replication & ISR

# All under-replicated partitions with their ISR
kafka-topics.sh --bootstrap-server <BROKER>:9092 --describe --under-replicated-partitions

# Trigger preferred leader election (rebalance leadership after broker restart)
kafka-leader-election.sh --bootstrap-server <BROKER>:9092 --election-type preferred --all-topic-partitions

# Move partitions off a broker (generate + execute a reassignment)
kafka-reassign-partitions.sh --bootstrap-server <BROKER>:9092 \
  --topics-to-move-json-file topics.json --broker-list "1,2,3" --generate

kafka-reassign-partitions.sh --bootstrap-server <BROKER>:9092 \
  --reassignment-json-file plan.json --execute --throttle 50000000

# Check reassignment progress
kafka-reassign-partitions.sh --bootstrap-server <BROKER>:9092 \
  --reassignment-json-file plan.json --verify

Remember to --verify after a throttled reassignment — verification removes the throttle. A forgotten throttle silently caps replication forever.

🔍 Message Inspection

# Consume from beginning with keys, timestamps, partition info
kafka-console-consumer.sh --bootstrap-server <BROKER>:9092 --topic <TOPIC> \
  --from-beginning --max-messages 10 \
  --property print.key=true --property print.timestamp=true --property print.partition=true

# Read one specific partition/offset
kafka-console-consumer.sh --bootstrap-server <BROKER>:9092 --topic <TOPIC> \
  --partition 3 --offset 12345 --max-messages 1

# Inspect segment files on disk (on the broker)
kafka-dump-log.sh --files /var/lib/kafka/<TOPIC>-0/00000000000000000000.log --print-data-log | head

# Check a topic's actual on-disk size
du -sh /var/lib/kafka/<TOPIC>-*

📈 Broker Logs & JVM

# Broker log — first stop for any incident
tail -f /var/log/kafka/server.log

# Controller decisions (leader elections, ISR changes)
tail -f /var/log/kafka/controller.log

# GC pauses — long pauses cause ZK/KRaft session loss and ISR shrink
tail -f /var/log/kafka/kafkaServer-gc.log
jstat -gcutil <KAFKA_PID> 1000

# File descriptor usage — Kafka needs a high ulimit (100k+)
ls /proc/<KAFKA_PID>/fd | wc -l
ulimit -n

# Disk: Kafka dies ugly on full disks
df -h /var/lib/kafka

🚨 Common Issues

Issue: Consumer Lag Keeps Growing

Diagnose:

kafka-consumer-groups.sh --bootstrap-server <BROKER>:9092 --describe --group <GROUP>
# Is lag on ALL partitions (consumers too slow) or ONE (hot key / stuck consumer)?

Solutions:

Lag everywhere: scale consumers (up to partition count), speed up processing, raise fetch.min.bytes/max.poll.records tuning
Lag on one partition: hot partition key — fix key distribution; or one stuck consumer — check its logs/GC
Consumers flapping: fix rebalance storm causes (see above)

Issue: Under-Replicated Partitions

Diagnose:

kafka-topics.sh --bootstrap-server <BROKER>:9092 --describe --under-replicated-partitions
tail -100 /var/log/kafka/server.log | grep -i 'shrink\|expand'

Common Causes:

A broker is down or restarting
Follower disk/network too slow to keep up (replica.lag.time.max.ms exceeded)
GC pauses on the follower

Solutions:

Restore the dead broker; watch ISR expand back
If chronic on one broker: check its disk latency (iostat -x 1) and network
After recovery, run preferred leader election to rebalance load

Issue: NotEnoughReplicas / Producers Blocked

Diagnose:

kafka-topics.sh --bootstrap-server <BROKER>:9092 --describe --under-min-isr-partitions
kafka-configs.sh --bootstrap-server <BROKER>:9092 --describe --entity-type topics --entity-name <TOPIC> | grep min.insync

Explanation: with acks=all and min.insync.replicas=2, losing enough replicas that ISR < 2 makes writes fail — this is the durability guarantee working. Fix the brokers, don’t lower min.insync.replicas in a panic (that trades away durability).

Issue: Clients Connect to Wrong Address (works locally, fails remotely)

Diagnose:

kafka-configs.sh --bootstrap-server <BROKER>:9092 --describe --entity-type brokers --entity-name 1 --all | grep advertised

Explanation: clients bootstrap, then reconnect to advertised.listeners. If a broker advertises localhost or an internal IP, external clients time out after metadata fetch. Set advertised.listeners to an address routable from the client’s network (classic Docker/K8s pitfall — use separate listeners for internal and external traffic).

Issue: Disk Filling Up

Diagnose:

du -sh /var/lib/kafka/* | sort -rh | head
kafka-configs.sh --bootstrap-server <BROKER>:9092 --describe --entity-type topics --entity-name <TOPIC> | grep retention

Solutions:

Lower retention.ms / retention.bytes on the offending topics (takes effect at next log cleanup cycle)
Check that log cleaner is alive for compacted topics: grep -i 'log cleaner' /var/log/kafka/server.log
Never delete segment files by hand while the broker runs

Issue: Messages Silently Missing

Checklist:

Producer used acks=0/1 and the leader died before replication
Topic retention already deleted them (retention.ms shorter than you think)
Compacted topic: older values for the same key are removed by design
Consumer started at auto.offset.reset=latest and skipped the backlog

📝 Best Practices

acks=all + min.insync.replicas=2 + RF=3 for anything you can’t lose
Alert on UnderReplicatedPartitions > 0 and OfflinePartitionsCount > 0 — the two canonical broker alerts
Monitor consumer lag continuously, not just when things break
Size partitions for target consumers — you can’t have more active consumers in a group than partitions
Use CooperativeStickyAssignor + static membership to tame rebalances
Keep GC pauses short (G1/ZGC, modest heap ~6GB, page cache does the heavy lifting)
Set high ulimits (files 100k+) before you need them
Throttle reassignments and always --verify to remove throttles
Never colocate Kafka data with other IO-heavy workloads
Test broker failure — kill one on purpose in staging and watch ISR/leader behavior

🔗 Useful Tools

kafka-* shell scripts — everything above ships with Kafka
kcat (kafkacat) — swiss-army CLI producer/consumer
Kafka UI / AKHQ / Redpanda Console — web UIs for topics, lag, configs
Cruise Control — automated rebalancing
Burrow — consumer lag monitoring as a service

⚠️ Security Notes

Enable TLS + SASL (SCRAM or mTLS) between clients and brokers — plaintext Kafka on a shared network leaks every message
Use ACLs: kafka-acls.sh --bootstrap-server <BROKER>:9092 --list
Restrict who can create/delete topics and change configs
Encrypt inter-broker traffic in multi-DC setups
Keep credentials in env vars or a secret store, never in committed properties files