⚡ Elasticsearch Debugging & Troubleshooting Guide

A comprehensive guide for diagnosing and troubleshooting Elasticsearch cluster issues.

📋 Table of Contents

Cluster Health & Status
Index Management
Shard Allocation
Mapping & Templates
Query & Search
Performance Monitoring
Common Issues

🏥 Cluster Health & Status

Check Cluster Health

# Basic cluster health
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cluster/health?pretty

# Detailed health per index
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/_cluster/health?level=indices&pretty"

# Shard-level health details
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/_cluster/health?level=shards&pretty"

Health Status:

🟢 Green - All shards allocated
🟡 Yellow - All primary shards allocated, some replicas unassigned
🔴 Red - Some primary shards unassigned

Check Node Information

# List all nodes
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cat/nodes?v

# Detailed node stats
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_nodes/stats?pretty

# JVM heap usage
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_nodes/stats/jvm?pretty

Cluster Settings

# View all cluster settings
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cluster/settings?pretty

# Include default settings
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/_cluster/settings?include_defaults=true&pretty"

📚 Index Management

List All Indices

# Simple list with basic info
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cat/indices?v

# Sorted by size
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/_cat/indices?v&s=store.size:desc"

# Sorted by document count
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/_cat/indices?v&s=docs.count:desc"

# Show only specific columns
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/_cat/indices?v&h=index,status,health,docs.count,store.size"

Index Statistics

# All indices stats
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_stats?pretty

# Specific index stats
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/<INDEX_NAME>/_stats?pretty

# Index settings
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/<INDEX_NAME>/_settings?pretty"

Search Across Index

# Simple search
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/<INDEX_NAME>/_search?pretty"

# Search with query
curl -s -u elastic:<YOUR_PASSWORD> 'http://localhost:9200/<INDEX_NAME>/_search?pretty' \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "match": {
        "field": "value"
      }
    }
  }'

# Prefix search example
curl -s -u elastic:<YOUR_PASSWORD> 'http://localhost:9200/k8s-logs/_search?pretty' \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "prefix": {
        "k8s.pod": "nclient"
      }
    }
  }'

🔄 Shard Allocation

Check Shard Status

# List all shards with details
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cat/shards?v

# Show only unassigned shards
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cat/shards?v | grep UNASSIGNED

# Shard allocation status
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cat/allocation?v

Explain Shard Allocation

# Why shards are unassigned
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cluster/allocation/explain?pretty

# Explain specific shard
curl -u elastic:<YOUR_PASSWORD> -X GET "http://localhost:9200/_cluster/allocation/explain?pretty" \
  -H 'Content-Type: application/json' -d'
{
  "index": "<INDEX_NAME>",
  "shard": 0,
  "primary": true
}
'

Recovery Status

# Show shard recovery progress
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cat/recovery?v

# Active recoveries only
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/_cat/recovery?v&active_only=true"

# Detailed recovery stats
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_recovery?pretty

🗺️ Mapping & Templates

View Index Mappings

# All indices mappings
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_all/_mapping?pretty

# Specific index mapping
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/<INDEX_NAME>/_mapping?pretty

# Wildcard pattern
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/k8s-logs-*/_mapping?pretty

# Specific field mapping
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/k8s-logs-*/_mapping/field/@timestamp?pretty"

Index Templates

# List all index templates (new format)
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_index_template?pretty

# List component templates
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_component_template?pretty

# List legacy templates
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_template?pretty

# Get specific template
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/_index_template/<TEMPLATE_NAME>?pretty"

Field Capabilities

# Check field types across indices
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/_field_caps?fields=*&pretty"

# Check specific fields
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/_field_caps?fields=timestamp,status&pretty"

🔍 Query & Search

Document Operations

# Get document by ID
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/<INDEX_NAME>/_doc/<DOC_ID>?pretty"

# Search for documents
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/<INDEX_NAME>/_search?q=field:value&pretty"

# Count documents
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/<INDEX_NAME>/_count?pretty"

Query DSL Examples

# Match query
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/<INDEX_NAME>/_search?pretty" \
  -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "message": "error"
    }
  }
}
'

# Range query
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/<INDEX_NAME>/_search?pretty" \
  -H 'Content-Type: application/json' -d'
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "now-1h"
      }
    }
  }
}
'

# Bool query (complex)
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/<INDEX_NAME>/_search?pretty" \
  -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        { "match": { "status": "error" } }
      ],
      "filter": [
        { "range": { "@timestamp": { "gte": "now-24h" } } }
      ]
    }
  }
}
'

📊 Performance Monitoring

Thread Pool Statistics

# Check write thread pool
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cat/thread_pool/write?v

# Check search thread pool
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cat/thread_pool/search?v

# All thread pools
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cat/thread_pool?v

JVM and Memory

# JVM stats
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_nodes/stats/jvm?pretty

# Heap usage
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/_cat/nodes?v&h=name,heap.percent,heap.current,heap.max,ram.percent,ram.current,ram.max"

# Garbage collection stats
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_nodes/stats/jvm?pretty | grep -A 10 "gc"

Disk Usage

# Node allocation and disk usage
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cat/allocation?v

# Detailed disk stats
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/_cat/nodes?v&h=name,disk.used,disk.avail,disk.total,disk.percent"

Task Management

# List running tasks
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_tasks?pretty

# List tasks with details
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/_tasks?detailed=true&pretty"

# Cancel a task
curl -u elastic:<YOUR_PASSWORD> -X POST "http://localhost:9200/_tasks/<TASK_ID>/_cancel"

🚨 Common Issues

Issue: Yellow Cluster Status

Cause: Unassigned replica shards (common in single-node clusters)

Solution for Single-Node Cluster:

# Set replicas to 0 for all existing indices
curl -u elastic:<YOUR_PASSWORD> -X PUT "http://localhost:9200/_all/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index": {
    "number_of_replicas": 0
  }
}
'

# Set default replicas to 0 for new indices
curl -u elastic:<YOUR_PASSWORD> -X PUT "http://localhost:9200/_template/default" \
  -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["*"],
  "settings": {
    "number_of_replicas": 0
  }
}
'

Issue: High Memory Usage

Check heap usage:

curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_nodes/stats/jvm?pretty

Solutions:

Increase JVM heap size (typically 50% of RAM, max 32GB)
Clear field data cache: curl -u elastic:<YOUR_PASSWORD> -X POST "http://localhost:9200/_cache/clear?fielddata=true"
Check for memory-intensive queries

Issue: Slow Queries

Enable slow log:

# Set slow query threshold
curl -u elastic:<YOUR_PASSWORD> -X PUT "http://localhost:9200/<INDEX_NAME>/_settings" \
  -H 'Content-Type: application/json' -d'
{
  "index.search.slowlog.threshold.query.warn": "10s",
  "index.search.slowlog.threshold.query.info": "5s"
}
'

Solutions:

Add appropriate indices
Use filters instead of queries where possible
Reduce result set size
Use scroll API for large result sets

Issue: Disk Space Running Low

Check disk watermarks:

curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cluster/settings?pretty | grep watermark

Solutions:

Delete old indices
Enable Index Lifecycle Management (ILM)
Increase disk space
Force merge old indices to reduce disk usage

Issue: Unresponsive Cluster

Check cluster state:

# Pending tasks
curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cluster/pending_tasks?pretty

# Check for blocked operations
curl -u elastic:<YOUR_PASSWORD> "http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s"

🛠️ Maintenance Operations

Force Merge

# Force merge index (reduces segment count)
curl -u elastic:<YOUR_PASSWORD> -X POST "http://localhost:9200/<INDEX_NAME>/_forcemerge?max_num_segments=1"

# Force merge all indices
curl -u elastic:<YOUR_PASSWORD> -X POST "http://localhost:9200/_forcemerge?max_num_segments=1"

Refresh Index

# Refresh to make documents searchable
curl -u elastic:<YOUR_PASSWORD> -X POST "http://localhost:9200/<INDEX_NAME>/_refresh"

Clear Cache

# Clear all caches
curl -u elastic:<YOUR_PASSWORD> -X POST "http://localhost:9200/_cache/clear"

# Clear specific cache types
curl -u elastic:<YOUR_PASSWORD> -X POST "http://localhost:9200/_cache/clear?fielddata=true"
curl -u elastic:<YOUR_PASSWORD> -X POST "http://localhost:9200/_cache/clear?query=true"
curl -u elastic:<YOUR_PASSWORD> -X POST "http://localhost:9200/_cache/clear?request=true"

Reindex

# Reindex data from one index to another
curl -u elastic:<YOUR_PASSWORD> -X POST "http://localhost:9200/_reindex?pretty" \
  -H 'Content-Type: application/json' -d'
{
  "source": {
    "index": "old_index"
  },
  "dest": {
    "index": "new_index"
  }
}
'

Delete Index

# Delete specific index
curl -u elastic:<YOUR_PASSWORD> -X DELETE "http://localhost:9200/<INDEX_NAME>"

# Delete with pattern
curl -u elastic:<YOUR_PASSWORD> -X DELETE "http://localhost:9200/old-logs-*"

🔧 Kubernetes Integration

Check Elasticsearch Pod

# Get pod status
sudo kubectl get pods -n netlinq elastic-0

# Execute commands inside pod
sudo kubectl exec -n netlinq elastic-0 -- curl -u elastic:<YOUR_PASSWORD> http://localhost:9200/_cluster/health?pretty

# View pod logs
sudo kubectl logs -n netlinq elastic-0 --tail=100 -f

📝 Best Practices

Monitor cluster health regularly - Set up alerts for yellow/red status
Use ILM policies - Automatically manage index lifecycle
Optimize shard size - Keep shards between 10-50GB
Set appropriate replica count - At least 1 replica for production
Enable slow query logging - Identify performance issues early
Regular maintenance - Force merge old indices, clear caches
Monitor disk space - Keep at least 15% free space
Use index templates - Standardize settings for new indices
Avoid wildcard queries - Use more specific queries when possible
Monitor JVM heap - Keep under 75% usage

🔗 Useful Resources

Elasticsearch Official Docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
REST API Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/rest-apis.html
Query DSL: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html

⚠️ Security Notes

Always use authentication in production
Never expose Elasticsearch directly to the internet
Use HTTPS/TLS for network communication
Regularly update to latest stable version
Implement role-based access control (RBAC)
Rotate passwords regularly
Use environment variables for sensitive credentials