🟩 NGINX Debugging & Troubleshooting Guide

A comprehensive guide for diagnosing and troubleshooting NGINX configuration, proxying, TLS, and performance issues.

📋 Table of Contents

• Config Validation & Reloads
• Process & Service Health
• Logs
• HTTP Status Code Triage
• Reverse Proxy & Upstreams
• TLS / SSL
• Performance & Limits
• Common Issues

✅ Config Validation & Reloads

# Syntax-check the config — ALWAYS before reload
nginx -t

# Test and dump the fully-resolved config (all includes expanded)
nginx -T

# Which config file and compile options is this binary using
nginx -V 2>&1 | tr ' ' '\n' | grep -E 'conf-path|with-'

# Graceful reload (workers finish in-flight requests)
nginx -s reload
systemctl reload nginx

# Find which server block actually matches a request
nginx -T | grep -nE 'server_name|listen'

nginx -t validates syntax, not behavior. It will happily accept an upstream host that doesn’t resolve at runtime (if using variables) or a root path that doesn’t exist.

🩺 Process & Service Health

# Service state and recent failures
systemctl status nginx
journalctl -u nginx --since "1 hour ago"

# Master + worker processes
ps aux | grep [n]ginx

# Is it actually listening where you think
ss -tlnp | grep nginx

# Worker connections in use vs limit
# Enable stub_status first:
#   location /nginx_status { stub_status; allow 127.0.0.1; deny all; }
curl -s http://127.0.0.1/nginx_status

stub_status fields: Active connections (current), accepts/handled (should be equal — a gap means connections dropped at accept), Reading/Writing/Waiting (Waiting = idle keepalive).

📜 Logs

# Default locations
tail -f /var/log/nginx/access.log /var/log/nginx/error.log

# Error log verbosity — set in nginx.conf, e.g.:
#   error_log /var/log/nginx/error.log warn;
# Levels: debug info notice warn error crit alert emerg

# Top requested URLs
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# Status code distribution
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

# All 5xx with timestamps
awk '$9 ~ /^5/ {print $4, $7, $9}' /var/log/nginx/access.log | tail -50

# Top client IPs
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

Add timing to the access log — the difference between request_time (full request incl. client) and upstream_response_time (backend only) tells you instantly whether NGINX or the app is slow:

log_format timed '$remote_addr [$time_local] "$request" $status '
                 'rt=$request_time urt=$upstream_response_time '
                 'us=$upstream_status conn=$connection';
access_log /var/log/nginx/access.log timed;

🚦 HTTP Status Code Triage

🔀 Reverse Proxy & Upstreams

Diagnosing 502/504

# What does the error log say (this names the real cause)
tail -50 /var/log/nginx/error.log
# "connect() failed (111: Connection refused)"  -> backend not listening
# "connect() failed (113: No route to host)"    -> network/firewall
# "upstream timed out (110)"                    -> backend too slow (504)
# "no live upstreams"                            -> all upstreams marked failed
# "(13: Permission denied) while connecting"     -> SELinux or socket perms

# Test the backend directly, bypassing NGINX
curl -sv http://127.0.0.1:<BACKEND_PORT>/health
curl -sv --unix-socket /run/app.sock http://localhost/health

# SELinux blocking outbound proxy connections (RHEL/CentOS)
getsebool httpd_can_network_connect
setsebool -P httpd_can_network_connect 1

Key Proxy Directives

location /api/ {
    proxy_pass http://backend;            # trailing slash rules matter — see below
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_connect_timeout 5s;
    proxy_read_timeout 60s;
    proxy_next_upstream error timeout;    # careful with non-idempotent requests
}

proxy_pass trailing-slash trap: proxy_pass http://b; forwards the URI unchanged; proxy_pass http://b/; replaces the matched location prefix. /api/x → http://b/api/x vs http://b/x. Most “proxy returns 404” reports are this.

DNS caching trap: proxy_pass http://api.internal; resolves once at startup. If the backend’s IP changes (K8s, ECS, ELB), NGINX keeps the stale IP until reload. Fix: resolver + a variable: set $backend api.internal; proxy_pass http://$backend;

WebSockets

location /ws/ {
    proxy_pass http://backend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 3600s;   # idle sockets die at proxy_read_timeout otherwise
}

🔒 TLS / SSL

# Inspect the served certificate (SNI matters — pass -servername)
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -dates -subject -issuer

# Verify the chain is complete (missing intermediates break mobile/curl clients)
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>&1 | grep -E 'verify|depth'

# Does cert match key (compare moduli hashes)
openssl x509 -noout -modulus -in cert.pem | openssl md5
openssl rsa  -noout -modulus -in key.pem  | openssl md5

# Test a specific protocol version
openssl s_client -connect example.com:443 -tls1_2 </dev/null

Frequent TLS failures:

• Wrong cert served → missing server_name/SNI mismatch; NGINX falls back to the default server’s cert
• “unable to get local issuer” → ssl_certificate must be the fullchain (leaf + intermediates), not the leaf alone
• Reload after renewal forgotten → certbot renewed on disk, NGINX still serves the old one (nginx -s reload)

📈 Performance & Limits

# Worker file-descriptor limit — each connection needs 1–2 FDs
nginx -T | grep -E 'worker_connections|worker_rlimit_nofile|worker_processes'
cat /proc/$(pgrep -o nginx)/limits | grep 'open files'

# "Too many open files" in error.log -> raise both:
#   worker_rlimit_nofile 65535;
#   worker_connections 16384;

# Connection queue overflows at the kernel
ss -s
netstat -s | grep -i 'listen\|overflow'

# Rate limiting state — 503s with limit_req configured
grep 'limiting requests' /var/log/nginx/error.log | tail

# Buffering: "an upstream response is buffered to a temporary file"
# -> raise proxy_buffers / proxy_buffer_size, or proxy_buffering off for streams/SSE
grep 'buffered to a temporary file' /var/log/nginx/error.log | tail

🚨 Common Issues

Issue: 502 Bad Gateway

Diagnose:

tail -20 /var/log/nginx/error.log
curl -sv http://127.0.0.1:<BACKEND_PORT>/
ss -tlnp | grep <BACKEND_PORT>

Common Causes:

• Backend process down or listening on a different port/socket
• SELinux denying outbound connections from nginx
• Backend sending oversized headers (proxy_buffer_size too small)
• PHP-FPM socket permissions

Issue: 413 Request Entity Too Large

Fix: client_max_body_size 50m; in the right context (http/server/location — a location-level setting doesn’t cover other locations). Default is only 1MB; uploads hit this constantly.

Issue: Config Reload “Succeeds” but Nothing Changes

Diagnose:

nginx -T | less          # is your change even in the resolved config?
nginx -V 2>&1 | grep -o 'conf-path=[^ ]*'   # editing the file nginx actually loads?
ps aux | grep [n]ginx    # old workers still around?

Common Causes:

• Editing a config in sites-available that was never symlinked into sites-enabled
• A duplicate server_name — nginx warns and uses the first block
• Multiple nginx installs (package + compiled) with different conf paths

Issue: Wrong Site / Default Page Served

Explanation: NGINX picks the server block by listen + server_name; no match → the default_server (or first defined). Check for typos in server_name, missing listen 443 ssl in the intended block, and requests arriving by IP (no Host header match).

Issue: 499s in the Access Log

Explanation: the client gave up before NGINX/backend answered. Usually an upstream LB or client timeout (e.g. 30s) shorter than backend response time. Fix backend latency first; matching timeouts second.

Issue: Real Client IP Shows as the Proxy’s IP

# Behind a load balancer / CDN — trust its header:
set_real_ip_from 10.0.0.0/8;        # the LB's address range
real_ip_header X-Forwarded-For;
real_ip_recursive on;

Only trust X-Forwarded-For from networks you control — clients can forge it.

📝 Best Practices

1. nginx -t before every reload, and reload (not restart) for zero downtime
2. Log request_time and upstream_response_time — cheap and answers “who’s slow” instantly
3. Set client_max_body_size deliberately, don’t discover the 1MB default in production
4. Serve fullchain certs and automate reload after renewal
5. Use resolver + variables in proxy_pass for backends with dynamic IPs
6. Keep an explicit default_server that returns 444 for unmatched Hosts
7. Raise worker_rlimit_nofile ahead of traffic growth
8. Version-control /etc/nginx — most outages are config regressions
9. Rate-limit auth and expensive endpoints with limit_req + burst
10. Restrict stub_status to localhost/monitoring networks

🔗 Useful Tools

• nginx -t / -T / -V — validation and introspection built in
• GoAccess — real-time terminal/web access-log analytics
• openssl s_client — TLS handshake debugging
• curl -v / –resolve — test virtual hosts before DNS changes
• wrk / ab — load testing
• nginx-amplify / VTS module — richer metrics than stub_status

⚠️ Security Notes

• Hide version info: server_tokens off;
• Keep TLS modern: ssl_protocols TLSv1.2 TLSv1.3;
• Add security headers (HSTS, X-Content-Type-Options, frame-ancestors CSP)
• Deny dotfiles: location ~ /\. { deny all; }
• Never expose stub_status, autoindex, or upstream admin ports publicly
• Run workers as a non-root user (default nginx/www-data)