Victor's room

Overview & Root Cause Summary: The error 502 Bad Gateway (connect() failed (111: Connection refused) while connecting to upstream) occurs in Nginx reverse proxies when Nginx acts as an edge proxy or load balancer but cannot establish a TCP connection or Unix socket link with the designated upstream backend server (such as Node.js, Gunicorn/Uvicorn, PHP-FPM, or a Docker container).

Understanding the Root Causes

• Upstream Application Daemon Offline: The backend process (e.g., Express.js, FastAPI, Spring Boot, PHP-FPM) has crashed, stopped, or is listening on a different port than the one specified in proxy_pass.
• Unix Domain Socket Permission Denied: When using Unix domain sockets (e.g., /var/run/php/php-fpm.sock), the Nginx worker process (running as www-data or nginx) lacks read/write file permissions.
• SELinux or Firewall Blocking Proxy Connections: On RHEL/CentOS/Rocky Linux systems, SELinux policy by default prevents Nginx from initiating outbound network connections (httpd_can_network_connect).
• Docker Container Localhost vs Host Mismatch: In containerized setups, configuring proxy_pass http://127.0.0.1:port inside Nginx attempts to reach the Nginx container’s local loopback instead of the backend service container.

Step 1: Quick Fix (Verify & Restart Upstream Backend Service)

Check whether the upstream application is running and actively listening on the configured port.

# Check if the backend application process is running
sudo ss -tulpn | grep -E ':(3000|8000|8080|9000)'
# Or using lsof:
sudo lsof -i :3000

# Restart backend service if stopped (e.g., PHP-FPM or systemd service)
sudo systemctl restart php8.2-fpm
# Or for custom systemd service:
sudo systemctl restart my-backend-app

# Inspect backend service logs for unhandled fatal crashes
journalctl -u my-backend-app -n 50 --no-pager

Step 2: Fix Unix Socket Permissions & Nginx Upstream Configuration

Ensure the Nginx configuration accurately maps to the backend address, and verify socket permissions.

# In /etc/nginx/sites-available/default or /etc/nginx/conf.d/app.conf:
# Example for TCP Proxy:
location / {
    proxy_pass http://127.0.0.1:3000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

# If using Unix domain socket (e.g., PHP-FPM):
# fastcgi_pass unix:/var/run/php/php8.2-fpm.sock;

# Grant www-data ownership to the socket if permission error exists:
sudo chown www-data:www-data /var/run/php/php8.2-fpm.sock
sudo chmod 660 /var/run/php/php8.2-fpm.sock

Step 3: Resolve SELinux Restrictions & Docker Container Networking

Enable proxy network connections under SELinux or configure Docker DNS host routing.

# For SELinux systems (RHEL, CentOS, Rocky Linux, AlmaLinux):
# Allow Nginx to make network connections to upstream ports
sudo setsebool -P httpd_can_network_connect 1

# For Docker / Docker Compose setups:
# Use container service names instead of localhost in proxy_pass:
# proxy_pass http://backend_service_name:3000;
# Ensure both Nginx and backend containers share the same Docker network.

Verification & Testing Steps

Test Nginx configuration syntax, reload the service, and verify HTTP upstream response codes.

# 1. Test Nginx syntax configuration
sudo nginx -t

# 2. Reload Nginx without downtime
sudo systemctl reload nginx

# 3. Test HTTP status from CLI
curl -I http://localhost

# 4. Monitor live Nginx error logs for upstream status
tail -f /var/log/nginx/error.log

Summary Comparison Table