Quick Summary: Solving obscure Node.js EADDRINUSE errors on Linux after restarts. Debugging ephemeral port range exhaustion and SO_REUSEPORT interaction in speci...

Alright, listen up. You've got a Node.js service, maybe an HTTP server, maybe a pure TCP listener, doesn't matter. It's stable, runs fine for days. Then you restart it. Boom. EADDRINUSE or worse, it just hangs, refusing to bind, only to randomly work hours later. You've checked the usual suspects, lsof, netstat, the port is CLEAR. What the hell is going on? This isn't a race condition you can fix with a simple setTimeout. This is deeper. This is a ghost in your socket, and it's driving you nuts.

Your application, post-restart, spews errors like 'Address already in use' even when netstat -tulpn | grep :<port> shows nothing listening. Or, it binds successfully but subsequent connections fail with 'Connection refused' or 'Network unreachable' for a few minutes. You've waited, you've cursed, you've maybe even rebooted the entire VM out of sheer desperation. This isn't a race to acquire a lock, it's a persistent, insidious bind failure that only happens after a clean shutdown and immediate restart, especially under moderate to high ephemeral port usage.

Affected Environments

This obscure issue often surfaces under specific combinations of OS and Node.js versions. Be warned:

OS Version	Node.js Version	Linux Kernel Version	Affected Network Library	Severity
Ubuntu 20.04 LTS	14.x, 16.x	5.4.x - 5.11.x	net module (all types)	High
Debian 11 (Bullseye)	14.x, 16.x	5.10.x	net module (all types)	High
CentOS 7.x, 8.x	14.x, 16.x	3.10.x - 4.18.x	net module (all types)	Medium-High
Alpine Linux 3.12+	14.x, 16.x	5.4.x - 5.11.x	net module (all types)	High

Debugging Attempts: What Didn't Work

You've probably done the lsof -iTCP -sTCP:LISTEN dance. You've grepped netstat. You've even gone into /proc/sys/net/ipv4/tcp_fin_timeout and tweaked it, convinced it's lingering TIME_WAIT states. You've tried server.unref() and server.close() with callbacks. None of it touches the core issue. The problem isn't your Node.js application holding the port; it's the kernel's interaction with the entire ephemeral port range under specific conditions.

The Root Cause

The culprit is a nasty confluence of Node.js's default socket options, specifically how it handles SO_REUSEPORT implicitly when dealing with the underlying operating system, and the Linux kernel's behavior regarding ephemeral port allocation when net.ipv4.ip_local_port_range is heavily utilized. When a Node.js server attempts to bind, it often asks for a specific port. However, when it's acting as a client (e.g., making outbound requests), it uses ephemeral ports.

The issue arises when a server, especially one making many outbound connections (think microservices chatter or a service connecting to a database like Redis/Postgres/MySQL using many pooled connections), restarts. The previous process's ephemeral ports, even after being closed, linger in a TIME_WAIT or similar state within the kernel's port allocation table, even if not visible via netstat on the listening port. Node.js, in versions 14.x and 16.x (especially pre-18.x), wasn't always as robust in its SO_REUSEADDR or SO_REUSEPORT handling when it came to client sockets implicitly releasing their ephemeral ports.

When your server attempts to bind to its listening port again, if that specific port happens to fall within the ip_local_port_range and is currently 'soft-held' by the kernel due to recent ephemeral use by the same process/user, it can lead to EADDRINUSE. This is particularly prevalent in systems with high connection churn, or those where ephemeral ports are exhausted rapidly, such as applications that make extensive use of external APIs or database connections. It's not a bug in Node.js itself, per se, but an impedance mismatch between how Node.js expects SO_REUSEPORT to behave and how certain Linux kernel versions manage the lifecycle of recently used ephemeral ports, especially if tcp_tw_reuse is disabled or too conservative. This problem forces you to think about system-level resource management beyond just your application code, echoing similar challenges found in engineering for zero-tolerance latency in HFT, where every millisecond of port availability counts.

The Fix

Forget tinkering with tcp_fin_timeout. Forget rebooting. The most effective, albeit slightly crude, solution is to ensure your service's systemd unit or startup script explicitly delays the restart for a few seconds. This gives the kernel sufficient time to truly clear those 'soft-held' ephemeral port states before Node.js attempts to re-bind its primary listener. It's ugly, but it works, and it ensures your service reliably comes back up. For systems that run multiple services and struggle with this kind of kernel-level resource contention, it's also worth revisiting your orchestration strategy; for some, simplicity often crushes over-engineering.

Terminal Command / Config Override (systemd)

For a systemd service unit, typically located at /etc/systemd/system/your-node-app.service, add or modify the RestartSec directive. This tells systemd to wait X seconds before attempting to restart the service after it has stopped. A 5-second delay is usually sufficient to clear the ephemeral port tables.

[Unit]
Description=My Critical Node.js Service
After=network.target

[Service]
ExecStart=/usr/bin/node /opt/your-app/index.js
WorkingDirectory=/opt/your-app
Restart=always
RestartSec=5  # Add this line or modify it
User=nodeapp
Group=nodeapp
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target

After modifying the service file:

sudo systemctl daemon-reload
sudo systemctl restart your-node-app

Watch your logs. The EADDRINUSE errors should vanish on subsequent restarts.

Why This Fix Works

The RestartSec directive directly addresses the timing issue. By forcing a brief pause, you're giving the Linux kernel's internal socket tables, specifically those managing the ephemeral port range (ip_local_port_range), enough breathing room to fully release all resources associated with the previous process instance. Even if tcp_tw_reuse is enabled, there's a small window where ports are not immediately available for a new bind attempt by the same user/process context if they fall into the same internal hash bucket or are still being processed. This short delay avoids that transient conflict. It's not ideal, but it's a practical workaround for an obscure kernel/application interaction.

Conclusion

This isn't a glamorous fix. It's a pragmatic hack to get your application stable. Understanding these low-level interactions between your application runtime, the operating system, and network protocols is critical for robust SRE work. Sometimes, the 'right' answer isn't a complex code change, but a simple, well-placed RestartSec to appease the kernel gods. Now go forth and conquer those rogue sockets.