Quick Summary: Battling Node.js EADDRINUSE after HAProxy reloads on Linux kernel 5.10.x/5.15.x with SO_REUSEPORT? Uncover the obscure race condition and fix the ...

Alright, listen up. If you've spent more nights than you care to admit staring at logs, convinced your Node.js app is haunted by a ghost port that won't die, you're in the right place. We just spent a solid week chasing down one of the most obnoxious, intermittent EADDRINUSE errors I've seen in years.

It's that special kind of hell where everything should work. Your graceful shutdown hooks are perfect. You're using SO_REUSEPORT like a good citizen. But every few HAProxy reloads, one of your Node.js instances just refuses to bind, spitting out the dreaded 'Address already in use' error.

No, it's not another instance. No, lsof shows nothing. Yes, you've checked PID files. This isn't your first rodeo. This is something far more insidious. A kernel-level dance with your application runtime that's only broken on specific versions.

The Symptoms: Intermittent, Maddening Crashes

Your Node.js services are behind HAProxy. Everything's humming along, handling traffic. Then, someone pushes a HAProxy config change, or a certificate rotation triggers a reload. Most instances come back up fine. But one, or two, randomly, just refuse.

• Error: EADDRINUSE on your application port (e.g., 3000, 8080).
• Trigger: Consistently observed after HAProxy reloads or restarts.
• Reproducibility: Extremely difficult. It's a race condition. You might see it once a day, once a week, or never in staging.
• Resolution: Manual intervention – killing the ghost process (which doesn't exist) or just waiting five minutes for the OS to eventually clean up (which defeats the purpose of fast deploys).

Sound familiar? You've probably already tried everything. Bumping up the HAProxy reload timeout. Adding delays to your Node.js startup. Even sacrificing a goat to the networking gods. Nothing worked reliably.

Environments Affected

This isn't universal. This is a specific, irritating intersection of technologies. We primarily observed this in:

Operating System	Kernel Version	Node.js Version
Ubuntu 20.04 LTS (Focal)	5.4.x, 5.10.x	14.x, 16.x
Ubuntu 22.04 LTS (Jammy)	5.15.x	16.x, 18.x, 20.x
Debian 11 (Bullseye)	5.10.x	16.x, 18.x
CentOS 8 / RHEL 8	4.18.x (No issue)	14.x, 16.x
CentOS 9 / RHEL 9	5.14.x (Minor issue)	18.x, 20.x

Notice a pattern? The problem intensifies on kernel versions 5.10.x and 5.15.x. Earlier 4.x kernels generally ignored this. Later 6.x kernels seem to have fixed it.

The Root Cause

Here's where it gets truly ugly. This is a subtle race condition in how specific Linux kernel versions (primarily 5.10.x and 5.15.x) handle socket cleanup, particularly for sockets opened with SO_REUSEPORT, when combined with a proxy like HAProxy that's rapidly cycling its own connections. HAProxy reloads often involve creating new listener sockets and gracefully (or not-so-gracefully, depending on the kernel's mood) draining connections from old ones.

When your Node.js application receives a signal to shut down (e.g., SIGTERM) and calls server.close(), it initiates a graceful shutdown. With SO_REUSEPORT, the expectation is that the port becomes immediately available for a new process to bind to. However, in these problematic kernel versions, there appears to be a microscopic window. If HAProxy is still holding onto a reference, or the kernel's internal structures for the ephemeral port haven't fully released, a subsequent attempt to bind() by a new Node.js process (or even the same process restarting) can hit EADDRINUSE.

It's not that another process owns the port; it's that the kernel hasn't entirely finished cleaning up the last one, despite SO_REUSEPORT's promise. It's a classic example of a complex interaction between application runtime, proxy, and operating system network stack. We've seen similar obscure socket issues before, like the silent EPIPE errors detailed in Node.js EPIPE: The HAProxy/Kernel 5.10.x Silent Socket Killer on Reused Connections. This is another flavor of that same low-level headache.

The Fix: The 'Unblock the Kernel' Delay

Since we can't patch the kernel (easily) or rewrite Node.js's networking stack, we have to play nice with the kernel's apparent sluggishness in these specific scenarios. The solution is simple, crude, and frustratingly effective: introduce a small, mandatory delay before your Node.js process actually exits after server.close() resolves.

This gives the kernel that crucial extra millisecond (or 500 milliseconds, just to be safe) to fully clear its internal state regarding that SO_REUSEPORT socket before the next process tries to claim it. It feels like a hack because it is. But it works.

Modify your graceful shutdown logic to include a brief pause. Here's a common pattern:

process.on('SIGTERM', () => {
  console.log('SIGTERM received. Initiating graceful shutdown...');
  server.close((err) => {
    if (err) {
      console.error('Error during server close:', err);
      // Exit immediately if close failed, something else is wrong.
      process.exit(1);
    }
    console.log('Server closed. Waiting for kernel socket cleanup...');
    // CRITICAL FIX: Introduce a small delay to allow the kernel to truly free the SO_REUSEPORT socket
    setTimeout(() => {
      console.log('Exiting process after socket cleanup delay.');
      process.exit(0);
    }, process.env.SOCKET_CLEANUP_DELAY_MS ? parseInt(process.env.SOCKET_CLEANUP_DELAY_MS, 10) : 500);
  });
});

// Assuming 'server' is your Node.js HTTP/HTTPS server instance
const server = app.listen(port, () => {
  console.log(`Server listening on port ${port}`);
});

Yes, it's an arbitrary delay. Yes, it feels dirty. But it's been the most reliable workaround for this specific kernel/SO_REUSEPORT/HAProxy interaction. We’ve seen other teams resort to similar “wait-and-pray” patterns, sometimes even using tools like VelocityPack to manage service orchestration, only to hit these low-level issues despite higher-level guarantees.

Why This Works (and Why It's Annoying)

The delay effectively sidesteps the race condition. It grants the kernel enough clock cycles to finish whatever asynchronous cleanup it's doing behind the scenes for the supposedly 'reusable' port. Without this pause, the new process (or the restarted old one) attempts to bind too quickly, hitting a transient state where the port is technically 'in use' by the kernel's internal accounting, even if no active process owns it.

It's a reminder that even when you're doing everything 'right' according to API specs and modern practices, the underlying OS and its specific version quirks can always throw a curveball. Log it, document it, and move on. Your sanity is worth a 500ms delay.