Quick Summary: Unraveling intermittent Node.js EPIPE errors behind HAProxy on Linux Kernel 5.10.x due to subtle keepAliveTimeout mismatches on reused connections...

Alright, folks. Let's talk about that garbage you're seeing in your Node.js application logs: the dreaded EPIPE error. Not the garden-variety kind, mind you. I'm talking about the phantom EPIPE that strikes intermittently, specifically on reused connections, leaving your Node.js server scratching its head while HAProxy just sails on, oblivious. If you're running on Linux kernel 5.10.x and using HAProxy, buckle up. This one's for you.

The Symptoms: Intermittent, Unpredictable, Infuriating

You’ll see logs like this:

Error: write EPIPE
    at Socket._write (node:internal/net:88:14)
    at writeOrBuffer (node:internal/streams/writable:389:12)
    at _write (node:internal/streams/writable:330:10)
    at Writable.write (node:internal/streams/writable:334:10)
    at ServerResponse.end (node:_http_outgoing:982:10)
    at someHandler (your-app.js:XX:YY)

The requests hit Node.js, your application logic runs, and then, inexplicably, when Node.js tries to send the response back, it bombs out with EPIPE. It doesn't happen on every request. It's often on connections that have been idle for a short period and are then reused by HAProxy for a subsequent request. The client might get a 502 Bad Gateway from HAProxy, or sometimes just a hung connection that eventually times out.

The Environments Where This Garbage Triggers

This isn't a universal problem. It's a specific, nasty cocktail of components:

The Root Cause: HAProxy's Aggressive Disconnect Meets Kernel's Grace Period

Here’s the deal: Node.js HTTP servers, by default, have a server.keepAliveTimeout of 5000 milliseconds (5 seconds). This means Node.js will keep an idle connection open for 5 seconds, hoping for another request. This is generally a good thing for performance, reducing TCP handshake overhead.

HAProxy, on the other hand, is a beast of its own. It has various timeouts, and the critical one here is often timeout http-request (or sometimes timeout tunnel or a combination of timeout client and timeout server). This timeout dictates how long HAProxy will wait for the entire HTTP transaction (request + response) to complete. Crucially, it also implicitly influences how long HAProxy keeps its backend connection to Node.js alive after a request is completed, especially if it's shorter than Node.js's keepAliveTimeout.

The architectural flaw emerges when HAProxy's internal logic, possibly influenced by an expired timeout http-request or an aggressive connection pooling strategy, decides to silently close its connection to the Node.js backend. It sends a TCP FIN packet and then closes its socket. Node.js receives this FIN. However, on Linux kernel 5.10.x, there appears to be a subtle behavioral difference in how the TCP stack handles this half-closed state, or how quickly Node.js’s event loop processes it and updates the socket’s internal state. For a deeper dive into kernel-level TCP shenanigans with Node.js, you might want to check out The Phantom Reset: Node.js keepAlive and Kernel Scheduling Under Load.

Node.js, unaware that HAProxy has already closed its end, keeps the connection alive for its full keepAliveTimeout duration. When a new request arrives, or if Node.js tries to send a response on what it thinks is a live, idle connection, it attempts to write to a socket that has effectively been closed on the other end. BOOM. EPIPE. You’re trying to write to a pipe that no longer exists.

This isn't an ECONNRESET (which typically implies an abrupt, unexpected termination, often due to Node.js trying to read from a dead socket). This is a write error because Node.js believes it has a valid write target, but the peer has already gracefully (from its perspective) closed the writing end. This specific interaction is often overlooked because Node.js's 5-second default is generally robust. But not here. For similar but distinct issues with EPIPE related to HAProxy and kernel 5.10.x, see also Node.js EPIPE Catastrophe: The HAProxy-Kernel 5.10.x keepAlive Silent Killer.

The Fix: Stop Being So Optimistic, Node.js!

The solution is brutally simple: make Node.js's keepAliveTimeout shorter than HAProxy's most aggressive connection-closing timeout (like timeout http-request). This forces Node.js to close its end of the connection before HAProxy has a chance to silently tear it down, preventing the half-closed state race condition.

Set Node.js's keepAliveTimeout to something comfortably less than HAProxy's timeout http-request. If HAProxy's timeout http-request is, say, 2 seconds (2000ms), then set Node.js to 1 second (1000ms).

Copy-Pasteable Node.js Configuration Override

Here's how you do it. Modify your Node.js HTTP server initialization:

const http = require('http');

const server = http.createServer((req, res) => {
  // Your application logic here
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.end('Hello from Node.js!');
});

// CRITICAL FIX: Set keepAliveTimeout to be shorter than HAProxy's timeout http-request
// Example: If HAProxy's timeout http-request is 2s (2000ms), set this to 1s (1000ms)
server.keepAliveTimeout = 1000; // 1 second

// Also, consider headersTimeout if you're still seeing issues,
// though keepAliveTimeout is usually the primary culprit for EPIPE.
// server.headersTimeout = 1100; // Slightly longer than keepAliveTimeout, but shorter than HAProxy

server.listen(3000, () => {
  console.log('Node.js server listening on port 3000');
});

Why This Works

By making Node.js’s keepAliveTimeout shorter, you ensure that Node.js will proactively close its side of the TCP connection (send a FIN) and fully tear down the socket before HAProxy, under its more aggressive timeout settings on kernel 5.10.x, has a chance to silently close its end first. This eliminates the race condition where Node.js attempts to write to a half-dead socket. It forces a clean slate for each connection or ensures connections are fully closed by Node.js before HAProxy decides to act.

Conclusion

This isn't rocket science, but it’s a classic example of subtle interaction between different layers of your stack. Don't let default timeouts bite you. Tune your Node.js server’s keepAliveTimeout. Save yourself the headache. Now get back to work.