Quick Summary: Node.js apps hitting EPIPE errors with http.Agent keepAlive? Pinpoint the interaction between Linux kernel 5.10.x, HAProxy/Envoy, and Node's defau...

Alright, listen up. If you've been tearing your hair out because your Node.js application, running behind HAProxy or Envoy, suddenly starts spewing EPIPE errors after a period of idle traffic, but ONLY on specific servers, then welcome to hell. I've spent too many late nights debugging this particular brand of misery, and I’m going to save you the headache.

This isn't your garden-variety ECONNRESET. This is far more insidious. It's a silent killer, where Node.js thinks it has a healthy pooled connection, tries to write to it, and boom – EPIPE. No warning, no immediate FIN from the other side. Just a sudden, unexpected pipe break.

We saw this repeatedly across different microservices, all using the default http.Agent (or https.Agent) for outbound requests, hitting internal APIs proxied by HAProxy or Envoy. The common denominator wasn't our application code; it was the infrastructure.

First, confirm you're in the blast zone. This particular issue is not universal. It's an unholy trinity of specific kernel versions, Node.js keepAlive defaults, and how certain proxies manage their idle connections.

Here’s where we consistently observed this failure mode:

Operating System	Kernel Version	Node.js Version(s)	Proxy/Load Balancer
Ubuntu LTS 20.04/22.04	5.10.x (specifically 5.10.0-105-generic to 5.10.0-112-generic)	14.x, 16.x, 18.x	HAProxy 2.x, Envoy 1.x
Debian 11 (Bullseye)	5.10.x (e.g., 5.10.0-13-amd64)	14.x, 16.x, 18.x	HAProxy 2.x, Envoy 1.x

Notice the kernel. That 5.10.x series? That’s your first red flag. This problem seemed to vanish when we upgraded to 5.15.x or newer kernels on some test environments, which makes me suspect some subtle TCP stack changes. For a similar (but distinct) ECONNRESET issue on a different kernel, you might find insights in The Ghost in the Machine: Node.js ECONNRESET with Keep-Alive on Kernel 5.15.x. It's a different beast, but the underlying keepAlive interaction with kernel TCP states is a common theme in this type of bug.

The Root Cause

The core problem lies in a nasty interaction between Node.js's default http.Agent keepAlive behavior, the Linux kernel's TCP state management on the 5.10.x series, and the default idle timeouts of HAProxy or Envoy.

Node.js, by default, reuses HTTP connections via keepAlive. It keeps a pool of idle sockets open, hoping to send subsequent requests faster. It sets a freeSocketTimeout (or similar internal logic depending on Node.js version and explicit configuration) after which it should close idle connections.

However, the specific 5.10.x kernels we observed exhibit a peculiar behavior with regard to TCP_USER_TIMEOUT and possibly TCP_KEEPALIVE (application-level, not kernel-level). When HAProxy or Envoy silently closes an idle backend connection due to its own timeout (often 30-60 seconds, which can align perfectly with Node's freeSocketTimeout), it doesn't always send a RST packet immediately. Sometimes, it's a graceful FIN or just a quiet close on the proxy's side after its own idle timeout.

The Node.js client side, running on kernel 5.10.x, often holds onto the socket for a bit longer, believing it's still alive. When Node.js finally picks this stale socket from the pool for a new request, it attempts to write to it. The kernel then realizes the remote end is gone, but instead of returning an ECONNRESET (which is more common for unexpected closes), it throws an EPIPE. This typically means "broken pipe," implying a write to a connection that no longer has a reader. The connection wasn't actively reset; it just went away, and the write failed. It’s like trying to shout into a phone that’s been silently disconnected.

This often gets compounded by high load scenarios, where connections are frequently opened and closed, and the race conditions expose this kernel/proxy interaction more readily. We've seen related issues when discussing The Phantom Reset: Node.js keepAlive and Kernel Scheduling Under Load, where network conditions and kernel scheduling can exacerbate keepAlive problems.

The Solution: Taming keepAlive Aggression

The most reliable, immediate fix is to make Node.js more aggressive about dropping idle connections, aligning its keepAlive timeout with or below your proxy's shortest idle timeout. Don't rely on the kernel to tell Node.js the connection is dead after Node tries to write. Make Node.js proactive.

You need to explicitly configure your http.Agent (or https.Agent) to have a shorter freeSocketTimeout and potentially a shorter timeout. The freeSocketTimeout is key here. By default, it’s 60 seconds. If your HAProxy timeout tunnel or Envoy idle_timeout is also 60s, you're playing a dangerous game of chicken. Drop Node.js's timeout below that.

Here's how you do it. This isn't just a suggestion; it's a non-negotiable directive. Apply this universally to all your outbound Node.js HTTP clients where keepAlive is enabled.

const http = require('http');
const https = require('https');

// Define a custom agent with a shorter freeSocketTimeout
const customHttpAgent = new http.Agent({
  keepAlive: true,
  maxSockets: Infinity, // Adjust as needed for your application's concurrency
  maxFreeSockets: 10,  // Adjust as needed to control memory usage
  freeSocketTimeout: 20000 // 20 seconds, MUST be less than your proxy's idle timeout
});

const customHttpsAgent = new https.Agent({
  keepAlive: true,
  maxSockets: Infinity,
  maxFreeSockets: 10,
  freeSocketTimeout: 20000 // 20 seconds
});

// Example of using it with fetch (Node.js 18+) or a library like axios
// For fetch:
// const res = await fetch('http://your-service.com/api', {
//   agent: customHttpAgent,
//   // ... other options
// });

// For libraries like axios:
// const axios = require('axios');
// const axiosInstance = axios.create({
//   httpAgent: customHttpAgent,
//   httpsAgent: customHttpsAgent,
//   timeout: 25000 // A slightly higher request timeout than freeSocketTimeout for safety
// });
// axiosInstance.get('http://your-service.com/api');

// For native http.request:
// const options = {
//   hostname: 'your-service.com',
//   port: 80,
//   path: '/api',
//   method: 'GET',
//   agent: customHttpAgent // Use the custom agent
// };
// const req = http.request(options, (res) => { /* ... */ });
// req.end();

// IMPORTANT: If you have different agents for different services, ensure you apply this everywhere.
// For global configuration in some frameworks, you might need to monkey-patch or use a wrapper.
// E.g., for global fetch agent in Node 18+ (use with caution, better to inject):
// const originalFetch = global.fetch; // Store original fetch
// global.fetch = (input, init) => originalFetch(input, { agent: customHttpAgent, ...init });
// Or better, inject agents into specific client instances where possible and avoid global monkey-patching.

Set that freeSocketTimeout to something like 20 or 30 seconds. Whatever value you pick, make absolutely certain it's at least 10-15 seconds less than the shortest idle timeout configured on any intermediary proxy (HAProxy timeout client/timeout server/timeout tunnel, Envoy idle_timeout). This forces Node.js to proactively close idle sockets before the proxy has a chance to quietly sever them, preventing Node.js from ever picking up a "zombie" connection.

This isn't a workaround; it's proper keepAlive hygiene. You need to explicitly manage these timeouts. Don't trust defaults when you're dealing with complex network interactions across different layers of software and hardware. The "obvious" fix often isn't. Get this done, and your EPIPE nightmares on kernel 5.10.x will be a thing of the past.