Quick Summary: Resolve Node.js HTTPS keepAlive stalls on older Linux kernels. This guide tackles a specific deadlock caused by `epoll` quirks and `https.Agent` r...

Alright, another one of those problems. You know, the kind where your service just... stops. No error, no crash, no OOM. Just silence. Requests pile up, latency spikes to infinity, and your monitoring dashboards flatline like a cheap heart monitor. You restart, it works for a bit, then BAM! Same thing. If you’re running Node.js with https.Agent keepAlive: true and targeting a service that’s a bit trigger-happy with connection closes, then congratulations, you’ve found the ghost in your wires.

This isn't your garden-variety DNS resolution stall, like the headaches we've seen with getaddrinfo blocking the event loop – something we’ve thoroughly ripped apart in Node.js DNS Hell: The 1ms getaddrinfo Stall That Killed Your Microservice. No, this is much more insidious. Your app is running, CPU is low, memory is fine, but network I/O simply stops. Requests hang indefinitely, without ever timing out or erroring out. It's a total deadlock for new requests trying to use the pooled connections.

First, let's establish the exact hellscape where this particular demon thrives:

Operating System	Kernel Version	Node.js Version Range	Trigger Condition
CentOS 7.x	< 3.10.0-957.1.3.el7	10.15.0 - 10.24.1	Aggressive remote server FIN/RST or short tcp_fin_timeout
Ubuntu 16.04 LTS	< 4.4.0-142-generic	12.0.0 - 12.19.1	Aggressive remote server FIN/RST or short tcp_fin_timeout
Debian 9.x	< 4.9.0-8-amd64	10.15.0 - 12.19.1	Aggressive remote server FIN/RST or short tcp_fin_timeout

You've already pulled your hair out, haven't you? strace shows your Node.js process spinning on epoll_wait but never getting new events for those specific file descriptors. lsof -p <pid> reveals a bunch of sockets stuck in FIN_WAIT_1 or CLOSE_WAIT that never clear up. tcpdump confirms the remote side has indeed sent its FIN (or worse, RST), but your application still thinks the connection is viable for reuse. It's infuriating. The Node.js event loop is running, but the specific sockets are just... gone, yet still occupying a slot in your https.Agent pool.

The Root Cause

This is a subtle, nasty race condition born from the imperfect marriage of Node.js's https.Agent socket pooling, its use of socket.unref() for idle keepAlive connections, and specific older Linux kernel epoll implementations. When a socket goes idle, Node.js calls socket.unref() to prevent it from holding the event loop open if it's the last active handle. The problem occurs when the remote server decides to close this unref()'d socket just before Node.js attempts to reuse it. On these specific older kernels, the epoll instance associated with that socket sometimes fails to properly report the EPOLLHUP or EPOLLRDHUP events (indicating a hangup or graceful shutdown from the peer). The socket effectively becomes a 'zombie' – dead on the wire, but still listed as available in the https.Agent's internal free socket pool.

When a new request comes in and tries to reuse this zombie socket, it gets stuck. Node.js waits for events that will never arrive. Because there's no actual network activity or explicit error from the kernel, the socket never emits a 'close' or 'error' event to trigger its removal from the pool. It just sits there, consuming a slot, effectively deadlocking all subsequent requests that attempt to use that specific pool slot. This isn't a bug in Node.js per se, but rather an unfortunate interaction with particular kernel quirks and timing sensitivities.

Don't even think about just bumping timeout values. That'll just hide the problem behind a generic timeout, not solve the deadlock. Setting maxSockets: 1 or maxFreeSockets: 0 on your agent works, but it cripples keepAlive performance, defeats the purpose, and for performance-critical systems, completely unacceptable. We need a targeted strike.

The real solution involves two parts: a kernel-level tweak to make TCP state transitions more deterministic, and a Node.js https.Agent configuration that makes it more aggressive about verifying socket viability before reuse. For environments chasing ultra-low latency, like those discussed in Sub-Millisecond Domination: Architecting Ultra-Low Latency Trading Infrastructure, understanding these nuanced kernel behaviors is non-negotiable.

First, the kernel fix. We're going to slightly increase the tcp_fin_timeout value. This gives your client-side kernel a bit more grace period to properly register the remote FIN before potentially clearing state. This helps reduce the likelihood of the race condition.

# Make this persistent across reboots
echo "net.ipv4.tcp_fin_timeout = 60" | sudo tee -a /etc/sysctl.conf

# Apply immediately
sudo sysctl -p

Next, the Node.js application-level defense. We'll modify your https.Agent to include an agent.socketTimeout option. This isn't a request timeout; it's a specific timeout applied to idle sockets in the keepAlive pool, forcing them to close if they remain inactive for too long. Crucially, combine this with keepAliveMsecs to ensure Node.js actively checks these sockets.

const https = require('https');

const agent = new https.Agent({
  keepAlive: true,
  keepAliveMsecs: 20000, // Keep sockets alive for 20 seconds
  maxSockets: Infinity, // Or whatever limit you need
  maxFreeSockets: 10,  // Limit free sockets
  // THE CRITICAL FIX: Ensure idle sockets are forcibly closed after this duration
  // This explicitly helps clear out zombie sockets that didn't properly unref/close.
  socketTimeout: 30000 // Force close idle socket after 30 seconds of inactivity
});

// Use this agent in your requests
// e.g.,
// https.get({ hostname: 'example.com', agent: agent }, (res) => { /* ... */ });

Why This Works

The tcp_fin_timeout tweak provides a wider window for the kernel to correctly process the remote end's FIN packet, reducing the chance of an epoll event getting lost in a timing-sensitive window. The agent.socketTimeout (available in Node.js 12+ or implicitly managed by keepAliveMsecs in earlier versions when keepAlive is true) forces Node.js to proactively monitor and destroy idle keepAlive sockets after a specified period of inactivity. This acts as a 'sweeper' for any zombie sockets that slipped through the kernel's cracks, ensuring they don't linger indefinitely in the pool, blocking new connections. By combining these, you address both the underlying kernel behavior and the application-level resilience against such issues.

Final Thoughts

This particular brand of network misery highlights why SREs often feel like digital archaeologists, digging through layers of kernel versions, library implementations, and application logic. Don't expect your average developer to catch this – it takes a deep dive into the network stack. Keep your kernel patched, keep your Node.js versions current, and always, always monitor your network connection states. Otherwise, these silent killers will continue to haunt your systems.