Quick Summary: Troubleshooting intermittent Node.js HTTPS stalls on CentOS 7 when serving large files via res.sendFile() with Cache-Control: no-cache. Deep dive ...

Alright, listen up. If you've ever spent days staring at Prometheus graphs, pulling your hair out because your Node.js HTTPS server intermittently chokes when serving a seemingly innocuous 10MB file, and you're stuck on CentOS 7, this one's for you. This isn't your garden-variety memory leak or CPU spike. This is the kind of problem that makes you question your life choices, a ghost in the machine that only appears under a very specific lunar alignment of concurrent requests, file sizes, and ancient kernel quirks.

We hit this beast on a critical API endpoint serving media assets. Everything seemed fine in dev, even under load tests on newer kernels. But deploy to our battle-hardened, slightly-too-long-in-the-tooth CentOS 7 production boxes? Bam. Random connections just... hung. Not a timeout, not a disconnect. Just stalled. The client sat there, waiting, sometimes for minutes, before eventually giving up. Meanwhile, the Node.js process seemed fine, handling other requests, but that one connection? Dead in the water, consuming resources, mocking us.

The Symptoms of a Lingering Problem

How do you know you're dealing with this particular flavor of hell? Look for these:

• Intermittent Stalls: Connections hang, especially when serving large files (typically > ~5-10MB). It's not constant, which makes it infuriating.
• Node.js Process Healthy: The application itself reports no errors, CPU isn't maxed out, memory is stable. Other requests might complete fine.
• High ksoftirqd CPU: On the host, you might see spikes in CPU usage for kernel processes like ksoftirqd, indicating network stack overload or contention.
• TCP Retransmissions / Zero Window: Network packet captures (tcpdump) reveal prolonged periods of TCP zero-window advertisements from the server, or excessive retransmissions, suggesting the kernel isn't pushing data out.
• Specific Header Trigger: The problem is significantly exacerbated, or exclusively triggered, when clients send Cache-Control: no-cache headers, forcing a full re-read.

The Environments Where This Error Triggers

This is where the specificity comes in. It's an old-kernel, specific-Node.js interaction:

Operating System	Kernel Version	Node.js Version	HTTPS Library/Method	Observation
CentOS 7.x / RHEL 7.x	3.10.0-xxx.el7.x86_64	12.x, 14.x (LTS)	https module with res.sendFile() or fs.createReadStream().pipe(res)	Confirmed Trigger: Intermittent stalls, high ksoftirqd, TCP zero window.
CentOS 8.x / RHEL 8.x	4.18.0-xxx.el8.x86_64+	12.x, 14.x, 16.x	Same as above	Not Observed: Issue seems resolved with newer kernels.
Ubuntu 18.04 LTS	4.15.0-xxx-generic	12.x, 14.x	Same as above	Not Observed: Issue seems less prevalent or absent.

The Frustrating, Futile First Steps

Before you get to the good stuff, you’ve probably already tried:

• Increasing Node.js heap size.
• Tuning the server's ulimit settings.
• Checking file system I/O latency.
• Upgrading Node.js to the latest patch within its LTS branch.
• Swapping out res.sendFile() for a manual fs.createReadStream().pipe(res) (spoiler: it doesn't help because the problem is lower-level).
• Even questioning your network cards.

All dead ends. Because the problem isn't in your application code, nor is it a simple resource exhaustion. It's a kernel-level dance gone wrong.

The Root Cause

This insidious problem stems from a specific, suboptimal interaction between the Linux kernel's TCP stack, its zero-copy mechanisms (like splice() or attempts at sendfile()), and the overhead introduced by TLS (HTTPS) on older kernels, specifically 3.10.x. While Node.js's res.sendFile() (or similar streaming approaches) aims for efficiency, under the hood, the kernel tries to move data from the file descriptor directly to the socket buffer. With HTTPS, this data must pass through OpenSSL for encryption, meaning true zero-copy is difficult. Instead, the kernel often employs splice() to move data efficiently between page cache and kernel TLS buffers.

On these older kernels, under high concurrent load, especially when clients demand Cache-Control: no-cache (forcing immediate disk reads or re-reads from page cache rather than just trusting the cache), the kernel's TCP slow-start mechanism, combined with its internal buffer management for `splice` and `epoll`, enters a problematic state. It's a race condition: the kernel might be too aggressive with TCP windowing or too slow to free up internal buffers after TLS encryption, causing the socket to advertise a zero window, even though the Node.js application is ready to provide more data. The connection effectively stalls waiting for the kernel's network stack to catch up, which it sometimes doesn't, leading to indefinite hangs. This is a very specific variant of the problem explored in The Phantom Hang: Node.js TLS, Large Files, and the sendfile() Trap on Older Linux Kernels.

The tcp_slow_start_after_idle parameter, designed to prevent network congestion by reducing the initial sending rate after a period of inactivity, becomes a liability here. When a stalled connection finally gets a tiny window, and then potentially stalls again, this mechanism can continuously re-engage, crippling throughput.

The Fix: Cutting the Red Tape

After weeks of chasing ghosts with strace, perf, and tcpdump, the solution turned out to be a simple, yet obscure, sysctl tune. You need to tell the kernel to stop being so damn cautious about slow-start behavior on idle connections. This allows the TCP window to open more aggressively, preventing the cascade of stalls.

sudo sysctl -w net.ipv4.tcp_slow_start_after_idle=0
sudo sysctl -p /etc/sysctl.conf # Make it persistent

Explanation: By setting net.ipv4.tcp_slow_start_after_idle to 0, you disable the feature that forces TCP into slow start after an idle period. In scenarios with intermittent stalls on older kernels, where connections might briefly become idle due to the aforementioned `splice`/TLS/epoll contention, re-engaging slow start was actively making the problem worse, perpetuating the stalls. Disabling it allows TCP to resume at full rate more readily, bypassing the problematic re-initialization phase that caused the indefinite hangs.

Long-Term Strategy: Upgrade or Re-Architect

While this `sysctl` provides immediate relief, understand that it's a workaround for an older kernel's suboptimal behavior. The real long-term fix involves:

• Kernel Upgrade: Moving to a more modern Linux kernel (CentOS 8+, Ubuntu 20.04+) is the most robust solution. Newer kernels have significantly improved TCP stacks, `splice()` implementations, and better handling of TLS-related network I/O. This aligns with the principles of continuously improving infrastructure for Battle-Hardened Scale: Deconstructing FAANG's Global Distributed Systems.
• Re-evaluate res.sendFile(): For critical paths serving large files, consider offloading to a dedicated Nginx or CDN, or implement explicit chunking and streaming in your Node.js application to gain finer control over buffering and flow, bypassing some kernel-level black magic.

This particular troubleshooting saga proves that sometimes, the most obscure issues aren't in your code, but in the underlying operating system's fundamental interactions with your application under specific, high-stress conditions. Don't be afraid to dive deep into network captures and kernel parameters when your application tells you it's stuck, but can't tell you why.