Quick Summary: Diagnose and fix intermittent Node.js HTTPS large file hangs on RHEL/CentOS 7 kernels. Uncover the `sendfile()` flaw and implement a critical work...

The Obscure Hang: Node.js HTTPS, Large Files, and the RHEL 7 Kernel sendfile() Stutter

Alright, you’ve hit it. That soul-crushing, intermittent issue that makes you question your career choices. Users report downloads stalling. Browsers time out. Your Node.js HTTPS server, which normally flies, starts acting like it’s running through treacle, but only sometimes, and only for big files. And guess what? It’s probably not your code. It’s a subtle, infuriating dance between your Node.js process, a specific kernel bug, and the way TLS interacts with system calls.

You’ve probably been pulling your hair out. Logs are clean, CPU is fine, memory isn’t maxed. Network looks okay. Yet, large file downloads just… stop. Or take an eternity. This isn't your garden-variety resource exhaustion; this is something far more insidious, a phantom hang in the lower layers that defies easy debugging.

The Symptoms: It's a Sporadic Nightmare

• Intermittent Latency Spikes: Large file downloads (typically >100MB) experience random, extreme latency. Sometimes they fly, sometimes they crawl to a halt for tens of seconds, then resume.
• Client Timeouts: Users get browser timeouts or download failures. Retries might work, or they might fail again.
• Server Appears Healthy: Your Node.js process metrics (CPU, memory, event loop lag) look perfectly normal. No obvious errors or warnings in your application logs.
• Stalled Network Activity: Network monitoring (netstat -p, ss) shows connections established, but minimal data transfer for extended periods on affected connections.
• No Obvious Resource Contention: Disk I/O (iostat) isn't saturated. Network bandwidth isn't maxed out.

This isn't just slow; it's stalled. It feels like the application is just waiting, frozen, even though everything on the surface screams 'healthy'.

The Investigation: Rule Out The Obvious (You Already Did, Didn't You?)

Let's pretend you haven't already checked these a dozen times:

1. Network Connectivity: Basic ping, traceroute, firewall checks. Your network team confirmed everything is green.
2. Disk Performance: Verify underlying storage isn't the bottleneck. fio benchmarks, iostat during peak load. Fast enough? Good.
3. Node.js Process Health: Is your event loop blocked? Is memory usage excessive? Use your APM tools. Chances are, they show happy green graphs, which makes this even more frustrating.
4. Node.js Version: Are you on a very old Node.js version? (Unlikely to be the sole cause here, but always good to check.)
5. TLS Configuration: Any weird ciphers or protocol versions? Usually not the smoking gun, but worth a glance.

If you've checked all that, and the problem persists, then congratulations. You're likely dealing with something far more specialized. This problem has a very specific home.

Environments Affected by This Pestilence

This particular beast thrives in a specific, frustratingly common, older ecosystem:

Operating System	Kernel Version Range	Node.js Versions Exhibiting Severity	Trigger Condition
CentOS 7.x (e.g., 7.6, 7.7)	3.10.0-957.x to 3.10.0-1062.x	v12.x, v14.x, some v16.x	Serving files > 100MB over HTTPS
RHEL 7.x (e.g., 7.6, 7.7)	3.10.0-957.x to 3.10.0-1062.x	v12.x, v14.x, some v16.x	Serving files > 100MB over HTTPS

Note: Newer kernels (4.18+ found in RHEL/CentOS 8) or significantly older RHEL 7 kernels (e.g., 3.10.0-693.x) seem less affected, or not at all. It's that sweet, rotten spot where the bug bites hardest.

The Root Cause: A Kernel's Misguided Optimization

The culprit here is the Linux sendfile() system call, and its interaction with TLS encryption on specific older RHEL 7 / CentOS 7 kernels. Node.js, via its underlying libuv library, uses sendfile() by default for efficient static file serving. This is usually a good thing. sendfile() allows direct data transfer between file descriptors (e.g., from a file to a socket) within the kernel, avoiding costly user-space copies.

However, when you introduce TLS/SSL encryption, the story changes. Data needs to be encrypted before being sent over the wire. The kernel's implementation of sendfile() in the 3.10.0-957.x to 3.10.0-1062.x range, particularly when combined with an HTTPS socket, seems to hit a critical flaw. It effectively tries to optimize a path that doesn't quite work for TLS data. The kernel gets stuck, or processes the data extremely slowly, when attempting to handle the zero-copy operation while simultaneously needing to perform crypto operations. The result is the connection stalling from the application's perspective, waiting on a system call that's simply not completing efficiently.

This isn't a Node.js bug. It's an OS kernel quirk that Node.js's efficient use of sendfile() exposes. We’ve seen similar, though distinct, issues before. For a deeper dive into related kernel sendfile() traps with Node.js and TLS, you might want to read our article: The Phantom Hang: Node.js TLS, Large Files, and the sendfile() Trap on Older Linux Kernels.

The Solution: Bruteforce and Disable That Feature

Since upgrading the kernel might be out of your hands, or you're stuck in a change freeze (classic enterprise pain), the most effective workaround is to prevent Node.js from using sendfile() for the problematic server. Node.js (specifically, its underlying libuv library) respects a very specific environment variable for this.

Just set this before launching your Node.js application:

export NODE_NO_SENDFILE=1
node server.js

Alternatively, if you're using a process manager like PM2 or Systemd, add this to your environment configuration:

# For systemd service file:
[Service]
Environment="NODE_NO_SENDFILE=1"
ExecStart=/usr/bin/node /path/to/your/server.js

# For PM2 ecosystem file:
module.exports = {
  apps : [{
    name: "my-node-app",
    script: "./server.js",
    env: {
      "NODE_NO_SENDFILE": "1"
    }
  }]
};

This forces Node.js to fall back to traditional read() and write() calls. While marginally less efficient (data copies from kernel to user space and back are introduced), it completely bypasses the kernel's problematic sendfile() behavior with TLS on these specific kernel versions. Your latency spikes will vanish, and files will stream correctly. The performance hit is often negligible for most applications compared to the absolute stalling you were experiencing.

Conclusion: A Win, But Not Without a Grumble

There you have it. Another obscure, frustrating, long-tail issue tackled. This one’s a classic example of how low-level OS interactions can manifest as seemingly application-level bugs, particularly when coupled with specific workloads and versions. The workaround is effective, but it’s a bypass, not a true fix for the underlying kernel regression. Keep an eye on kernel updates for your RHEL 7 instances; ideally, a future patch might resolve this properly, allowing you to re-enable sendfile() for peak performance. Until then, NODE_NO_SENDFILE=1 is your best friend against this particular ghost in the pipes. For related challenges with Node.js HTTPS and older CentOS versions, you might also find insights in The Ghost in the Pipes: Node.js HTTPS, Large Files, and the CentOS 7 Stutter.