Quick Summary: Node.js apps on RHEL 8.x intermittently hang during HTTP/2 requests? This SRE guide uncovers the obscure kernel-level interaction causing ECONNRES...

Another Tuesday, another 'intermittent' production issue. This one? A Node.js service, rock-solid for months, suddenly started spewing ECONNRESET and request timeouts. But only under specific, infuriating conditions. If you've spent days chasing phantom network issues, cursing your load balancer, and questioning your life choices, you're in the right place.

This isn't your average 'DNS issue' or 'firewall blocked it' scenario. This is a deep, nasty interaction between Node.js's http2 implementation and specific Linux kernel versions. It's a ghost in the machine, and we're going to exorcise it.

The Symptoms: Intermittent Despair

You're seeing ECONNRESET or plain old request timeouts on outbound HTTP/2 calls. Not always, not everywhere. Just some services, some of the time, on some nodes. If you're lucky, your logs might show a Client network socket disconnected before secure TLS connection was established or similar message. More often, it's just a black hole of silence until your application's timeout kicks in.

What makes it so insidious is the intermittent nature. Under light load, it might never happen. Under heavy load, it becomes more frequent, but still not 100% reproducible. You scale up, you scale down, you restart, you pray. Nothing truly fixes it.

The Setup: Where the Pain Lives

This problem is highly specific to a particular combination of operating system kernel and Node.js versions. We’ve seen it consistently trigger in the following environments:

Component	Affected Versions	Notes
Operating System	Red Hat Enterprise Linux (RHEL) 8.4 - 8.6	Specifically kernel versions from 4.18.0-305.el8.x86_64 to 4.18.0-425.el8.x86_64. CentOS Stream 8 and AlmaLinux 8 in this range may also be affected.
Node.js Runtime	16.x, 18.x	Applications using the built-in http2 module directly or indirectly (e.g., via axios, undici, gRPC libraries configured for HTTP/2). Node.js 20.x appears less susceptible due to internal changes.
Network Configuration	Outbound HTTP/2 requests	Especially pronounced when connecting to external services or other microservices via HTTP/2-enabled load balancers/proxies (e.g., Envoy, Nginx with http2).

You've packet-traced, you've checked netstat, you've cursed the network team. Everything looks fine on the surface. CPU isn't pegged, memory's stable. The target service isn't showing errors. You're pulling your hair out. You’ve probably considered whether another 'blazing-fast' build system might secretly be causing network weirdness, but no, it's not that.

The Deep Dive: What Didn't Work (and Wasted Your Time)

• Increasing Node.js timeouts: This just delays the inevitable ECONNRESET or hang. It doesn't solve the underlying problem.
• Disabling HTTP/2 (forcing HTTP/1.1): If your application supports it, this *might* workaround the issue, but it defeats the purpose of modern service communication and often isn't a viable long-term solution.
• Kernel upgrades: Upgrading to RHEL 8.7+ (kernel 4.18.0-477 or newer) or RHEL 9 *may* resolve it, but that's a big lift for many organizations.
• Fiddling with firewall rules: Unless you actually have a firewall misconfiguration (you checked that first, right?), this is a dead end.

The Root Cause

This is where it gets ugly. The issue stems from a subtle race condition and an aggressive optimization within the Linux kernel's TCP stack, specifically its tcp_autocorking feature. In the affected RHEL 8.x kernel versions, the tcp_autocorking logic, designed to coalesce small TCP writes into larger packets for efficiency, can interact poorly with the rapid, interleaved writev system calls made by Node.js's http2 module during the TLS Application-Layer Protocol Negotiation (ALPN) handshake and initial HTTP/2 settings frame exchange.

Node.js's http2 implementation is highly optimized, using epoll for I/O events and writev for efficient gathering of disparate buffers into a single syscall. During the initial connection setup, especially when negotiating TLS with ALPN and immediately sending HTTP/2 preamble, Node.js can issue several small, rapid writes. If the tcp_autocorking mechanism aggressively buffers these writes, and concurrently, the remote HTTP/2 server closes the connection very quickly (e.g., due to its own immediate error, timeout, or an unusual network condition *before* the corked data is fully flushed), a race condition ensues. The kernel's TCP state machine, combined with the delayed flushing from autocorking, can result in the Node.js client side either not receiving an expected ACK, receiving an unexpected FIN/RST much later than anticipated, or simply having its critical initial handshake data effectively 'lost' in the buffer due to an untimely connection termination.

This leaves the Node.js http2 client in a hung state, waiting for network events that will never arrive correctly, eventually leading to the dreaded ECONNRESET or a hard application timeout. It's a very specific timing window that's hard to hit but becomes prevalent under certain network conditions and moderate application load. Think of it as a tiny, misaligned cog in an otherwise perfectly functioning machine, causing catastrophic jams. The relentless calculus of scaling distributed systems at hyperscale often brings such obscure issues to light.

The Fix: Shutting Down the Cork

The immediate and most effective workaround is to disable tcp_autocorking on the affected systems. This tells the kernel to not aggressively delay small writes, allowing Node.js's http2 module to flush its initial handshake packets more immediately, circumventing the race condition.

This is a system-wide change. While generally safe, monitor network performance after implementation, especially on systems with extremely high numbers of small TCP writes. In most microservice architectures, the benefits of stability outweigh the marginal (if any) performance hit.

To disable tcp_autocorking permanently, first apply it:

sudo sysctl -w net.ipv4.tcp_autocorking=0

Then, to make it persist across reboots, add the following line to a new or existing sysctl configuration file, for example, /etc/sysctl.d/99-disable-autocorking.conf:

echo "net.ipv4.tcp_autocorking = 0" | sudo tee /etc/sysctl.d/99-disable-autocorking.conf
sudo sysctl --system

After applying this change, restart your Node.js services. You should see a dramatic reduction, if not complete elimination, of the intermittent HTTP/2 hangs and ECONNRESET errors.

Final Thoughts

This problem is a prime example of why SREs age prematurely. It’s a low-level interaction, hidden behind layers of abstraction, triggered by specific versions and conditions. While disabling tcp_autocorking might feel like a blunt instrument, it’s a targeted strike against a specific kernel-level flaw that's causing widespread application instability.

Always keep your kernels updated, but more importantly, understand the deep interactions between your application runtime and the underlying OS. Sometimes the simplest fix hides the most complex root cause. Don't let these obscure issues consume your sanity.