Quick Summary: Struggling with intermittent Node.js HTTP/2 connection resets on localhost in cgroup-throttled containers on RHEL 7? This SRE guide uncovers the o...

Alright, listen up. You’ve probably torn your hair out, thrown monitors, and questioned your life choices trying to debug this particular nightmare. You're seeing random, infuriating HTTP/2 connection resets, but only on your Node.js services talking to each other via localhost. And it only happens in specific container environments. You've checked everything: network configurations, Node.js process memory, even tried updating npm packages. Nothing. You're convinced it's a ghost in the machine.

It's not a ghost. It's an obscure, nasty interaction between specific Linux kernel versions, cgroup CPU throttling, Node.js's SO_REUSEPORT behavior, and epoll edge-triggered notifications. This isn't your average 'high CPU' or 'GC pause' issue. This is deeper. This is the stuff that makes veterans want to retire.

The Problem: Intermittent HTTP/2 Resets on Localhost

You're running multiple Node.js services inside Docker or Podman containers. These services communicate with each other over localhost using HTTP/2. Critically, these containers are resource-constrained by cgroups, specifically CPU quotas (cpu.cfs_quota_us). Your logs show frequent ERR_HTTP2_SESSION_ERROR or similar connection reset messages, but only when the system is under load and the CPU throttling kicks in. On bare metal, or in a container with uncapped CPU, it works perfectly. Upgrade to a bleeding-edge kernel, and the problem often vanishes, only to reappear when you deploy to your 'stable' production environment.

Sound familiar? Good. You're in the right place.

Triggering Environments

This exact concoction of misery typically manifests under these specific conditions:

Component	Version(s) Where Triggered	Notes
Operating System	RHEL 7.x (Kernel < 4.18), CentOS 7.x (Kernel < 4.18)	Specifically older 3.10.0-x and some 4.x kernels.
Node.js Version	12.x, 14.x, 16.x (LTS)	Versions that leverage libuv and SO_REUSEPORT by default for servers.
Container Runtime	Docker, Podman, Kubernetes (cgroup v1 environments)	Any environment where cgroup CPU throttling is active.
Network Endpoint	localhost (127.0.0.1) or loopback interface	Problem is less pronounced, or absent, on external IPs.
Networking Stack	HTTP/2	HTTP/1.1 shows fewer, but similar, issues.
Symptoms	ERR_HTTP2_SESSION_ERROR, intermittent ECONNRESET, stalled requests.	Exacerbated under load and cgroup CPU pressure.

Initial Head-Scratchers (What NOT to do, or already tried)

You've probably gone down the rabbit hole of increasing Node.js's maxSockets, playing with TCP keepalives, or even trying different libuv versions. You might have seen some general network spikes and assumed it was related to other cgroup issues. While Node.js Network Spikes in Throttled Containers: The CGroup-GC-TCP Head Scratcher addresses a different beast, it highlights how insidious cgroup interactions can be. This particular problem, however, isn't about GC pauses or general TCP window management. It's lower level.

You might have also optimized your Node.js application for maximum throughput, trying to squeeze every last millisecond out of your event loop. If you're building systems that need sub-millisecond latency, these hidden kernel quirks become absolutely critical.

The Root Cause

Here’s the deal: Node.js (via libuv) uses epoll in edge-triggered mode for non-blocking I/O. When setting up an HTTP/2 server, especially if you have multiple worker processes listening on the same port (e.g., using Node's cluster module implicitly leveraging SO_REUSEPORT), things get weird on older RHEL 7 kernels under cgroup CPU throttling.

The core issue is a subtle race condition related to how these older kernels handle SO_REUSEPORT, epoll edge-triggered notifications, and immediate CPU descheduling. When a Node.js worker receives data on a loopback socket, epoll_wait wakes it up. However, if the cgroup scheduler immediately deschedules the process before it can fully read and drain the socket's receive buffer (which is common with HTTP/2's frame-based protocol, often leading to partial reads), a race condition emerges. On these specific kernel versions, the kernel's internal state machine for epoll combined with SO_REUSEPORT's shared queue mechanism can sometimes fail to re-notify the process (or any other `SO_REUSEPORT` listener) that there's still data available. The edge trigger, having fired once, might not fire again for the 'remaining' data if the kernel incorrectly considers the event 'handled' or if the socket buffer state is ambiguously managed across deschedules and shared listeners.

This effectively means data gets 'stuck' in the kernel's buffer, or subsequent packets are never delivered to the application layer. The application layer waits, times out, or the remote end eventually gives up and sends a RST packet, leading to the dreaded HTTP/2 session error. The loopback interface exacerbates this because the round-trip latency is negligible, making the deschedule window even more critical.

The Fix: Disable SO_REUSEPORT

The most reliable, immediate fix on affected kernels, without upgrading your entire OS, is to explicitly disable SO_REUSEPORT for your Node.js HTTP/2 server. You do this by setting an environment variable.

NODE_CLUSTER_SCHED_POLICY=none node server.js

Or, if you're using Node.js's cluster module programmatically:

const cluster = require('cluster');

if (cluster.isMaster) {
  cluster.schedulingPolicy = cluster.SCHED_NONE; // Explicitly disable SO_REUSEPORT
  // ... rest of master code
} else {
  // ... worker code, create http2 server
}

Why This Works

Setting NODE_CLUSTER_SCHED_POLICY=none tells Node.js (specifically libuv) not to use SO_REUSEPORT. Instead, all incoming connections are accepted by a single listener socket (handled by the master process, or the first worker if no master is explicit) and then explicitly distributed to worker processes using IPC. This entirely bypasses the problematic SO_REUSEPORT shared queue and the race condition it introduces with epoll edge triggers under cgroup CPU pressure on those specific kernel versions.

While SO_REUSEPORT is typically excellent for performance and load distribution across CPU cores, in this specific, obscure scenario, it’s actively detrimental. Removing it centralizes the initial connection acceptance, making the `epoll` event management more predictable and resilient to aggressive CPU descheduling.

Final Thoughts

This is a classic SRE problem: a highly specific interaction between kernel features, runtime behavior, and resource management. It's the kind of bug that makes you appreciate why kernel versioning matters and why blanket 'upgrade everything' isn't always feasible in production. If you can upgrade your kernel to 4.18 or newer (e.g., RHEL 8 or 9), you should. But if you’re stuck on RHEL 7.x, this workaround will save your sanity and your services.