Quick Summary: Debugging an elusive ECONNRESET or ETIMEDOUT in Node.js Docker containers on Linux kernel 5.4.x. Resolve phantom internal service timeouts with sy...

Alright, listen up. I've been doing this SRE gig for longer than most of you have been alive. I've seen things. But the last few weeks felt like I was back in '98, wrestling with a dial-up modem connected to a Java applet that somehow also needed ActiveX. Pure, unadulterated frustration.

The problem? Intermittent, infuriating HTTP timeouts and connection resets (ECONNRESET, ETIMEDOUT) in a Node.js microservice. Not to external APIs, mind you. Those were humming along. Only to internal services, deployed right next door in the same Kubernetes cluster, same Docker network. And only under moderate-to-high load. Below a certain RPS, everything was peachy. Above it? Random connection drops that would make you want to throw your monitor into the nearest datacenter rack.

It was a ghost in the machine. A phantom ECONNRESET that defied logic. The requests wouldn't even reach the upstream service's logs. It was dying somewhere in the network ether, but only for our specific Node.js containers.

We tore it apart. tcpdump on the host, tcpdump in the container. Network traces showed SYN, SYN-ACK, sometimes even a data packet, then... silence, or an abrupt RST. No clear culprit. We checked Docker's network config, Kubernetes service definitions, CoreDNS logs. Nothing. Everything seemed textbook. We even toyed with Node's http.Agent keepAlive settings, thinking it was connection reuse gone wrong. Didn't matter.

The true horror? If we ran the Node.js application directly on the host, no Docker, the problem vanished. If we deployed it on a VM with a different, older kernel (like 4.15.x) or a much newer one (like 5.10.x), also fine. This screamed 'kernel version specific network weirdness'.

The Environments Where This Bites You

Component	Version Range	Notes
Operating System	Ubuntu 20.04 LTS, CentOS 8	Any system primarily running kernel 5.4.x
Linux Kernel	5.4.0-x-generic (e.g., 5.4.0-100-generic to 5.4.0-150-generic)	The sweet spot for this particular nightmare.
Container Runtime	Docker 19.03+, containerd 1.4+	Standard bridge networking.
Node.js	14.x, 16.x	Impacts any Node.js app using default http.Agent keep-alive.

The Root Cause

After far too many hours staring at conntrack -L output and kernel mailing list archives, we finally hit it. This isn't just a Node.js thing, or a Docker thing. It's a subtle, nasty interaction between specific Linux kernel 5.4.x versions, Docker's netfilter rules for bridge networking, and Node.js's default http.Agent connection reuse. Specifically, the kernel's default handling of the FIN_WAIT_2 TCP state for connection tracking.

In these kernel versions, under certain load conditions and with rapid connection turnover (common with microservices and keep-alive agents), the netfilter connection tracking (conntrack) module could prematurely expire or mishandle entries in the FIN_WAIT_2 state. When Node.js's http.Agent tries to reuse a socket for a new request immediately after a FIN-ACK sequence from the previous one, the kernel, via conntrack, gets confused. It might still have a stale entry or misinterpret the state, leading it to drop subsequent packets for the *new* connection as if they belonged to the half-closed, aged-out 'old' connection. The result? ECONNRESET or ETIMEDOUT for what appears to be a perfectly valid new request.

It's a race condition exacerbated by Docker's NAT and the ephemeral port dance. This kind of obscure network issue is precisely why a deep understanding of infrastructure is paramount, especially when you're aiming for microsecond mastery in your network interactions or trying to scale a truly resilient system. It's not just about application code; it's about the very plumbing underneath.

The Fix: The Hammer You Needed

The solution, in hindsight, is deceptively simple. You need to tell the kernel to be less aggressive about pruning `FIN_WAIT_2` entries in netfilter conntrack. We need to increase the timeout for that state.

sysctl -w net.netfilter.nf_conntrack_tcp_timeout_fin=60

Run this command on your host system, not inside the Docker container. This ups the timeout from the default (often 30 seconds) to 60 seconds. This seemingly minor tweak gives the kernel's conntrack module enough breathing room to properly clean up and avoid prematurely dropping packets for new connections.

When you're fighting these types of low-level network ghosts, it really underlines the challenges of surviving hyper-growth in distributed systems. Every layer introduces potential subtle interactions.

Long-Term & Persistence

For persistence across reboots, add this line to a file in /etc/sysctl.d/, for example, /etc/sysctl.d/99-conntrack-fix.conf:

net.netfilter.nf_conntrack_tcp_timeout_fin = 60

Then run sysctl -p /etc/sysctl.d/99-conntrack-fix.conf or simply sysctl --system to apply it without rebooting. If you're running managed Kubernetes, you might need to apply this as a DaemonSet with privileged access or via a Node Feature Discovery (NFD) operator that allows kernel parameter tuning.

This issue is largely resolved in newer kernel versions (5.10+), so upgrading your hosts is the ultimate long-term solution. But if you're stuck on 5.4.x for compatibility or enterprise release cycles, this is your immediate lifeline.

So, there you have it. Weeks of head-scratching, resolved by a single sysctl command. It's the kind of battle scar that makes you a veteran. Don't let the phantom ECONNRESET haunt your production.