Quick Summary: Node.js apps in Docker on CentOS 7 hit EAI_AGAIN resolving sibling containers after Docker restarts. Learn how stale nf_conntrack entries cause th...

Alright, folks. Settle in. You've hit it, haven't you? That infuriating, intermittent EAI_AGAIN error that only shows its ugly head after a host reboot, a Docker restart, or when the wind blows just right. Specifically, when your Node.js application, caged within a Docker container on an old-school CentOS 7 server, decides it can’t find its best buddy, say, the Redis container, sitting right next to it on the same custom Docker bridge network. Yeah, I’ve been there. You’re tearing your hair out because ping redis eventually works, external DNS is fine, but your application code just keeps screaming getaddrinfo EAI_AGAIN redis at startup, preventing your whole stack from coming up clean. It’s a ghost in the machine, and it’s costing you precious uptime and sanity.

This isn't your garden-variety DNS misconfiguration. Oh no. If it were, you'd fix /etc/resolv.conf and move on. This is far more insidious, lurking in the forgotten corners of the Linux network stack, specifically how older kernels manage connection tracking and how Docker tries to play nice with it. You've checked your Docker Compose YAML a dozen times. The service names are correct. You can docker exec -it my-app sh and resolve external domains. You might even ping the sibling container's IP address successfully, but ping by name fails initially. The kicker? It often self-resolves after 5-10 minutes of retries. Completely unacceptable for production systems where every second of startup matters. Reminds me of the headaches we used to face when optimizing for sub-millisecond algorithmic execution – reliability is paramount.

The Environments Where This Specter Haunts You

This particular beast thrives in specific conditions. If your setup matches this table, you're likely in its crosshairs:

Component	Version Range (or specific)	Notes
Host OS	CentOS 7.x, RHEL 7.x	Kernel 3.10.x series is key here.
Docker Engine	18.09.x to 20.10.x	Any version where Docker uses iptables for network setup and internal DNS.
Node.js Application	Any Node.js version (10.x to 18.x)	Applications relying on dns.lookup (default for most HTTP clients like axios, node-fetch) for service name resolution.
Networking	Custom Docker Bridge Network	Containers communicating via service names (e.g., redis, db) within the same bridge.

Initial Frustration & What NOT to Do (Again)

You’ve probably already done all this, but let’s tick the boxes:

• Checked /etc/resolv.conf inside the container: It correctly points to 127.0.0.11. Good.
• Verified Docker network connectivity: docker inspect shows containers on the same network. IPs are reachable.
• Restarted Docker countless times: Only to get the same initial failures.
• Blamed Node.js DNS caching: While Node.js does cache, this issue manifests even on fresh starts.
• Fiddled with container restart policies: This just postpones the pain.

None of that actually solves the core problem. You're just restarting the timer on the same bomb. The real issue is deeper, a perfect storm involving how Docker manages internal DNS resolution and the host's kernel network tracking.

The Root Cause

The culprit is the Linux kernel's nf_conntrack module, specifically its interaction with Docker's internal DNS resolver (which typically runs on the host and proxies requests from 127.0.0.11 inside your containers). On older CentOS 7 / RHEL 7 kernels (3.10.x), the default timeout for TCP connections in the TIME_WAIT state, nf_conntrack_tcp_timeout_time_wait, is often set ridiculously high (e.g., 120 seconds). When you restart Docker (systemctl restart docker) or a host network interface flaps, Docker tears down and rebuilds its iptables rules and internal networking components.

During this tear-down and rebuild, particularly for the DNS requests flowing between your containers (via 127.0.0.11) and the dockerd DNS proxy on the host, the nf_conntrack table can retain stale entries for these connections. Even if Docker has re-established its internal DNS forwarding paths, the kernel’s connection tracker might still be holding onto old, now invalid, entries for DNS traffic destined for the previously active (but now re-initialized) Docker bridge. This creates a brief but critical period of packet loss or misrouting for DNS queries, leading to EAI_AGAIN errors.

Your Node.js app tries to resolve redis. The query goes to 127.0.0.11, then out to dockerd. If nf_conntrack is still holding onto a stale TIME_WAIT entry for that specific DNS flow, the query might be dropped, or routed incorrectly by the kernel, even though Docker's infrastructure is otherwise functional. After the nf_conntrack_tcp_timeout_time_wait expires, these stale entries are finally cleared, and magically, DNS resolution starts working. This explains the infuriating self-healing behavior.

This isn't too dissimilar to the EAI_AGAIN Hell we've seen with phantom iptables rules, but here, the ghost is in conntrack itself.

The Solution: Shorten the Kernel's Memory

The fix is to reduce the nf_conntrack_tcp_timeout_time_wait value on your host system. This forces the kernel to purge stale TIME_WAIT connections much faster, significantly reducing the window of opportunity for EAI_AGAIN errors after a Docker restart or network event. We're essentially telling the kernel: "Forget old connections faster, especially when things are changing under the hood."

Step-by-Step Implementation

1. Check Current Value (Host): Log into your CentOS 7 host machine and check the current setting.

sysctl net.netfilter.nf_conntrack_tcp_timeout_time_wait

You'll likely see something like net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120. That’s 120 seconds, or 2 agonizing minutes of potential DNS failure.

2. Set the New Value (Temporarily): For immediate testing, you can change it on the fly. We'll aim for 30 seconds, which is a common, safe value for most workloads.

sudo sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

Restart Docker and your Node.js application after this change. Observe if the EAI_AGAIN errors persist or if the recovery time is drastically reduced. If your application still struggles, you might try an even lower value like 15, but be cautious with overly aggressive settings as they can impact legitimate long-lived connections, although DNS queries are short-lived. For a detailed discussion on balancing network performance and stability, particularly in complex distributed systems, you might want to review articles like "FlowPilot: Blazing Fast or Blazing a Trail to Production Chaos?", which emphasizes careful configuration.

3. Make it Permanent (Host): To ensure this change survives reboots, you need to add it to a sysctl configuration file. Create or edit /etc/sysctl.d/99-docker-conntrack.conf (or similar).

sudo vi /etc/sysctl.d/99-docker-conntrack.conf

Add the following line:

net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30

Then, apply the configuration:

sudo sysctl --system

Verify it’s active:

sysctl net.netfilter.nf_conntrack_tcp_timeout_time_wait

Final Thoughts: Take Control

This fix addresses a fundamental interaction problem between an older kernel's default network parameter and Docker's dynamic network setup. By reducing the TIME_WAIT timeout, you're giving the system less time to hold onto stale information, allowing Docker's internal DNS to re-establish itself cleanly and quickly after a disruptive event. No more waiting minutes for your Node.js apps to find their database. No more mysterious EAI_AGAIN errors that vanish as soon as you start debugging. Just reliable service startup, every single time. Sometimes, the most obscure problems have the simplest, yet deepest, root causes.