Quick Summary: Debugging Node.js network hangs in Docker on CentOS 7/RHEL 7. Uncover the obscure `dns.lookup` IPv6 timeout when host IPv6 is disabled. Fix `getad...

Alright, another Tuesday, another 'unexplained' production incident. You’ve got a Node.js service, rock-solid for months, suddenly flailing. Intermittent network hangs, phantom EAGAIN errors, CPU spikes out of nowhere. No code changes. No apparent infrastructure shift. Just… pain. You’re tearing your hair out. The logs are useless, pointing everywhere and nowhere. Sound familiar? Good. Because I’ve seen this exact hellscape before. This isn't your average 'memory leak' or 'unhandled promise rejection'. This is deeper. More insidious. A ghost in the machine born from an unholy alliance of old kernels, Docker, and Node.js’s well-intentioned but sometimes naive network stack.

First, let's nail down where this particular brand of misery thrives. This isn't universal. It’s a very specific cocktail of versions that creates this perfect storm. Ignore this table at your peril.

You’ve checked everything. Network engineers swear up and down their firewalls are fine. df -h shows ample disk space. Memory looks okay. Yet, your application, typically handling hundreds of requests per second, just… freezes. Requests pile up. Load balancers start draining. You restart the container, and it's fine for a bit, then it's back. Sometimes, it’s a quick 5-second stall. Other times, it's a full-blown meltdown requiring a bounce. The worst part? It's often during peak load, or worse, during some background task that does a burst of network calls. You’re probably staring at strace output, seeing futex calls or sendto just hanging, waiting for something that never comes.

When you hit this wall, standard debugging is a joke. pm2 logs? Useless. top? Maybe showing high CPU on an unexpected node process, or nothing at all while everything is stalled. You need to go lower. Way lower. Start with strace -p <pid> on the affected node process inside the container. You might see getaddrinfo calls taking an eternity, or getting stuck. Also, try lsof -p <pid> to see open file descriptors. Are there an unusual number of DNS related entries hanging open, or sockets in a weird state? If you see timeouts related to DNS lookups, even when your DNS server is demonstrably fine, you're getting warm.

You might think, 'How can a single network lookup stall a whole application?' Well, in systems designed for high throughput, even a small, unexpected latency in a foundational operation like DNS resolution can cascade into a full-blown outage. We've seen similar unexpected bottlenecks cripple systems even in environments striving for hyper-scale architectures.

The Root Cause

Here’s the deal: Node.js, by default, uses dns.lookup for name resolution, which ultimately relies on the underlying system’s getaddrinfo C function. By default, dns.lookup uses family: 0, meaning it tries to resolve both IPv4 and IPv6 addresses. On older Linux kernels (pre-4.x, common in CentOS 7 or RHEL 7, especially if not fully patched), combined with certain glibc versions and Docker’s network virtualization, something breaks. If your host system has IPv6 explicitly disabled (e.g., net.ipv6.conf.all.disable_ipv6 = 1 in /etc/sysctl.conf or aggressive ip6tables rules), the getaddrinfo call within the container can get utterly confused. It attempts the IPv6 lookup, gets no meaningful response, and instead of failing fast with EAI_AGAIN or EAI_FAIL, it just… hangs. For seconds. Sometimes tens of seconds. This happens on one of libuv's internal thread pool threads used for blocking I/O (like DNS lookups). If all threads in that pool get tied up waiting for phantom IPv6 lookups, your entire Node.js event loop grinds to a halt. Everything stalls. You see high CPU because it’s trying to process the backlog, or no CPU because it’s just waiting.

The good news? The fix is relatively simple, once you’ve spent a week of your life debugging this obscure nonsense. You need to tell Node.js to stop bothering with IPv6 DNS lookups if you know your environment doesn't support it or if it's causing issues. There are two primary ways to do this, depending on your Node.js version and how global you want the fix to be. For most modern Node.js versions (12+), dns.setDefaultResultOrder('ipv4first'); is a good start. But for bulletproof, 'I never want to see this again' stability in these environments, force IPv4.

Set this environment variable before your Node.js application starts:

NODE_OPTIONS="--dns-result-order=ipv4first" node your-app.js

Alternatively, if you're stuck on an older Node.js or want to be super explicit in code (though less recommended for a global fix like this):

const dns = require('dns');
dns.setDefaultResultOrder('ipv4first'); // Or 'verbatim' followed by family: 4 in lookups
// For specific HTTP/HTTPS requests, you can often pass 'family: 4' in options
// e.g., http.request({ ..., family: 4 })
// But the global option is usually preferred for consistency.

The --dns-result-order=ipv4first option (available since Node.js 12.0.0, stable in 16+) makes Node.js prioritize IPv4 addresses. If it finds one, it uses it and doesn't bother with IPv6. This largely mitigates the issue, as the problematic lookup will be skipped or resolved quickly. For absolute certainty, some environments might even prefer NODE_OPTIONS="--dns-result-order=verbatim" combined with explicitly setting family: 4 in dns.lookup or http.request options, but ipv4first is often enough.

Understanding these low-level interactions is crucial. It highlights why relying purely on default runtime behaviors, even in supposedly 'modern' environments, can bite you hard. While new runtimes like Bun are gaining traction, the core principles of network interaction and OS syscalls remain. Always know what your runtime is doing under the hood, especially concerning I/O.

This particular issue is a prime example of how layered abstractions can hide critical problems. Docker abstracts the OS, Node.js abstracts libuv, libuv abstracts getaddrinfo. Each layer adds convenience, but also potential obscurity. When debugging, you often have to peel back every single one. Save yourself the headache. If you're on older kernels and seeing network strangeness, check your IPv6 configuration and apply this fix. You'll thank me later. Or curse the ghosts that haunt your network stack. Either way, now you know.