Quick Summary: Node.js apps freezing in Docker on Linux with systemd-resolved? Dive deep into the getaddrinfo deadlock and learn the fix for intermittent network...

Alright, listen up. You've got a Node.js service running in Docker. Everything seems fine, then BAM! Intermittent, inexplicable network request hangs. Not timeouts, not connection refused. Just... hangs. Your service goes unresponsive for seconds, sometimes a full minute, then miraculously recovers. CPU looks fine. Memory looks fine. Network metrics? Maybe a tiny blip, but nothing screaming 'problem'. You’re pulling your hair out, blaming the network, the database, a rogue cosmic ray. Sound familiar? Congratulations, you've found the ghost in the machine.

I've been there. Too many times. This isn't your typical networking issue, easily debugged with ping or traceroute. This is a subtle, insidious interaction between Node.js's internal DNS resolution, Docker's default networking choices, and modern Linux distributions' handling of DNS with systemd-resolved. It's a deep-seated architectural flaw that only manifests under specific, aggravating conditions, making it a nightmare to diagnose.

The Symptom: The Intermittent Network Freeze

Your Node.js application, typically a high-throughput API gateway or a background worker making frequent external calls, suddenly stops processing requests. Outbound HTTP calls using libraries like axios or node-fetch stall. Inbound requests stack up, leading to increased latency and eventual 5xx errors. Logs go eerily silent for a period, followed by a burst of delayed entries. It looks like a network bottleneck or resource starvation, but your monitoring says otherwise.

The worst part? It's intermittent, making it impossible to reliably reproduce. If you're lucky enough to catch one of these freezes with advanced profiling tools like strace or perf, you'll see your Node.js process spending an inordinate amount of time—often tens of seconds—in getaddrinfo system calls. This is the critical, undeniable clue: DNS resolution is the culprit, even for hostnames you'd expect to be cached.

The Environments Where This Bites You Hard

This particular beast thrives in modern Linux environments where systemd-resolved is the default local DNS resolver:

Operating System (Host)	Kernel Version (Host)	Node.js Version (Container)	Docker Engine Version
Ubuntu 22.04 LTS (Jammy Jellyfish)	5.15.x and later	16.x, 18.x, 20.x, 21.x	20.10.x and later
Debian 11 (Bullseye) / 12 (Bookworm)	5.10.x and later (Bullseye), 6.1.x and later (Bookworm)	16.x, 18.x, 20.x, 21.x	20.10.x and later
Any Linux distribution using systemd-resolved as its primary DNS resolver	5.10.x and later	16.x, 18.x, 20.x, 21.x	20.10.x and later

The Root Cause

Here’s the deal. Modern Linux distributions rely on systemd-resolved as their local DNS resolver, listening on 127.0.0.53. Docker's default bridge networks often configure containers to point their /etc/resolv.conf to Docker's internal DNS proxy, 127.0.0.11, which then forwards queries to the host's systemd-resolved instance.

Node.js uses a thread pool (libuv's uv_getaddrinfo) for blocking I/O operations like DNS lookups via getaddrinfo. While typically non-blocking for the main event loop, if the system call itself takes an unusually long time to complete—because systemd-resolved is slow—it can tie up a thread. If enough concurrent DNS lookups happen or one lookup is excessively slow, the thread pool can become exhausted, blocking the entire application as other I/O operations also rely on this pool.

When systemd-resolved experiences a hiccup (due to its own caching, upstream latency, or host CPU contention), that blocking getaddrinfo call in Node.js brings your application to a grinding halt. The 127.0.0.53 stub resolver isn't designed for high-concurrency forwarding from dozens of containers. This is a subtle but deadly combination, a variant of the issue discussed in The Phantom DNS Freeze: Node.js, Alpine, and systemd-resolved's Container Conspiracy.

What NOT to Do

Don't fall into the trap of trying to mitigate this within your Node.js application. Adding an in-memory DNS cache or custom dns.lookup wrappers might temporarily mask the symptom but complicates your app and doesn't solve the root problem. This is an infrastructure problem, not an application logic one.

The Fix: Bypass the Stub Resolver Entirely

You need to tell Docker to stop using the host's systemd-resolved stub resolver for your containers. Instead, instruct it to use public, robust DNS resolvers directly. This eliminates the systemd-resolved bottleneck and the 127.0.0.11 Docker proxy, ensuring Node.js gets quick, direct answers, or at least fails fast without blocking its thread pool.

Edit your Docker daemon configuration file, typically located at /etc/docker/daemon.json. If it doesn't exist, create it. Add or modify the dns key:

{
  "dns": ["8.8.8.8", "8.8.4.4", "1.1.1.1"]
}

I recommend using at least two, preferably three, reliable public DNS servers. Google DNS (8.8.8.8, 8.8.4.4) and Cloudflare DNS (1.1.1.1) are excellent choices for stability and performance.

After modifying daemon.json, you absolutely MUST restart the Docker daemon for these changes to take effect. Existing running containers will NOT be affected; only newly created containers will pick up the new DNS configuration. So, plan for a full deployment cycle:

sudo systemctl daemon-reload
sudo systemctl restart docker

Why This Works, And Why It's Critical

By specifying dns in daemon.json, you force all newly created Docker containers to use these public, highly reliable DNS servers directly. Docker injects these into the container’s /etc/resolv.conf, effectively overriding the default behavior of proxying to systemd-resolved.

The container will no longer attempt to use the host's systemd-resolved stub resolver. Node.js sends its DNS queries directly to the specified servers. These are globally distributed, optimized for massive query volumes, and engineered for extreme resilience. This dramatically reduces the chance of getaddrinfo blocking due to a slow upstream resolver, ensuring your libuv thread pool remains unblocked and your Node.js event loop continues to process requests without inexplicable pauses.

This approach might seem heavy-handed, but it’s a robust, pragmatic solution for a common, frustrating issue in containerized Node.js environments. Understanding these underlying interactions and making informed decisions about each layer is critical for architecting for FAANG-scale resilience. Don't trust abstractions blindly; sometimes you need to dig deep and untangle the mess yourself.

Final Thoughts

This problem is a harsh reminder of how complex modern infrastructure has become. What appears as a simple "network issue" can cascade into a deep dive involving kernel versions, systemd intricacies, Docker networking, and Node.js's C++ bindings. Don't waste precious on-call hours tweaking application-level settings that won't address the root cause. Go straight for the jugular: ensure your containers have a reliable, direct path to DNS resolution. You’ll thank me later when your monitoring graphs finally stop showing those inexplicable flat lines.