Quick Summary: Fix Node.js 16+ Docker containers failing DNS resolution (`ENOTFOUND`) on older Linux kernels under high load. Debug `systemd-resolved` AF_UNIX so...

You’ve hit it. That infuriating wall where your seemingly robust Node.js application, humming along in Docker, suddenly starts throwing getaddrinfo ENOTFOUND errors. Not all the time, just when things get spicy. Under load. And only for internal service names. External lookups? Sometimes they work, sometimes they don’t. You’re tearing your hair out because nothing in your app code changed. This isn't a networking misconfiguration, not entirely. It's worse. It’s an obscure dance between Node.js, an older Linux kernel, and systemd-resolved.

I’ve seen this exact nightmare scenario play out too many times. Developers blame Docker, ops blame the app, and everyone points fingers while production grinds to a halt. Stop the blame game. We’re going to fix it.

The Problem: Intermittent ENOTFOUND on Internal Services

Your Node.js 16+ microservice, deployed in Docker, occasionally fails to resolve internal hostnames (like my-backend-service). Errors manifest as getaddrinfo ENOTFOUND or EAI_AGAIN. This behavior is inconsistent but reliably triggered by increased request volume or concurrent connections. Restarting the container or even the Docker daemon sometimes provides temporary relief. Crucially, the issue often vanishes if you run the same Node.js code outside Docker or on a newer OS.

If you’ve checked your /etc/resolv.conf inside the container and it looks sane (e.g., pointing to 127.0.0.11), and your Docker network is configured correctly, you’re in the right place. This isn’t a simple DNS server misconfig. It’s deeper.

Affected Environments

This particular beast thrives in specific conditions. Here’s where you’re most likely to encounter it:

Operating System / Kernel	Node.js Version	Docker Daemon Version	Systemd-resolved State
Ubuntu 18.04 LTS (Kernel < 5.3)	16.x, 18.x, 20.x+	19.03.x - 20.10.x	DNSStubListener=yes (default)
CentOS 7.x (Kernel < 4.18)	16.x, 18.x, 20.x+	19.03.x - 20.10.x	DNSStubListener=yes (default)
Debian 10 (Buster, Kernel < 5.0)	16.x, 18.x, 20.x+	19.03.x - 20.10.x	DNSStubListener=yes (default)

Note: Newer kernels (5.3+) or hosts running a Docker daemon version that ships with a more robust runc might mitigate this, but it's not a guarantee. You might also encounter similar network issues related to ephemeral ports, which can sometimes masquerade as DNS problems. For those, check out "EADDRNOTAVAIL in Docker: The Ghost of Ephemeral Ports Haunting Node.js Microservices" for another deep dive.

The Root Cause

The core problem lies in how Node.js versions 16 and above (specifically due to changes in libuv's internal resolver behavior) interact with systemd-resolved on hosts with older Linux kernels, especially within Docker. By default, systemd-resolved provides a DNS stub listener on 127.0.0.53:53. Docker, when configured with default network settings, routes DNS requests from containers to 127.0.0.11, which acts as a proxy to the host's configured DNS, often systemd-resolved's stub listener.

When systemd-resolved is running, it also exposes an AF_UNIX socket at /run/systemd/resolve/stub-resolv.conf. Node.js's underlying libuv, particularly with newer versions, might sometimes prefer or fall back to this AF_UNIX socket for DNS resolution, especially under specific conditions or for certain types of lookups. The problem is that on older Linux kernels, under heavy, rapid-fire connection attempts to this specific AF_UNIX socket, a resource exhaustion or a race condition within systemd-resolved itself (or its interaction with the kernel's IPC mechanisms) can occur. This leads to the socket temporarily rejecting connections with ECONNREFUSED, which Node.js translates into an ENOTFOUND error.

This isn't a bug in Node.js per se, but an exposure of a brittle interaction at the OS/systemd level that Node.js's updated resolver path triggers more readily. It's a concurrency bottleneck on a critical system component.

The Fix: Force TCP/UDP Resolution for Node.js

Since we can't reliably upgrade every kernel or rewrite systemd-resolved, the simplest, most effective workaround is to force Node.js to bypass the potentially flaky AF_UNIX socket interaction. We do this by instructing Node.js to use standard UDP/TCP DNS resolution directly, effectively making it less reliant on the host's systemd-resolved stub listener's AF_UNIX interface.

This is achieved by setting a specific environment variable for your Node.js application. Add NODE_OPTIONS='--dns-result-order=ipv4first' to your Node.js process, and more importantly, ensure your Docker container's /etc/resolv.conf points directly to a reliable, external DNS server or your Docker daemon's internal DNS (if configured robustly) instead of relying on the host's 127.0.0.11. Better yet, specify a strong DNS directly in your Docker Compose or Docker run command.

Here’s the copy-pasteable Docker Compose snippet:

version: '3.8'
services:
  your_node_app:
    image: your_org/your_node_app:latest
    environment:
      - NODE_OPTIONS=--dns-result-order=ipv4first
      # Optional, but highly recommended if 127.0.0.11 is problematic
      # This directs container DNS queries to Google's public DNS or your custom DNS server
      # Remove if you have a robust internal DNS or Docker's default works for most things.
      # You might use your internal DNS server IP here (e.g., 10.0.0.2)
    # If running Docker daemon < 20.10.0, consider specifying an explicit DNS
    # This ensures resolv.conf within the container uses these specific servers
    # Overrides 127.0.0.11 if specified here.
    dns:
      - 8.8.8.8
      - 8.8.4.4
    # Ensure your container's DNS is actually pointed here.
    # For older Docker versions/hosts, `dns` directive is key.
    # For robust multi-service orchestration, consider specific DNS servers to avoid such pitfalls.
    # This echoes advice often given for building resilient systems, much like what you'd find in
    # articles discussing complex pipelines such as "n8n's Apex: Building a Resilient Multi-Service Orchestration Pipeline".

For a standalone docker run command, it would look like this:

docker run -e NODE_OPTIONS='--dns-result-order=ipv4first' \
           --dns 8.8.8.8 --dns 8.8.4.4 \
           your_org/your_node_app:latest

Why This Works

By specifying NODE_OPTIONS='--dns-result-order=ipv4first', we guide Node.js's internal resolver to prioritize IPv4 address lookups using standard UDP/TCP mechanisms, effectively reducing its reliance on libuv's potential interactions with the AF_UNIX socket for systemd-resolved. The explicit dns: configuration in Docker Compose or --dns in docker run forces the container's /etc/resolv.conf to point to reliable, external DNS servers, bypassing the host's systemd-resolved stub listener entirely. This dual approach cuts off the problematic interaction at its source.

Final Thoughts

This issue is a prime example of how layered abstractions can hide insidious problems. What appears to be a simple DNS lookup failure is, in reality, a complex interaction between a specific Node.js runtime, an older kernel’s IPC handling, and systemd-resolved’s internal mechanisms. Don’t waste days debugging network policies or application code when the problem lives deeper. Pinpoint the environment, apply the fix, and move on. You’ve got bigger fish to fry.