Quick Summary: Node.js apps in Alpine Docker containers hitting `EAI_AGAIN`? This obscure DNS resolution failure is often a Musl libc quirk. Learn the fix.

You've been there. Staring at logs, pulling your hair out. Your Node.js application, humming along in its sleek Alpine container, suddenly starts spewing EAI_AGAIN errors. Intermittent. Maddening. Everything looks fine: DNS servers are up, network connectivity is perfect, other services are happy. Yet, your app keeps choking on hostname lookups.

This isn't your garden-variety ENOTFOUND, where a hostname genuinely doesn't exist. No, EAI_AGAIN screams: "Temporary failure! Try again!" And often, a retry does work. But in a high-volume microservice environment, "temporary failure" means intermittent service degradation, frustrated users, and you debugging a ghost.

We've chased this phantom across countless deployments. The symptoms are always the same: bursts of HTTP 50x errors or internal service connection failures, all correlating with a spike in EAI_AGAIN. You'll see it in your application logs:

Error: getaddrinfo EAI_AGAIN api.downstream-service.internal
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:67:26)

You'll confirm DNS resolution works manually from within the container:

$ docker exec -it <container_id> sh
/ # nslookup api.downstream-service.internal
Server:         127.0.0.11
Address:        127.0.0.11:53

Name:   api.downstream-service.internal
Address: 10.0.0.42

Perfectly fine. So what the hell is going on?

The Frustrating Environment

This particular brand of pain thrives in specific conditions. Check if your setup matches:

Component	Version Range (Where Issue Triggers)
OS/Container Base	Alpine Linux (v3.12 - v3.18)
Node.js	v14.x, v16.x, v18.x (any with standard dns.lookup)
Docker Engine	Any recent version (v20.10+), especially with default bridge network DNS.
Networking	Docker default bridge network, Kubernetes (default CoreDNS settings), high DNS query load.

The Root Cause

Here's the gut-punch reality: it's Musl libc. Alpine uses Musl, not glibc. Musl's DNS resolver, while perfectly valid, handles search domains and the ndots option in /etc/resolv.conf differently under certain conditions, especially under load or with transient network issues. This isn't necessarily a bug, but a behavioral difference that trips up assumptions often made by applications built on glibc.

When your application tries to resolve a hostname like api.downstream-service.internal, the resolver in your Alpine container consults /etc/resolv.conf. Docker, by default, often generates a resolv.conf with a search directive (e.g., search . or your local domain) and a default ndots value. Crucially, it typically includes options timeout:1 attempts:1.

When ndots is non-zero (or effectively acts like it due to search domains), and your hostname doesn't contain enough dots, the resolver will first try to append search domains before trying the name as-is. If any of those initial search queries fail (e.g., a DNS server is slow, the query itself times out quickly due to timeout:1 attempts:1, or the upstream DNS service is under stress), Musl's resolver can be less forgiving than glibc. Instead of gracefully falling back through all search domains or retrying more robustly, it sometimes throws its hands up and returns EAI_AGAIN – a temporary failure to resolve the address information.

This is further complicated by how Node.js's dns.lookup function interacts with the underlying OS resolver. While Node.js has its own DNS resolver, dns.lookup (used by most network operations like HTTP requests) delegates to getaddrinfo, which in turn uses the system's /etc/resolv.conf and libc resolver. It's a subtle interplay, causing headaches for SREs everywhere. If you want to dive deeper into other Node.js-Alpine network quirks, check out The Phantom getaddrinfo ENOTFOUND – a related but distinct issue of full lookup failure versus our temporary EAI_AGAIN.

The Fix: Asserting Control Over DNS Resolution

The solution is to minimize the number of queries and assert more control over how Musl resolves names. We want to tell it: "Don't bother with search domains unless I explicitly tell you." We do this by forcing ndots:0 and explicitly listing robust DNS servers. This makes the resolver treat names with dots as fully qualified directly, reducing the chances of it trying an unnecessary search domain first and hitting an EAI_AGAIN dead-end.

Here’s the most reliable way to enforce this, usually applied at the Docker daemon level or via Kubernetes Pod DNS policies. For Docker, modify your daemon.json (typically located at /etc/docker/daemon.json). If it doesn't exist, create it:

{
  "dns": ["1.1.1.1", "8.8.8.8"],
  "dns-opts": ["ndots:0", "timeout:2", "attempts:3"]
}

Explanation:

• "dns": ["1.1.1.1", "8.8.8.8"]: Explicitly sets your Docker containers to use reliable public DNS resolvers. Replace with your internal, highly available DNS servers if appropriate.
• "dns-opts": ["ndots:0", "timeout:2", "attempts:3"]: This is the critical part.
  • ndots:0: This tells the resolver that any hostname containing at least one dot (like api.downstream-service.internal) should be considered fully qualified. The resolver will attempt to resolve this name directly *first*, without appending any search domains. This drastically reduces the initial query load for multi-part hostnames and prevents Musl's search domain logic from causing an early EAI_AGAIN. For hostnames without dots (e.g., my-service), it will still try to resolve them directly, then fall back to search domains if they exist and direct resolution fails.
  • timeout:2 and attempts:3: While ndots:0 is the primary fix for the Musl EAI_AGAIN, these options provide more robustness. They give the DNS resolver more time (2 seconds per query) and more attempts (3) before failing entirely, helping with genuinely flaky network conditions. This also helps mitigate other networking-related issues, such as those that can lead to Alpine HTTP Agent Drains.

After modifying daemon.json, you must restart the Docker daemon for changes to take effect:

sudo systemctl restart docker

For Kubernetes, you'd typically define a dnsPolicy: ClusterFirst and then use dnsConfig in your Pod spec to override `options` or `nameservers` for that specific Pod.

Verification

Spin up your Node.js application container. Execute a shell inside it and check /etc/resolv.conf:

$ docker exec -it <container_id> sh
/ # cat /etc/resolv.conf
nameserver 1.1.1.1
nameserver 8.8.8.8
options ndots:0 timeout:2 attempts:3

You should see your specified nameservers and, critically, options ndots:0 timeout:2 attempts:3. Monitor your application logs under load. The intermittent, mysterious EAI_AGAIN errors should vanish. You might still encounter very rare, genuine network issues, but the consistent, Musl-induced EAI_AGAIN will be gone.

Don't let obscure libc quirks waste another second of your life. Pinpoint the problem, slap it with a definitive fix, and get back to building amazing things.