Quick Summary: Diagnose and fix elusive Node.js EAI_AGAIN errors on Alpine containers. Learn how ndots:1 and specific DNS configurations cause intermittent inter...

Alright, listen up. You've probably seen it. That infuriating, intermittent EAI_AGAIN error popping up in your Node.js application logs, usually when trying to connect to some internal service in Kubernetes. It’s a DNS error, sure, but it’s not a consistent failure. Sometimes it works, sometimes it doesn't. You've checked the DNS server IPs, you've cursed the network team, you've even considered a career change. I've been there. This isn't just a minor annoyance; in hyperscale systems, a flaky DNS resolution can cascade into widespread outages. Let's fix this once and for all.

The Problem: Intermittent EAI_AGAIN on Internal Service Lookups

Your Node.js app, running in an Alpine Linux container, tries to resolve my-internal-service.my-namespace. Most of the time, it's fine. Then, out of nowhere, you get:

Error: getaddrinfo EAI_AGAIN my-internal-service.my-namespace

This happens during startup, during API calls, you name it. It's almost always a short, internal service name, often already containing a dot. The frustrating part? A simple kubectl exec -it my-pod -- nslookup my-internal-service.my-namespace often works perfectly fine inside the failing pod. What the hell gives?

Initial Misdirection & Wasted Hours

You’ve done the standard drill: Checked /etc/resolv.conf in the container. Looks fine, points to kube-dns (or CoreDNS) service IP. The search domains are there: my-namespace.svc.cluster.local svc.cluster.local cluster.local. All correct. You’ve bumped up pod CPU/memory, thinking it’s resource contention. Nope. You’ve even redeployed CoreDNS, suspecting flakiness. Still happens. You’re pulling your hair out. You’ve probably Googled “Node.js EAI_AGAIN Kubernetes Alpine” a hundred times, finding only generic advice.

This is where the environment specifics become critical:

Component	Version(s) Where Error Triggers
Operating System	Alpine Linux 3.12, 3.13, 3.14, 3.15
Node.js Runtime	14.x, 16.x, 18.x (any LTS Node.js running on Alpine)
Container Orchestration	Kubernetes 1.18+ (especially with default CoreDNS configs)
DNS Resolver Library	Musl libc (used by Alpine)

The Root Cause

The culprit is a subtle interaction between Alpine's Musl libc DNS resolver, Kubernetes' default DNS configuration for pods, and specifically the ndots option in /etc/resolv.conf. Kubernetes, by default, often sets options ndots:5 in your pod's resolv.conf. This means any hostname with fewer than 5 dots will first have your search domains appended before being tried as an absolute FQDN.

However, for services like my-internal-service.my-namespace (which has one dot), if certain configurations or a dnsPolicy like ClusterFirstWithHostNet are used, or if the ndots value is implicitly or explicitly lowered (e.g., to ndots:1), the Musl resolver's behavior changes drastically. With ndots:1, a name like my-internal-service.my-namespace is considered to have 1 dot. Since it meets or exceeds the ndots threshold, it's first treated as an absolute FQDN (my-internal-service.my-namespace.) and attempted without appending search domains. Only if that lookup fails, does the resolver then fallback to appending the search domains (e.g., my-internal-service.my-namespace.svc.cluster.local).

The issue arises because my-internal-service.my-namespace is not directly resolvable as an absolute FQDN without the .svc.cluster.local part. The initial lookup attempt without search domains will time out or fail. This initial failure/timeout is what triggers the intermittent EAI_AGAIN, especially if your DNS server is under load or slightly delayed. The Musl resolver on Alpine handles this 'first try as FQDN' behavior differently and is less forgiving or slower to fallback than glibc, leading to the observed intermittent failures and timeouts for Node.js's underlying getaddrinfo calls.

The Fix: Force a Higher ndots

The solution is to explicitly force a higher ndots value for your pod, ensuring that your internal service names (which typically have 1-2 dots) are always treated as partial hostnames and have the search domains appended first. We want them to hit my-internal-service.my-namespace.svc.cluster.local directly, without the initial FQDN probe.

You achieve this by overriding the pod's DNS configuration via dnsConfig in your Deployment YAML. We'll set ndots:5, which is standard for Kubernetes and generally safe, but you can adjust based on your deepest internal service names if they contain more than 4 dots.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-node-app
spec:
  selector:
    matchLabels:
      app: my-node-app
  template:
    metadata:
      labels:
        app: my-node-app
    spec:
      containers:
      - name: app
        image: my-alpine-node-image:latest
        ports:
        - containerPort: 3000
      dnsConfig:
        options:
          - name: ndots
            value: "5"

Apply this change, and watch your EAI_AGAIN errors vanish. Just like with the infuriating Node.js TCP_WAIT hell, these seemingly small configuration differences can have catastrophic impacts on your application's network stability.

Crucial Caveats and Best Practices

• Don't just copy-paste blindly: Understand your environment. If you have services that are truly absolute FQDNs (e.g., external domains, or services registered directly without Kubernetes search paths), ensure ndots:5 doesn't negatively impact their resolution.
• Monitor CoreDNS: While this fix addresses the client-side resolver behavior, ensure your CoreDNS/kube-dns is healthy and not actually overloaded. This fix primarily prevents a problematic lookup path, but if the DNS server itself is struggling, you'll still have issues.
• Test thoroughly: Always deploy such changes to a staging environment first.

This problem is a classic example of how minor differences in C library implementations (Musl vs. Glibc) and subtle Kubernetes DNS defaults can combine to create extremely frustrating, intermittent issues in specific environments like Alpine-based containers. Now go, fix your apps, and get some sleep.