Quick Summary: Debugging a specific Node.js DNS timeout when resolving internal .local domains in Kubernetes, caused by resolv.conf search path conflicts and old...

Alright, listen up. If you're a fellow SRE, you know the drill. That moment when an application starts randomly flaking out, slow connections, intermittent timeouts, and the logs are as useful as a chocolate teapot. But this one? This was a special kind of hell. We're talking about Node.js applications in Kubernetes, making internal service calls, suddenly deciding they needed a 30-second coffee break before connecting. Not all calls. Just some. And only sometimes. Classic distributed systems garbage.

The symptoms were clear but the cause was a ghost. Our Node.js services, consuming other internal K8s services (e.g., http://my-internal-svc.my-namespace.svc.cluster.local), would sporadically experience massive connection delays. We'd see EAI_AGAIN or ETIMEDOUT errors after a painfully long pause. External API calls? Flawless. Database connections? Smooth. It was specifically internal service-to-service communication. And it was driving us nuts.

This particular beast primarily rears its ugly head in these exact conditions. Don't waste your time looking elsewhere if your stack doesn't match:

Component	Affected Versions	Notes
Operating System	CentOS 7.x, Ubuntu 20.04 LTS	Specifically older glibc versions (2.28-2.33) and Linux kernel versions pre-5.10.
Node.js	14.x (14.17.0 - 14.20.1) 16.x (16.13.0 - 16.19.1)	Relies on older libuv which in turn uses glibc's getaddrinfo in a specific, problematic way. Node.js 18+ is generally more resilient.
Kubernetes	1.20, 1.21, 1.22	Default CoreDNS and dnsPolicy: ClusterFirst configurations.

Initial Misdirections and Wasted Weekends

We tore our hair out tuning CoreDNS, adjusting ndots values in dnsConfig to extreme levels, even trying hostAliases and dnsPolicy: None. We suspected network policies, CNI plugins, even rogue proxies. Nothing. The problem persisted, mocking us with its randomness. If you've been grappling with similar issues, you know the brutal reality of distributed systems at FAANG-scale often boils down to something incredibly mundane and overlooked.

The Hunt: Why Does getaddrinfo Take 30 Seconds?

After days of strace-ing Node.js processes, tcpdump-ing DNS traffic, and staring at packet captures until our eyes bled, we started seeing a pattern. When a Node.js process would hang, its DNS queries weren't just failing; they were being sent to the wrong places and timing out, sequentially. Each timeout added 5-10 seconds to the overall lookup. It was like watching a snail run a marathon, but the snail kept getting lost.

The Root Cause

The culprit? A nasty interaction between older Node.js (specifically its underlying libuv library's use of getaddrinfo), specific glibc resolver versions, and the Kubernetes-generated /etc/resolv.conf inside the Pod. Here’s the typical problematic resolv.conf:

nameserver 10.96.0.10 # Your Kube-DNS service IP
search my-namespace.svc.cluster.local svc.cluster.local cluster.local eu-west-1.example.com example.com
options ndots:5 timeout:5 attempts:2

See that eu-west-1.example.com and example.com? Those are external, corporate search domains, likely injected by your cluster's initial setup or a sloppy custom configuration. The problem emerges when a Node.js app tries to resolve a short name like my-service. With ndots:5, it first tries to resolve my-service as an absolute FQDN (fails). Then it starts appending the search domains:

1. my-service.my-namespace.svc.cluster.local (Success!)
2. ... but if it also attempts fully qualified names with search domains appended (e.g. my-internal-svc.my-namespace.svc.cluster.local.eu-west-1.example.com), this is where the trouble begins.

Crucially, getaddrinfo, particularly in older glibc and how libuv used it, performs these search path lookups sequentially. When it hits my-service.eu-west-1.example.com, CoreDNS forwards this query. If the external DNS server for example.com doesn't know about my-service.eu-west-1.example.com (which it won't), it will time out. Each timeout:5 and attempts:2 adds 10 seconds of delay for each problematic search entry before the resolver moves on or finally hits the correct internal entry.

This isn't about CoreDNS failing; it's about glibc's resolver logic in Node.js being forced to exhaust irrelevant, externally-bound search paths, one excruciating timeout after another, before it either finds the right answer or gives up. It's a fundamental flaw in how the default glibc resolver processes ambiguous names in complex resolv.conf setups, magnified by slow network responses from external DNS servers for non-existent domains. We've seen similar obscure issues plague container orchestration systems, making the debate around Kubernetes vs. Nomad all the more relevant when you consider such low-level network quirks.

The Fix: Override That Infernal resolv.conf

The immediate, pragmatic fix is to surgically remove the problematic external search domains and tighten the DNS options directly in your Pod's dnsConfig. This gives you absolute control over how /etc/resolv.conf is generated.

Apply this to your Deployment or StatefulSet YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-node-app
spec:
  selector:
    matchLabels:
      app: my-node-app
  template:
    metadata:
      labels:
        app: my-node-app
    spec:
      dnsConfig:
        nameservers:
          - 10.96.0.10 # Your Kube-DNS service IP
        searches:
          - my-namespace.svc.cluster.local
          - svc.cluster.local
          - cluster.local
        options:
          - name: ndots
            value: "1" # Crucial: Treat names with 1+ dots as FQDNs initially.
          - name: timeout
            value: "1" # Fail faster if a query doesn't resolve.
          - name: attempts
            value: "2" # Reduce overall retries.
      containers:
        - name: my-node-app
          image: my-node-app:1.0.0
          # ... other container configuration ...

Why This Works

By explicitly defining searches, you eliminate the external TLDs that were causing the unnecessary, delayed lookups. Setting ndots:1 tells the resolver to treat any name with at least one dot (like my-service.my-namespace) as an FQDN first, reducing the reliance on appending search domains for complex internal names. The reduced timeout and attempts ensure that even if there's a lookup that goes nowhere, it fails much faster, preventing those agonizing 30-second delays.

Final Thoughts

This problem is a perfect storm of legacy glibc behavior, specific Node.js versions, and common Kubernetes network configurations. While upgrading Node.js to 18+ (which uses a more robust, non-blocking DNS resolver) and ensuring your kernel/glibc are up to date is the ultimate long-term solution, this dnsConfig override is your immediate lifeline. Always be vigilant about what's in your resolv.conf. It's often the last place people look, and the first place where obscure network demons hide.