Quick Summary: Unravel the mystery of intermittent Node.js EAI_AGAIN errors under high load. This SRE guide pinpoints the obscure glibc getaddrinfo exhaustion an...
Alright, listen up. You've been there. The pager screams at 3 AM. Your Node.js microservice, which was humming along just fine, is now a flaming dumpster fire of EAI_AGAIN errors. Every single external API call is failing. DNS resolutions? Forget about it. You restart the service, and like magic, it works. For a few hours. Then it's back to square one.
This isn't your garden-variety DNS misconfiguration. Your DNS servers are fine. Your /etc/resolv.conf looks sane. Other services on the same host are resolving names perfectly. You're pulling your hair out, convinced the universe hates your specific Node.js process. Welcome to the club.
This obscure, insidious problem is a deep cut. It manifests only under sustained, high-concurrency workloads. It's a ghost in the machine, a subtle interaction between Node.js, libuv, and the underlying C library's handling of DNS lookups.
The Symptoms
- External API calls (HTTP, gRPC, anything requiring hostname resolution) intermittently start failing en masse.
- Log messages are flooded with
getaddrinfo EAI_AGAIN. - Internal service-to-service communication (e.g., within a Kubernetes cluster using cluster DNS) might remain unaffected, adding to the confusion.
- A simple service restart temporarily resolves the issue, suggesting a transient state or resource exhaustion.
- CPU usage might not be exceptionally high, but network I/O or event loop delays might be noticeable just before the failure cascade.
The Specific Environments Where This Triggers
This isn't universal, thankfully. It typically rears its ugly head in older, or specific containerized environments, interacting with particular Node.js versions.
| Operating System / Base Image | Typical glibc / musl libc Version | Affected Node.js Versions | Common Symptoms |
|---|---|---|---|
| Ubuntu 18.04 LTS (Bionic Beaver) | glibc 2.27 | 16.x, 18.x | Intermittent EAI_AGAIN under sustained load, especially with complex resolv.conf |
| RHEL 7.x / CentOS 7.x | glibc 2.17 | 16.x, 18.x | Similar to Ubuntu 18.04, often exacerbated by containerization layers |
| Alpine Linux 3.12 - 3.14 | musl libc 1.2.x | 16.x, 18.x | More frequent `EAI_AGAIN` due to musl's different getaddrinfo implementation, especially with multiple nameservers or `search` options |
This issue often hits hard when you're managing complex, distributed systems at FAANG scale, where even minor inefficiencies can cascade into major outages. It's the kind of problem that makes you question your life choices as an SRE.
The Root Cause
Here's the punchline: it's not strictly a Node.js bug, nor is it your DNS server. It's a resource exhaustion issue deep within the system's C library (glibc or musl libc) when interacting with libuv's worker thread pool under extreme concurrency.
When Node.js performs a DNS lookup using dns.lookup() for a hostname not in /etc/hosts, it first tries its internal c-ares resolver. However, for many scenarios, especially with hostnames that resolve to multiple IPs, or when c-ares has issues, it falls back to the system's getaddrinfo function. Node.js (via libuv) runs these blocking getaddrinfo calls in its worker thread pool.
The problem arises because glibc's (or musl libc's) implementation of getaddrinfo, particularly in the versions listed above, can hit an undocumented internal limit or suffer from inefficient resource handling when repeatedly invoked at extremely high rates. This is especially true if your /etc/resolv.conf contains multiple nameserver entries, a search domain list, or the options rotate directive. Each call might involve some internal state management, file descriptor operations for /etc/resolv.conf, or locks that aren't designed for thousands of concurrent invocations per second.
Under sustained pressure, glibc's internal getaddrinfo context can temporarily exhaust its resources or hit a slow path, causing it to return EAI_AGAIN – a signal for a temporary failure in name resolution due to resource constraints, not necessarily a non-existent host or an unreachable DNS server. The libuv worker pool might be ready, but the underlying system call is choking.
The Fix: Expand the Worker Pool & Simplify DNS
Since the issue stems from glibc's struggle under high getaddrinfo concurrency, our strategy is twofold: alleviate the pressure on glibc by having more threads ready to process these requests and simplify glibc's job by streamlining resolv.conf.
First, we increase the size of libuv's default worker thread pool. By default, this pool is only 4 threads. For high-concurrency Node.js applications, especially those doing a lot of file I/O or DNS lookups that hit getaddrinfo, this is woefully inadequate. A larger pool means fewer pending getaddrinfo calls waiting for an available thread, which in turn reduces the likelihood of internal glibc resource contention.
Second, we simplify /etc/resolv.conf. If you have multiple nameserver entries or options rotate, remove them if possible. Configure your system to use a local caching resolver (like dnsmasq or systemd-resolved) and point resolv.conf to 127.0.0.1. This ensures glibc only has one, always-available target to query, reducing its internal overhead.
Here’s the copy-pasteable command to set the libuv worker pool size:
# For systemd services, modify your .service file:
# Example: /etc/systemd/system/your-node-service.service
[Service]
Environment=UV_THREADPOOL_SIZE=128
ExecStart=/usr/local/bin/node /app/index.js
# Or, if running directly or via Docker/Kubernetes:
export UV_THREADPOOL_SIZE=128
node /app/index.js
# For Kubernetes deployments, add to your Pod spec environment variables:
# spec:
# containers:
# - name: your-node-app
# env:
# - name: UV_THREADPOOL_SIZE
# value: "128"
Recommended UV_THREADPOOL_SIZE: Start with 64 or 128. The optimal value depends on your workload and number of CPU cores. Don't go excessively high (e.g., 1024) without testing, as it consumes more memory per thread. Remember, for ultra-low latency algorithmic trading architectures, every millisecond counts, and tuning these underlying system parameters is paramount.
Proactive Measures and Monitoring
Simplify /etc/resolv.conf: Ensure it points to a fast, local caching resolver (e.g., 127.0.0.1) and avoid multiple nameserver entries or options rotate. Let your local resolver handle the complexity.
Upgrade Your OS: If feasible, move off Ubuntu 18.04/RHEL 7.x. Newer Linux distributions generally ship with more robust glibc versions that handle concurrency better.
Monitor libuv: Instrument your Node.js application to track libuv's thread pool usage if possible. Look for metrics on pending requests or blocked operations.
This problem is a stark reminder that even seemingly 'application-level' failures can have deep, obscure roots in the operating system's fundamental libraries. Dig deep, question assumptions, and never underestimate the power of resource exhaustion at scale.