Quick Summary: Node.js applications freezing or timing out on DNS lookups? This guide dissects the obscure `getaddrinfo` hang on older Linux `glibc` and offers a...

The Phantom DNS Hang: When Node.js `getaddrinfo` Chokes on Ancient Linux

Alright, listen up. You’ve been there. Your Node.js service, humming along beautifully, suddenly starts throwing intermittent EAI_AGAIN, ETIMEDOUT, or outright connection freezes. It looks like DNS. You spend hours debugging your DNS servers, validating /etc/resolv.conf, cursing at network engineers, and restarting everything. Nothing. The problem persists, an elusive ghost in the machine, striking only under specific load patterns or after prolonged uptime. It’s infuriating. It’s inefficient. It’s what happens when Node’s getaddrinfo hits a brick wall on an older Linux.

This isn't your average DNS issue. This isn't your DNS server failing. This is far more insidious: a subtle, almost imperceptible hang within the client-side DNS resolution library itself – specifically, how glibc's getaddrinfo function interacts with Node.js via libuv on certain legacy environments. It's a race condition, a deadlock, or just plain old inefficiency in how resource lookups are handled internally. Your application isn't resolving hostnames, and it’s blocking. Badly.

Symptoms and Misdiagnoses

You’ll see a surge in errors like:

• Error: getaddrinfo EAI_AGAIN domain.com
• Error: connect ETIMEDOUT 123.45.67.89:443 (often preceded by a long delay from the DNS resolution phase)
• Application threads appearing to 'hang' or 'freeze' for several seconds, then recovering, only to repeat the cycle.

Your initial reaction (and mine, honestly, every single time) is to blame everything *but* Node.js or the OS. You'll:

• Check DNS server reachability (dig, nslookup, host). All clear.
• Inspect /etc/resolv.conf. Looks fine.
• Verify nscd or systemd-resolved status. Maybe restart it? Helps for a bit, then back to square one.
• Run strace on your Node.js process. You'll see futex calls, then getaddrinfo, and then... nothing for an uncomfortably long time. It’s blocked.

This phantom hang primarily manifests in environments running older Linux distributions with specific glibc versions, especially when your Node.js application is making a high volume of concurrent outbound HTTP requests or connecting to many diverse endpoints.

The Unholy Alliance: Environments Where This Strikes

Here's where this particular brand of hell thrives:

Component	Versions Where Issue is Common
Operating System	CentOS 7.x, Ubuntu 16.04/18.04, Debian 9 (Stretch)
Linux Kernel	Versions < 4.15
glibc Library	2.17 - 2.27 (especially problematic: 2.17, 2.23, 2.25)
Node.js Runtime	12.x LTS, 14.x LTS, 16.x LTS (current & older patch versions)

The combination of these older components, often stuck in enterprise environments, creates a perfect storm for this subtle resolution deadlock.

The Root Cause

Node.js, under the hood, uses libuv for its asynchronous I/O operations. When your application calls functions like net.connect() or http.request(), libuv internally dispatches DNS resolution (i.e., getaddrinfo) to a thread pool. The problem isn't usually in libuv itself, but in how glibc's getaddrinfo behaves in specific older versions.

glibc's getaddrinfo is a complex beast. It relies on nsswitch.conf (Name Service Switch) to determine how to resolve hostnames (e.g., from files, then dns). On older glibc versions, or systems where nscd (Name Service Caching Daemon) or systemd-resolved is misconfigured, getaddrinfo can experience internal deadlocks or excessively long blocking periods when under heavy concurrent requests, especially if it has to parse /etc/resolv.conf repeatedly or interact with a struggling local caching daemon. These blocks can be subtle, lasting just long enough to trigger timeouts or appear as application freezes. The thread handling the resolution simply gets stuck, waiting for a resource that isn't freeing up fast enough, or it hits an internal `glibc` bug that causes it to spin or pause.

When this happens, the Node.js application's worker thread (from libuv's pool) becomes unresponsive, effectively blocking any other DNS resolutions or CPU-bound tasks assigned to that thread. Since the default http.Agent relies on this mechanism, all outbound HTTP traffic eventually grinds to a halt.

The Surgical Strike: Bypassing the Problematic Resolver

The solution isn't to fix glibc (unless you enjoy recompiling system libraries on production servers, which you don’t). It's to make Node.js bypass glibc's problematic getaddrinfo for its critical outbound connections. You achieve this by providing a custom lookup function to Node's http.Agent that uses a more robust or direct DNS resolution method.

Step-by-Step Fix

1. Install a Reliable DNS Resolver Library: We'll use the dns-lookup-cache package, which provides a drop-in replacement for the default dns.lookup, but with sensible caching and more robust error handling that mitigates the glibc issue.

npm install dns-lookup-cache

2. Integrate into Your Node.js Application: The trick is to replace the default lookup function used by the global http.Agent (and https.Agent). This ensures all your outbound HTTP/HTTPS requests use the new, more resilient resolver.

Here’s the complete, copy-pasteable configuration override you need to implement early in your application's bootstrap process (e.g., in your main app.js or server.js file, before any outbound requests are made):

const http = require('http');
const https = require('https');
const dns = require('dns');
const { lookup } = require('dns-lookup-cache')({ maxAge: 30000, cache: new Map() });

console.log('[SRE HACK] Overriding default http/https agent DNS lookup with dns-lookup-cache...');

const originalLookup = dns.lookup;

function customLookup(hostname, options, callback) {
    if (typeof options === 'function') {
        callback = options;
        options = {};
    }
    if (hostname === 'localhost' || hostname.match(/^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/)) {
        originalLookup(hostname, options, callback); // Bypass cache for localhost and IPs
    } else {
        lookup(hostname, options, callback); // Use cached lookup for external hostnames
    }
}

http.globalAgent.options.lookup = customLookup;
https.globalAgent.options.lookup = customLookup;

// If you are using custom Agents, you'll need to set this on those too.
// E.g., for custom agents:
// const myHttpAgent = new http.Agent({ /* ... */ });
// myHttpAgent.options.lookup = customLookup;

Why This Works

By overriding http.globalAgent.options.lookup and https.globalAgent.options.lookup, you're telling Node.js to use your custom lookup function (powered by dns-lookup-cache) instead of its default resolution path which relies heavily on dns.lookup, which in turn leverages libuv's wrapper around glibc's getaddrinfo. The dns-lookup-cache library either uses Node.js's dns.resolve directly (which bypasses getaddrinfo for CNAME/A/AAAA records and provides more control) or wraps the default dns.lookup with intelligent caching and retry mechanisms, preventing the glibc hangs from propagating.

This fix doesn't directly solve the glibc bug, but it circumvents it entirely from Node.js's perspective, providing a stable, performant DNS resolution path even on ancient, temperamental Linux systems. You get the stability you need without waiting for OS upgrades that might never come.

Final Thoughts

Debugging issues like this is a grueling reminder that reliability often means digging through layers of abstraction, from your application code down to the operating system's core libraries. It's not glamorous, but it's essential. Implement this fix, and watch those phantom DNS hangs disappear. Your incident response pager will thank you.