Quick Summary: Troubleshoot Node.js http.Agent connection exhaustion caused by blocking DNS lookups on older glibc versions. Fix high-concurrency external servic...

Alright, let’s get straight to it. You’ve got a Node.js service, humming along, making thousands of outbound HTTP calls. It’s critical, probably part of some high-throughput data pipeline. Then, without warning, it starts choking. Connections time out. Requests pile up. Eventually, your logs are a sea of ECONNREFUSED or ETIMEDOUT errors to upstream services that you know are perfectly healthy. You restart the Node.js app, and bam, everything's fine for another few hours... or minutes. Sound familiar?

You’ve checked the usual suspects: server load, database health, network latency. You've upped ulimit to insane levels. You've even peered into netstat like it holds ancient secrets, and what do you find? Nothing. No excessive CLOSE_WAIT or TIME_WAIT states. No open files warnings. Just a Node.js process that decided to collectively forget how to make HTTP requests.

This isn't your average network hiccup. This is a subtle, insidious beast that only rears its head under specific, high-concurrency conditions on certain system configurations. And it's almost always related to Node.js's built-in http.Agent and how it interacts with the underlying OS for DNS resolution.

First, confirm you're in the danger zone:

Operating System (glibc version)	Node.js Version (and default `http.Agent` behavior)	Likelihood of Triggering
CentOS 7 (glibc 2.17)	14.x, 16.x, 18.x, 20.x (default `keepAlive: true`)	High
Ubuntu 18.04 LTS (glibc 2.27)	14.x, 16.x, 18.x, 20.x (default `keepAlive: true`)	Medium-High
Debian 9 'Stretch' (glibc 2.24)	14.x, 16.x, 18.x, 20.x (default `keepAlive: true`)	High
Any Linux with glibc < 2.28	All recent Node.js LTS versions	Varies with concurrency & DNS load

The Root Cause

Here’s the deal: Node.js uses its internal http.Agent to manage connection pooling, especially with keepAlive: true, which is the default for many versions. This agent is designed to reuse TCP sockets, saving overhead. When an HTTP request needs a connection, the agent either grabs an idle one from its pool or creates a new one. If it creates a new one, it needs to resolve the hostname to an IP address.

For DNS resolution, Node.js primarily uses the dns.lookup function. Under the hood, dns.lookup (unless explicitly configured otherwise) defaults to calling the operating system’s getaddrinfo, which is part of the glibc library on Linux systems. This is where the whole thing falls apart on older glibc versions (specifically, those prior to 2.28, which introduced non-blocking getaddrinfo for a wider range of scenarios).

On these older systems, getaddrinfo can be a blocking call. Imagine this: your Node.js application is making hundreds or thousands of concurrent requests to various upstream services. Each new hostname requires a DNS lookup. If your DNS server is even slightly flaky, or if the lookup involves external resolving (e.g., to an internet endpoint), that getaddrinfo call can block the Node.js event loop for milliseconds. While this blocking call is happening, the http.Agent's internal state machine gets out of sync.

Sockets that should be returned to the pool after use are momentarily delayed. The agent thinks it has fewer available connections than it actually does. New requests try to establish fresh connections, triggering more blocking DNS lookups, exacerbating the problem. Eventually, the connection pool becomes exhausted, not because actual network resources are depleted, but because the agent's internal state is corrupted by these brief, synchronous delays. Just when you think you've wrestled every beast in the Node.js networking jungle—like the infamous EADDRINUSE kernel trap—another one rears its ugly head.

In complex architectures, where services rely on high-volume, interdependent calls, often orchestrated by data tools like those debated in our take on 'Stream Weaver', such subtle connection issues can cascade into full-blown outages.

The Fix: Bypassing `glibc`'s `getaddrinfo` for Node.js `http.Agent`

The ideal solution is to upgrade your operating system to one with glibc 2.28 or newer. But let’s be real, that's often a major project you can't just drop into production tomorrow.

The immediate, practical solution for Node.js is to force http.Agent to use Node.js's native, asynchronous DNS resolution mechanisms, completely bypassing the problematic glibc getaddrinfo blocking calls for all hostname lookups managed by your agent.

You can do this by providing a custom lookup function to your http.Agent configuration. This function will use Node.js's dns.promises.resolve4 (or resolve6) and then format the result to match what http.Agent expects.

Step-by-Step Implementation:

1. Import necessary modules: You'll need dns, http (or https), and potentially util for promisify.
2. Create a custom lookup function: This function will replace the default DNS resolver used by the agent.
3. Instantiate your http.Agent with the custom lookup.
4. Use this custom agent for all your outbound requests.

Here’s the complete, copy-pasteable code:

const dns = require('dns');
const http = require('http');
const https = require('https');

// We need to provide a custom lookup function for http.Agent.
// This ensures Node.js uses its internal, asynchronous DNS resolver
// rather than glibc's potentially blocking getaddrinfo for every new connection.
const customLookup = (hostname, options, callback) => {
  // Default options for lookup (family, hints, etc.)
  // For simplicity, we'll only resolve IPv4. Adjust '4' for IPv6 if needed.
  dns.promises.resolve4(hostname)
    .then(addresses => {
      // http.Agent expects (error, address, family)
      // Return the first resolved address, assuming IPv4 for family 4.
      if (addresses.length === 0) {
        return callback(new Error(`No IPv4 addresses found for ${hostname}`));
      }
      callback(null, addresses[0], 4);
    })
    .catch(err => {
      callback(err);
    });
};

// Create your custom HTTP/HTTPS agents
// Important: Make sure to set keepAlive: true if you want connection reuse,
// but with this fix, the blocking issue related to DNS should be gone.
const httpAgent = new http.Agent({
  keepAlive: true,
  maxSockets: 100, // Adjust as per your concurrency needs
  lookup: customLookup // This is the critical part!
});

const httpsAgent = new https.Agent({
  keepAlive: true,
  maxSockets: 100, // Adjust as per your concurrency needs
  lookup: customLookup // This is the critical part!
});

// Now, use these agents in your HTTP/HTTPS requests
// Example using 'fetch' (or any http.request call)
// For 'fetch' in Node.js, you'd typically pass the agent in the 'agent' option.
async function makeRequest(url, isHttps = true) {
  const agent = isHttps ? httpsAgent : httpAgent;
  try {
    const response = await fetch(url, { agent });
    if (!response.ok) {
      console.error(`HTTP error! status: ${response.status}`);
      return null;
    }
    const data = await response.json();
    return data;
  } catch (error) {
    console.error(`Request to ${url} failed:`, error.message);
    throw error; // Re-throw to handle upstream
  }
}

// --- How to use with standard http.request --- //
// const options = {
//   hostname: 'example.com',
//   port: 80,
//   path: '/data',
//   method: 'GET',
//   agent: httpAgent // Use your custom agent here
// };

// const req = http.request(options, (res) => {
//   let data = '';
//   res.on('data', (chunk) => data += chunk);
//   res.on('end', () => console.log(data));
// });

// req.on('error', (e) => console.error(`Problem with request: ${e.message}`));
// req.end();

// --- Example usage with fetch (requires Node.js 18+ or a polyfill) --- //
// (async () => {
//   try {
//     const httpResult = await makeRequest('http://httpbin.org/get', false);
//     console.log('HTTP Result:', httpResult);

//     const httpsResult = await makeRequest('https://jsonplaceholder.typicode.com/todos/1');
//     console.log('HTTPS Result:', httpsResult);
//   } catch (err) {
//     console.error('An error occurred during example usage:', err);
//   }
// })();

By implementing this custom lookup function, you are effectively telling Node.js to use its highly optimized, non-blocking internal DNS resolver for all outbound connections managed by these agents. This circumvents the glibc blocking issue, allowing the http.Agent to manage its connection pool correctly, even under extreme concurrency and varying DNS response times.

This fix has saved countless hours of debugging and averted numerous outages in high-load Node.js services running on older infrastructure. It’s a classic example of a subtle OS-level interaction causing cascading failures in a higher-level application framework. Don't let these ghost-in-the-machine problems haunt your production systems.