Quick Summary: Troubleshoot Node.js HTTP Agent ECONNREFUSED from stale DNS lookups on RHEL 7.x with dynamic hostnames. Learn to bypass glibc's getaddrinfo caching.

Alright, listen up. You've been there. Staring at an ECONNREFUSED error from your Node.js app. The service it's calling? Up and running. You dig the hostname from the same server, and it resolves to the correct IP. Yet, your application is flailing, desperately trying to connect to a non-existent host. Sound familiar? Welcome to my world. This isn't your average 'DNS cache is stale' problem. This is a special flavor of hell, particularly if you're stuck on RHEL 7 with Node.js.

We hit this wall migrating a critical microservice. Performance requirements meant we needed HTTP keepAlive, but after every deployment of the backend service (which caused its IP to change), our Node.js app would spend a maddening 5-10 minutes sporadically failing to connect. It was a race against time, hitting an old, dead IP while dig, bless its heart, gave us the golden truth.

The Problem: Intermittent ECONNREFUSED to Dynamically Changing IPs

Your Node.js application, running on specific RHEL 7 configurations, using http.Agent (often implicitly) and making requests to hostnames whose IP addresses can change (think Kubernetes service IPs, auto-scaling instances, or blue/green deployments). After a backend IP change, your app sporadically throws ECONNREFUSED or ETIMEDOUT. What's infuriating is that a manual dig hostname or nslookup hostname from the same server works perfectly, returning the correct, new IP.

You've cleared caches, restarted services, maybe even rebooted the damn server out of desperation. Nothing. The problem persists for a frustrating period, then magically resolves itself, only to reappear with the next IP change. This isn't a simple DNS TTL issue; it's deeper.

Affected Environments

This particular beast tends to rear its head in the following combinations. Be especially wary if you match this table:

Operating System	Node.js Version	Affected Configuration / Symptom
Red Hat Enterprise Linux 7.x (e.g., 7.5-7.9)	14.x, 16.x (LTS)	Dynamic hostnames, http.Agent in use, nscd daemon absent or disabled. Intermittent ECONNREFUSED / ETIMEDOUT.
CentOS 7.x	14.x, 16.x (LTS)	Similar to RHEL 7.x, especially in minimal installations lacking name service caching.
Ubuntu 18.04 LTS	14.x, 16.x (LTS)	Less common, but can occur if systemd-resolved is misconfigured or bypassed, forcing reliance on older glibc resolver behaviors.

The Root Cause

The core issue lies in how Node.js's default dns.lookup function interacts with the underlying operating system's resolver, specifically getaddrinfo from glibc. On older RHEL 7.x distributions, particularly in environments where nscd (Name Service Caching Daemon) is explicitly disabled, not installed, or simply not functioning correctly, getaddrinfo can exhibit an insidious form of inconsistent or excessively 'sticky' caching behavior for dynamically changing DNS records. This is critical for systems demanding ultra-low latency trading APIs where every millisecond of connection stability counts.

Even when Node.js's http.Agent needs to establish a new connection, triggering a fresh DNS lookup, getaddrinfo might, on occasion, return a stale IP address from its internal, undocumented, or poorly managed cache. This happens despite the official DNS record having a short TTL and direct tools like dig showing the correct, updated IP. Node.js's dns.lookup effectively becomes a victim of glibc's resolver oddities. The http.Agent then proceeds to dutifully attempt a connection to this stale IP, resulting in the dreaded connection refused errors, and throwing a wrench into your scaling playbook for distributed systems.

The Fix: Bypass `dns.lookup` with `dns.resolve4`

Since dns.lookup is the problematic middleman, we cut it out. Node.js provides dns.resolve4 (and dns.resolve6 for IPv6) which performs a direct network-level DNS query, bypassing the OS's getaddrinfo and its associated headaches. We'll inject this into our http.Agent.

Here’s how you override the lookup behavior. You need to create a custom Agent and pass it to your requests:

const http = require('http');
const dns = require('dns');

// Create a custom lookup function that always uses dns.resolve4
// This bypasses glibc's getaddrinfo and Node.js's default dns.lookup cache issues.
const customLookup = (hostname, options, callback) => {
  // If options is a function, it's the callback
  if (typeof options === 'function') {
    callback = options;
    options = {};
  }

  dns.resolve4(hostname, (err, addresses) => {
    if (err) {
      return callback(err);
    }
    // dns.resolve4 returns an array of IPs. dns.lookup expects a single IP and family.
    // We'll just take the first one and assume IPv4 (family 4).
    if (addresses.length === 0) {
      return callback(new Error(`No IPv4 addresses found for ${hostname}`));
    }
    callback(null, addresses[0], 4);
  });
};

// Create a custom HTTP Agent configured with our custom lookup function.
// Set keepAlive to true if you still want persistent connections.
const customHttpAgent = new http.Agent({
  keepAlive: true,
  maxSockets: 100, // Adjust as per your concurrency needs
  maxFreeSockets: 10, // Adjust if you need to be more aggressive with socket purging
  lookup: customLookup // THIS IS THE CRITICAL LINE
});

// Example usage: Make a request using the custom agent
function makeRequestToDynamicService() {
  const options = {
    hostname: 'your-dynamic-service.example.com',
    port: 80,
    path: '/api/status',
    method: 'GET',
    agent: customHttpAgent // Use our custom agent here
  };

  const req = http.request(options, (res) => {
    console.log(`STATUS: ${res.statusCode}`);
    res.setEncoding('utf8');
    res.on('data', (chunk) => {
      console.log(`BODY: ${chunk}`);
    });
    res.on('end', () => {
      console.log('No more data in response.');
    });
  });

  req.on('error', (e) => {
    console.error(`Problem with request: ${e.message}`);
  });

  // Write data to request body
  req.end();
}

// Call it periodically, or on service start, etc.
// makeRequestToDynamicService();

Implementation Notes

• Global Agent Override (Use with Caution): You could theoretically replace the global http.globalAgent, but I strongly advise against it. It's a shotgun approach that might break other parts of your application or third-party modules that rely on standard dns.lookup behavior. Explicitly passing the agent to your requests is safer and more targeted.
• IPv6 Consideration: If your dynamic service uses IPv6, you'll need to use dns.resolve6, or combine both resolve4 and resolve6 in your customLookup, returning the appropriate family.
• Error Handling: The customLookup function must correctly handle DNS errors and pass them back via the callback.
• Agent Reuse: If you use multiple dynamic hostnames, you might consider creating a map of customHttpAgent instances, one per hostname, or ensure your existing agent configuration is robust enough to handle the varying demands.

Wrapping Up

This issue is a prime example of how deeply OS-level networking quirks can impact application behavior, especially in highly dynamic environments. It's not always your code, and it's not always simple DNS configuration. Sometimes, you have to dig into the guts of system libraries to find the phantom causing your service disruptions. Implementing this custom lookup function saved us countless hours of intermittent failures and allowed our services to gracefully handle IP changes without the frustrating 'ghost' connections to dead addresses. Stay vigilant, fellow SREs.