Quick Summary: Troubleshoot Node.js ECONNRESET/ETIMEDOUT from http.Agent's stale keep-alive connections in dynamic environments. Learn to manage socket lifecycle...

Node.js http.Agent Keep-Alive: The Stale Connection Time Bomb

Alright, listen up. You’re here because your Node.js service, humming along beautifully for a while, suddenly starts spitting out an infuriating barrage of ECONNRESET or ETIMEDOUT errors to an internal dependency. It’s not always, just sometimes. And it’s always after some uptime, usually after a deployment of that dependent service. You’ve checked the network, firewalls, load balancers – everything looks fine for new connections. But the old ones? They’re dying a slow, painful death. Sound familiar? Welcome to the pain of http.Agent's stubborn keep-alive connections in dynamic environments.

This isn’t about basic connection issues. This is about Node.js clinging to old IP addresses like a digital hoarder, even when the world around it has moved on. If your services live in Kubernetes, an autoscaling group, or any environment where target IPs are ephemeral, you’ve hit this wall. Let’s stop wasting cycles and fix this.

Environments Where This Nightmare Thrives

This issue is insidious because it often appears only under specific conditions of Node.js version and underlying OS/network configuration. Here’s where we’ve seen it most commonly:

Operating System	Node.js Version Range	Key Networking Factor
Linux (e.g., Ubuntu 20.04+, Debian 11+, Alpine 3.14+ in Docker)	12.x, 14.x, 16.x, 18.x (up to < 18.10)	Kubernetes service IP changes, Load Balancer backend recycling
Any OS (especially Docker)	14.x, 16.x, 18.x	Aggressive DNS caching (e.g., systemd-resolved), frequent pod/container restarts
Windows Server	14.x, 16.x	Network Policy Manager (NPM) updates, VM live migrations

Yes, newer Node.js versions have made strides, but the core behavior of http.Agent regarding established sockets to stale IPs can still bite you. If you’re also dealing with related issues like Node.js ECONNRESET in Docker caused by Conntrack, this problem often compounds the misery.

The Root Cause

Here’s the deal: Node.js’s default http.Agent, when configured with keepAlive: true (which is the default for many libraries like Axios), maintains a pool of open TCP sockets for a given host and port. This is great for performance, as it avoids the overhead of establishing a new connection for every single request.

The critical flaw? When an http.Agent reuses a socket from its pool, it does so based on the exact IP address that was resolved for the hostname when that socket was originally created. It does not re-resolve DNS for subsequent requests that go over an *already established* keep-alive socket. Think about that for a second. Even if your OS-level DNS cache has updated, even if your systemd-resolved is correctly fetching new IPs (and trust me, that’s a whole other can of worms, see Node.js UDP Sockets and Docker: The systemd-resolved Black Hole), your `http.Agent` still thinks it's talking to the valid IP it saw 3 hours ago. If the target service's underlying pods or VMs have been recycled, that old IP is now dead or belongs to a different service.

What happens then? Your Node.js client sends a request over a perfectly healthy-looking (to the client) socket to a defunct IP. The server-side (or a network intermediary) has no idea what you’re talking about and slams the connection shut, or just drops packets. Result: ECONNRESET, ETIMEDOUT, or a mysterious hanging request.

The No-Nonsense Fix: Proactive Socket Lifecycle Management

You need to tell http.Agent to stop being so clingy. We can’t rely on idle timeouts when connections are constantly active. We need to force sockets to be torn down and recreated after a reasonable amount of time or a certain number of requests. This ensures new DNS lookups occur regularly, keeping your client pointed at valid target IPs.

The trick is to wrap or extend the default http.Agent to introduce a lifecycle management layer. Here's a copy-pasteable utility function that creates a custom agent with this logic. Use it. Live longer.

const http = require('http');
const https = require('https');

/**
 * Creates a custom HTTP/HTTPS agent that proactively destroys sockets
 * after a set lifetime or request count to mitigate stale DNS issues
 * in highly dynamic environments (e.g., Kubernetes).
 *
 * @param {object} options - Options for the underlying http.Agent.
 * @param {number} [options.socketLifetimeMs=30000] - Max age of a socket before destruction (ms).
 * @param {number} [options.maxRequestsPerSocket=50] - Max requests per socket before destruction.
 * @param {object} [options.agentOptions={}] - Additional options passed directly to http.Agent.
 * @returns {http.Agent} A custom agent instance.
 */
function createDynamicAgent({
  socketLifetimeMs = 30000, // Default: destroy sockets after 30 seconds
  maxRequestsPerSocket = 50, // Default: destroy sockets after 50 requests
  ...agentOptions
} = {}) {
  const agent = new http.Agent({ keepAlive: true, ...agentOptions });

  const originalAddRequest = agent.addRequest.bind(agent);
  const socketStats = new WeakMap(); // Tracks stats for each socket

  agent.addRequest = function (req, options) {
    const name = this.getName(options);
    const freeSockets = this.freeSockets[name];

    if (freeSockets && freeSockets.length > 0) {
      for (let i = 0; i < freeSockets.length; i++) {
        const socket = freeSockets[i];
        let stats = socketStats.get(socket);

        if (!stats) {
          // Initialize stats for a new socket coming into the pool
          stats = { birthTime: Date.now(), requestCount: 0 };
          socketStats.set(socket, stats);
          socket.on('close', () => socketStats.delete(socket)); // Clean up stats on socket close
        }

        stats.requestCount++;

        // If the socket is too old or has handled too many requests, destroy it.
        // This forces a new socket creation (and thus a new DNS lookup).
        if (Date.now() - stats.birthTime > socketLifetimeMs || stats.requestCount > maxRequestsPerSocket) {
          console.warn(`[DynamicAgent] Destroying stale socket for ${name} (age: ${Date.now() - stats.birthTime}ms, reqs: ${stats.requestCount}).`);
          socket.destroy();
          freeSockets.splice(i, 1); // Remove it from the free pool
          i--; // Adjust index as element was removed
          continue; // Look for the next available socket
        }

        // If the socket is deemed fit, use it and move it to the active pool
        freeSockets.splice(i, 1);
        this.sockets[name] = this.sockets[name] || [];
        this.sockets[name].push(socket);
        req.onSocket(socket);
        return; // Request handled
      }
    }
    // If no suitable free socket was found (or all were destroyed), proceed
    // with the default addRequest logic, which will create a new socket.
    originalAddRequest(req, options);
  };

  return agent;
}

// --- How to use this Agent with Axios or native http/https modules ---

// const axios = require('axios');

// const dynamicAgent = createDynamicAgent({
//   socketLifetimeMs: 15000,     // Force socket recreation every 15 seconds
//   maxRequestsPerSocket: 10,   // Force socket recreation after 10 requests
//   maxSockets: 100,            // Max total sockets in the pool
//   freeSocketTimeout: 2000     // Still useful for truly idle sockets
// });

// const httpClient = axios.create({
//   httpAgent: dynamicAgent,
//   httpsAgent: dynamicAgent, // Use for HTTPS too
//   timeout: 5000             // Request-level timeout
// });

// // Now use httpClient for all your requests to dynamic services.
// httpClient.get('http://your-dynamic-k8s-service.internal/health')
//   .then(response => console.log('Service is healthy:', response.data))
//   .catch(error => console.error('Service call failed:', error.message));

Why This Works

By overriding the addRequest method, we inject custom logic right at the point where http.Agent decides whether to reuse an existing free socket. Before letting a request use a socket, we check its age and how many requests it has already handled. If it exceeds our defined thresholds, we brutally destroy it. This forces the agent to create a *new* socket, which in turn triggers a fresh DNS lookup for the target hostname. This simple, yet effective, proactive destruction prevents your Node.js application from endlessly sending traffic to stale, non-existent IP addresses.

This approach gives you the benefit of connection pooling (keepAlive: true) for short bursts, while gracefully handling the inevitable IP address churn in modern, distributed systems. Configure socketLifetimeMs and maxRequestsPerSocket to values that make sense for your environment’s churn rate. Too low, and you negate the benefits of keep-alive. Too high, and you’re back to square one. Find your balance.

Stop chasing ghosts in your network logs. Take control of your Node.js HTTP client's connection lifecycle and eliminate this insidious source of transient errors.