Quick Summary: Node.js EAI_AGAIN or ENOTFOUND errors in Docker, but ping works. Solve the elusive DNS resolution failure on older kernels with a definitive fix.

You've hit it. That soul-crushing moment. Your Node.js app in Docker is throwing EAI_AGAIN or ENOTFOUND errors. But here's the kicker: ping works. nslookup works. Even dig works. Your mental health is eroding. Welcome to the club, we've got t-shirts and therapy. This isn't your average DNS misconfiguration. This is a subtle, insidious beast that only rears its head under specific, hellish conditions. Let's kill it.

This problem is a phantom. It appears after a host network restart, sometimes after a Docker daemon restart, or even just during periods of high DNS query volume. It specifically targets your Node.js application, leaving other services (like Nginx sidecars or database connections) untouched. You pull your hair out, thinking it's your code, but it's not. It's deeper.

Symptoms: The Mind Games Begin

• Your Node.js logs are full of: getaddrinfo EAI_AGAIN api.example.dev or getaddrinfo ENOTFOUND auth.myapp.app.
• Yet, from within the same Docker container:
  • ping api.example.dev works perfectly.
  • nslookup api.example.dev returns the correct IP address.
  • dig api.example.dev also resolves correctly.
• Sometimes, cURL inside the container also fails for the same domains, but not always.
• It might be intermittent, making it feel like a random cosmic ray hitting your network stack.
• It often affects specific TLDs like .dev, .app, or internal private domains more than standard ones like .com.

The Setup: Where This Hell Triggers

This isn't universal. This particular flavor of hell is environmental. Here's where we've consistently seen it crop up:

OS Version (Host)	Node.js Version (Container)	Trigger Conditions
Ubuntu 18.04 LTS (Kernel < 5.0)	14.x, 16.x	Docker using systemd-resolved, host network restarts, dynamic DNS updates
CentOS 7 (Kernel < 4.18)	12.x, 14.x	Docker's internal DNS (127.0.0.11) as primary, high DNS query volume, specific TLDs
RHEL 8 (Kernel 4.x)	14.x, 16.x	Intermittent failures after host system reboots, specific TLDs (.dev, .app)

The common denominator? Older kernel versions, how Docker manages DNS inside containers, and potentially an interaction with systemd-resolved or another local DNS caching proxy on the host.

Initial Sanity Checks (The ones you already did and failed)

Let's be real. You've already done this. You checked your docker-compose.yaml for DNS settings. You logged into the container and looked at /etc/resolv.conf. You restarted the Docker daemon. You even tried docker network prune. You restarted the host. Nothing. The problem persists, mocking you.

This issue, while obscure, can completely derail critical microservices, especially in highly distributed architectures where intermittent DNS failures can cascade. If you're engineering systems at FAANG scale, you know how catastrophic even a minor networking glitch can be.

The Root Cause: Glacial Glitches in glibc's getaddrinfo and Docker's DNS Shuffle

The underlying issue is a nasty interplay between older glibc versions (specifically their getaddrinfo implementation), how Docker manages resolv.conf inside containers, and the often-misunderstood behavior of systemd-resolved or dnsmasq on the host. Node.js's default dns.lookup function, which relies on getaddrinfo, is the unsuspecting victim.

When Docker dynamically updates /etc/resolv.conf (e.g., after a host network restart or when systemd-resolved flushes its cache), it often writes 127.0.0.11 (Docker's internal DNS resolver) as the primary nameserver. This resolver is supposed to proxy requests to the host's actual DNS setup. For most queries, it works.

However, older glibc getaddrinfo implementations can become confused. They might cache negative responses too aggressively, especially for specific TLDs or when the internal Docker resolver returns non-standard or delayed responses for certain query types (e.g., AAAA vs A records). The getaddrinfo call, unlike ping (which often uses simpler gethostbyname or direct socket(AF_INET, SOCK_DGRAM) calls for DNS requests), performs more complex lookups, trying various address families and potentially getting stuck in a loop or returning EAI_AGAIN when it should eventually resolve via a fallback nameserver.

It's a race condition. The internal resolver might briefly fail or return a malformed response for a given TLD, glibc caches that failure, and subsequent Node.js requests (which depend on glibc) fail even if the underlying DNS path is now clear. Meanwhile, ping and nslookup bypass this getaddrinfo complexity, often using simpler, direct UDP queries, thus succeeding.

The Fix: Shut Down the Phantom

This requires a two-pronged approach: enforce reliable DNS at the container level and ensure Node.js behaves predictably.

# --- Part 1: Force external DNS at the Docker Container Level ---
# This ensures Docker doesn't inject its 127.0.0.11 resolver as primary
# and forces the use of reliable, external DNS servers (e.g., Google DNS).

# Option A: In your Dockerfile (recommended for consistent builds)
# Add these lines BEFORE any 'RUN' commands that might need DNS resolution.
# Use your preferred reliable public DNS (e.g., Google, Cloudflare, corporate DNS).
# FROM node:16-alpine
# RUN echo "nameserver 8.8.8.8" > /etc/resolv.conf && \
#     echo "nameserver 8.8.4.4" >> /etc/resolv.conf && \
#     chmod 644 /etc/resolv.conf # Ensure correct permissions


# Option B: As a runtime override for 'docker run' (e.g., for testing or quick fixes)
# docker run --dns 8.8.8.8 --dns 8.8.4.4 -p 3000:3000 my-node-app:latest

# Option C: In your docker-compose.yaml
# services:
#   my-node-app:
#     image: my-node-app:latest
#     dns:
#       - 8.8.8.8
#       - 8.8.4.4
#     ports:
#       - "3000:3000"


# --- Part 2: Configure Node.js to be Robust ---
# Add this code early in your Node.js application's entrypoint (e.g., app.js or server.js)
const dns = require('dns');
const http = require('http');
const https = require('https');

// CRITICAL STEP 1: Force Node.js DNS to use the external resolvers directly.
// This bypasses glibc's getaddrinfo for specific lookups and avoids local DNS proxies.
// Ensure these match the DNS servers configured at the Docker level.
dns.setServers(['8.8.8.8', '8.8.4.4']);

// CRITICAL STEP 2: Set default result order to 'verbatim'.
// This prevents Node.js from reordering DNS results, which can hide resolution issues
// and ensures predictable behavior across different glibc versions.
// https://nodejs.org/api/dns.html#dnssetdefaultresultorderorder
dns.setDefaultResultOrder('verbatim');

// OPTIONAL (but highly recommended for older Node.js versions or specific issues):
// If you are still seeing intermittent issues, especially with IPv6 lookups,
// you can force Node.js's HTTP/HTTPS agents to prefer or strictly use IPv4.
// This ensures that even if dns.setServers is overridden or behaves unexpectedly,
// your network requests use a stable family.
const agentOptions = {
  family: 4, // Force IPv4 resolution
  // You can also provide a custom lookup function here if needed,
  // but dns.setServers and dns.setDefaultResultOrder should often suffice.
  // lookup: (hostname, options, callback) => {
  //   dns.lookup(hostname, { ...options, verbatim: true, family: 4 }, callback);
  // }
};
const httpAgent = new http.Agent(agentOptions);
const httpsAgent = new https.Agent(agentOptions);

// If using 'node-fetch', 'axios', or other libraries, ensure they use these agents.
// For 'node-fetch':
// import fetch from 'node-fetch';
// fetch('https://api.example.dev', { agent: httpsAgent });

// For native http/https.request:
// https.request({
//   hostname: 'api.example.dev',
//   path: '/',
//   method: 'GET',
//   agent: httpsAgent // Apply the custom agent
// }, (res) => { /* ... */ }).end();

Why This Works (And Why It's Annoying)

By forcing Docker to use external, reliable DNS servers directly, you remove the problematic 127.0.0.11 internal proxy from the resolution path. This eliminates the first point of failure.

Then, by explicitly setting Node.js's DNS servers and result order, you bypass the default behavior that relies on glibc's potentially buggy getaddrinfo implementation. Node.js's internal DNS resolver is more robust in these edge cases. Forcing family: 4 (IPv4) further simplifies the lookup, avoiding potential issues with problematic IPv6 resolutions on misconfigured networks or older kernels. Ignoring these networking subtleties can lead to brittle applications and broken automation pipelines, creating far more operational overhead than necessary.

Final Thoughts: Don't Trust, Verify.

This problem is a prime example of why SREs are always paranoid about networking. What seems like a simple DNS lookup can hide layers of kernel quirks, glibc bugs, and Docker daemon intricacies. Always test your applications under conditions that mimic production environments as closely as possible, especially concerning network restarts and DNS server changes.

You've wrestled the phantom. Go get some sleep. You've earned it.