Quick Summary: Troubleshooting an obscure Node.js DNS resolution issue in Alpine containers on Kubernetes, leading to 'EAGAIN' errors after idle periods. Deep di...

Alright, listen up. If you're here, you've probably spent the last three weeks staring at logs, chugging stale coffee, and muttering obscenities at a problem that defies all logic. I've been there. This isn't your average 'DNS misconfigured' crap. This is a subtle, insidious beast that only rears its ugly head under a very specific constellation of circumstances. We're talking about Node.js apps in Alpine containers, randomly failing to resolve internal Kubernetes service names after a period of idleness. External DNS? Works fine. Restart the pod? Works fine... for a bit. Then, BAM! EAGAIN or ETIMEDOUT on an internal lookup, and your service goes belly-up.

It's the kind of bug that makes you question your career choices. The kind that makes you want to throw your monitor out the window. But we're SREs, damn it. We fix the unfixable. So, let's get this done.

The Symptoms: Intermittent, Insidious, and Infuriating

• Your Node.js application, running inside an Alpine-based Docker container in Kubernetes, suddenly starts throwing DNS resolution errors for internal services. Think my-backend-service.my-namespace.svc.cluster.local.
• Errors will manifest as EAGAIN (Resource temporarily unavailable) or ETIMEDOUT on dns.lookup or database connection attempts that rely on service discovery.
• Crucially, this happens after a period of low traffic or inactivity. Hit the endpoint a few times, it works. Leave it alone for 5-10 minutes, hit it again, and it breaks.
• External DNS lookups (e.g., google.com) still work perfectly from within the same container. This is your first clue that it's not a general network issue.
• Restarting the affected Node.js pod immediately resolves the issue, temporarily.
• Metrics show a sudden drop in success rates for calls to internal services, but not external ones.

The Environments Where This Bug Bites Hard

This particular flavor of hell seems to thrive in specific combinations. Before you start pulling your hair out, confirm you're in the blast radius:

Component	Versions Where Symptoms Are Prevalent	Notes
Operating System (Base Image)	Alpine Linux <= 3.12 (specifically musl-libc versions)	Less common in Debian/Ubuntu (glibc) based images, but not impossible if using older versions or specific resolv.conf tweaks.
Node.js Runtime	Node.js <= 14.x (especially 12.x and 10.x) Node.js 16.x (some specific patch versions)	Higher versions (18.x+) show more resilience due to internal resolver improvements, but can still be affected by root cause. If you're comparing the merits of various back-end frameworks, Spring Boot vs. NestJS: The Enterprise War might touch on runtime differences, though this issue is lower-level.
Kubernetes CoreDNS	CoreDNS <= 1.8.x (with default health and autopath plugins)	Default CoreDNS idle timeout settings play a significant role.
Kubernetes Network Policy	Strict Egress Network Policies	Policies that restrict UDP traffic or create unusual ephemeral port allocation patterns can exacerbate.

Initial Blunders and Why They Fail

You've already tried these, haven't you? Don't lie. We all have:

• Checking /etc/resolv.conf: Looks pristine. Points to kube-dns (or coredns) service IP. Has your search domains. Looks perfectly normal.
• nslookup / dig from inside the container: Works perfectly every single time, even when your app is failing. Infuriating! This is because nslookup/dig typically open new UDP sockets for each query, bypassing any stale connection state.
• tcpdump on the pod: You see DNS requests going out, sometimes even responses coming back, but your app still fails. Or, you see no requests go out at all when the app is failing, suggesting the lookup is dying before it even hits the wire.
• Increasing ndots, timeout, attempts in resolv.conf: Might delay the problem, but doesn't solve it. The fundamental interaction remains flawed.

The Root Cause: Musl, CoreDNS, and Node's Lazy Resolver

This is where it gets nasty. The problem is a three-way dance of death between:

1. Alpine's musl-libc Resolver: Unlike glibc, musl has a simpler, more minimalist DNS resolver. It's fantastic for small container images, but its internal caching and handling of stale UDP connections can be less robust. Specifically, when Node.js calls getaddrinfo (which is what dns.lookup eventually uses for hostnames not managed by its internal cache), musl's resolver library is engaged.
2. CoreDNS UDP Idle Timeout: Kubernetes' default CoreDNS configuration often closes idle UDP connections after a relatively short period (e.g., 5 seconds). When your Node.js app is idle, the UDP socket used by musl's resolver to query CoreDNS can become stale. Future DNS queries might attempt to reuse this stale socket. This behavior is usually fine with more robust resolvers, which detect a closed connection and re-establish, but musl (in older versions) combined with specific resolv.conf options can falter.
3. Node.js dns.lookup / getaddrinfo Interaction: Node.js's dns.lookup, when it defers to the underlying OS's getaddrinfo, isn't always aggressive enough in handling transient resolver failures like EAGAIN or `ETIMEDOUT` when they stem from a stale UDP socket. It expects the underlying OS resolver to handle the connection lifecycle gracefully. When musl fails to re-establish the connection correctly or quickly enough, Node.js simply propagates the error, leading to your application crashing. The issue is compounded by search domains and ndots in resolv.conf, which can lead to multiple iterative queries, increasing the chance of hitting the stale connection issue. For truly massive, FAANG-scale distributed systems, these resolver interactions are ruthlessly optimized to avoid such pitfalls.

So, your app sits idle. CoreDNS closes the UDP port. Your Node.js app wakes up, tries to resolve an internal service. The musl resolver tries to use a stale socket. It fails. Node.js gets EAGAIN and dies. External lookups still work because they often trigger a fresh connection or use a different resolver path for public DNS.

The Fix: Bypassing the OS Resolver for Internal Names

The most robust solution, short of rebuilding musl or constantly pinging CoreDNS, is to make Node.js bypass the problematic OS resolver for your critical internal Kubernetes service names. We achieve this by using Node.js's built-in dns.Resolver class directly and pointing it at your CoreDNS service IP. This gives us explicit control.

First, find your CoreDNS (kube-dns) service IP. In most Kubernetes clusters, it's 10.96.0.10 or similar. You can find it with kubectl get svc -n kube-system kube-dns or kubectl get svc -n kube-system coredns.

Now, modify your Node.js application to use a custom resolver for internal services:

const dns = require('dns');
const { Resolver } = require('dns');

// IMPORTANT: Replace with YOUR CoreDNS service IP
// In many clusters, this is the .10 address in your service CIDR.
// E.g., for a K8s service CIDR of 10.96.0.0/12, the DNS service is usually 10.96.0.10
const KUBERNETES_DNS_SERVICE_IP = process.env.KUBERNETES_DNS_SERVICE_IP || '10.96.0.10';

const internalK8sResolver = new Resolver();
internalK8sResolver.setServers([KUBERNETES_DNS_SERVICE_IP]);

/**
 * Wrapper function to resolve internal Kubernetes service hostnames.
 * Uses a dedicated resolver that talks directly to CoreDNS, bypassing OS resolver.
 * Fallbacks to default dns.lookup for external hostnames.
 * @param {string} hostname - The hostname to resolve.
 * @returns {Promise} - A promise that resolves to an array of IP addresses.
 */
async function resolveK8sService(hostname) {
    // Check if the hostname looks like an internal K8s service name
    // Adjust this regex based on your specific naming conventions if needed
    const isInternalK8sService = /\.svc\.(cluster\.local|internal)$/.test(hostname) ||
                                   /\.my-namespace\.svc\.cluster\.local$/.test(hostname); // Example: my-service.my-namespace.svc.cluster.local

    if (isInternalK8sService) {
        console.log(`[DNS Override] Resolving internal service: ${hostname} via ${KUBERNETES_DNS_SERVICE_IP}`);
        try {
            const addresses = await internalK8sResolver.resolve4(hostname);
            return addresses;
        } catch (error) {
            console.error(`[DNS Override] Failed to resolve internal service ${hostname} using direct resolver:`, error);
            // Optionally, fallback to default resolver on error, but be cautious
            // return dns.promises.lookup(hostname).then(result => [result.address]);
            throw error; // Re-throw to indicate failure
        }
    } else {
        // For external hostnames, use the default OS resolver
        console.log(`[DNS Default] Resolving external hostname: ${hostname}`);
        const result = await dns.promises.lookup(hostname);
        return [result.address];
    }
}

// --- How to use it in your application ---

// Example 1: Resolving a database service
(async () => {
    try {
        const dbServiceHost = 'my-postgres.my-namespace.svc.cluster.local';
        const dbIp = await resolveK8sService(dbServiceHost);
        console.log(`${dbServiceHost} resolved to: ${dbIp}`);
        // Your database connection code here, using dbIp
    } catch (error) {
        console.error('Error connecting to DB:', error);
    }

    try {
        const externalHost = 'example.com';
        const externalIp = await resolveK8sService(externalHost);
        console.log(`${externalHost} resolved to: ${externalIp}`);
    } catch (error) {
        console.error('Error resolving external host:', error);
    }
})();

// If you need to patch global behavior for specific modules, this is dangerous
// but sometimes necessary for legacy code or modules that don't allow custom resolvers.
// For instance, if 'axios' or 'node-fetch' don't expose resolver config.
// BE CAREFUL: This is a last resort and can introduce other issues.
/*
const originalLookup = dns.lookup;
dns.lookup = function (hostname, options, callback) {
    if (typeof options === 'function') {
        callback = options;
        options = {};
    }
    if (typeof options === 'number') {
        options = { family: options };
    }

    const isInternalK8sService = /\.svc\.(cluster\.local|internal)$/.test(hostname);
    if (isInternalK8sService) {
        internalK8sResolver.resolve4(hostname, (err, addresses) => {
            if (err) {
                return originalLookup(hostname, options, callback); // Fallback on error
            }
            callback(null, addresses[0], 4); // Assume IPv4 for simplicity in this example
        });
    } else {
        originalLookup(hostname, options, callback);
    }
};
*/

Why This Works

By using dns.Resolver directly, we are telling Node.js to act as its own DNS client for these specific lookups. It opens a fresh UDP socket for each query to the specified CoreDNS IP, completely bypassing the OS's getaddrinfo and, by extension, musl's potentially problematic resolver cache and connection handling. This eliminates the stale UDP socket issue that CoreDNS's idle timeout triggers.

Final Thoughts: Control Your Stack

This problem is a brutal reminder that even seemingly simple components like DNS resolution can hide terrifying complexity when different parts of the stack (OS resolver, application runtime, network services) interact poorly. As SREs, our job isn't just to keep the lights on, but to understand the dark corners where performance and reliability go to die. Sometimes you have to take explicit control to ensure stability, rather than relying on default behaviors.

Don't just copy-paste; understand why this works. And for God's sake, monitor your DNS resolution times! You might not catch every EAGAIN, but slow resolution is often the canary in the coal mine.