Quick Summary: Fix intermittent EAI_AGAIN or ENOTFOUND DNS errors in Node.js 18.x apps on CentOS 7/Nomad/Consul. Uncover the getaddrinfo IPv6 interaction flaw. S...

What a nightmare this was. You’re running a critical Node.js service, happily humming along in your containerized Nomad environment. Suddenly, out of nowhere, you start seeing intermittent DNS resolution failures. Not for external services, oh no. For your internal Consul-backed service mesh names. EAI_AGAIN, ENOTFOUND, random timeouts. Under load, it gets worse. You want to tear your hair out because everything looks correct. And it only happens on some instances, sometimes. Sound familiar? Good. Because I’ve been there, and I’m going to save you weeks of your life.

This isn't your average 'misconfigured DNS' problem. This is a subtle, insidious bug buried deep in the interaction between a specific Node.js version, an older Linux kernel, and how libc handles getaddrinfo in a containerized environment with custom DNS. Specifically, we're talking about Node.js 18.x on CentOS 7, deployed via Nomad on Docker.

The Symptoms: What You're Seeing

Your Node.js application logs are peppered with:

• Error: getaddrinfo EAI_AGAIN internal-service.service.consul
• Error: getaddrinfo ENOTFOUND internal-service.service.consul
• Sporadic FetchError: request to http://internal-service.service.consul:3000 failed, reason: connect EHOSTUNREACH after a long timeout.

These errors are not constant. They appear under moderate to heavy load, or just randomly enough to be maddening. Retries sometimes work, sometimes don't. Your older Node.js 16.x applications? Rock solid. Newer Node.js 20.x? Also fine. It’s Node.js 18.x that's the problem child.

The Triggering Environments

This particular brand of hell seems to manifest most reliably under these conditions:

Component	Version Where Error Triggers	Notes
Operating System	CentOS 7.x (Kernel < 4.18)	Older kernels, especially those without modern systemd-resolved integration or specific getaddrinfo patches. Ubuntu 18.04 LTS might also see this.
Node.js Runtime	18.x (specifically 18.12.0 - 18.19.1)	Versions before 18.0.0 and after 19.x (or 20.x LTS) are generally unaffected.
Container Runtime	Docker (any recent version)	Using default bridge networking with custom DNS servers injected.
Orchestrator/DNS	HashiCorp Nomad / Consul	Consul agent providing DNS resolution on 127.0.0.1:8600 within the container.

What Didn't Work (Save Yourself the Headache)

Before we get to the fix, let's cover the dead ends I slammed into:

• Checking /etc/resolv.conf inside the container: It looked fine. nameserver 127.0.0.1, options ndots:0, search domains correct.
• Increasing Node.js DNS cache timeout: dns.setDefaultResultOrder('verbatim'); and dns.setServers(['127.0.0.1:8600']); – no change.
• Docker DNS settings: Explicitly setting --dns 127.0.0.1 in Docker daemon or Nomad config didn’t magically solve the underlying problem.
• Consul health checks: Consul itself was happy. The agent was responsive, resolving names via dig or nslookup from within the container consistently and quickly.
• Upgrading Node.js modules: Unrelated.

This wasn't about Consul. This wasn't about Docker's DNS settings. This was deeper.

The Root Cause

Here’s where it gets truly obnoxious. Node.js 18.x introduced a change in how it handles DNS resolution, specifically moving towards preferring IPv6 when getaddrinfo is called. It uses uv_getaddrinfo which, in turn, relies on the system's getaddrinfo (part of libc). On older kernels, particularly CentOS 7, the libc getaddrinfo implementation, when asked for both IPv6 (AF_INET6) and then IPv4 (AF_INET4), can exhibit a race condition or an unexpected blocking behavior, especially if IPv6 is technically available but not properly routed or configured for the specific target within the container's network namespace.

When Node.js 18.x tries to resolve internal-service.service.consul, it effectively asks getaddrinfo for both AAAA (IPv6) and A (IPv4) records. Even if your Consul DNS server only returns A records (IPv4), the initial IPv6 lookup can get "stuck" or time out within libc before it properly falls back to IPv4, leading to the intermittent EAI_AGAIN or ENOTFOUND errors. It’s like a traffic cop sending you down a perfectly valid but currently jammed IPv6 highway, while the clear IPv4 road sits right next to it, ignored. The issue isn’t the Consul DNS; it’s the underlying system resolver taking too long or failing unexpectedly when trying to negotiate IPv6 in a not-quite-right environment.

This problem becomes more pronounced under load because more concurrent getaddrinfo calls expose the race condition or blocking behavior more frequently. It’s a classic case of an architectural mismatch between modern software expectations (Node.js 18.x's resolver preference) and an older, established infrastructure component (CentOS 7's libc and kernel networking stack).

For those obsessed with latency and Nanosecond Nirvana: Architecting Ultra-Low Latency Trading Infrastructure, this kind of intermittent resolution delay is an absolute killer. Every millisecond counts. Similarly, for applications relying on Sub-Millisecond Warfare: Weaponizing APIs for Alpha Dominance, unpredictable DNS is a non-starter. This isn't just an annoyance; it's a performance bottleneck that can cripple your service's ability to respond.

The Fix: Force Node.js to IPv4

The simplest, most direct solution is to tell Node.js to explicitly prefer IPv4 when resolving hostnames. This bypasses the problematic IPv6 lookup path in libc on those specific older kernels. You do this by setting an environment variable for your Node.js process:

NODE_OPTIONS="--dns-result-order=ipv4first" node your-app.js

Alternatively, if you're using a package.json script, you can integrate it there:

{  "scripts": {    "start": "NODE_OPTIONS=\"--dns-result-order=ipv4first\" node index.js"  }}

Or, if you’re using Nomad, inject it into your task definition's env block:

job "my-node-service" {  datacenters = ["dc1"]  group "app" {    count = 3    network {      port "http" {}      dns {        servers = ["127.0.0.1:8600"]      }    }    task "node-app" {      driver = "docker"      config {        image = "my-registry/my-node-app:18.19.1"        ports = ["http"]      }      env {        NODE_OPTIONS = "--dns-result-order=ipv4first"      }      resources {        cpu    = 500        memory = 512      }    }  }}

Why This Works (and Why It's Annoying)

By forcing ipv4first, you instruct Node.js to prioritize IPv4 address lookups. This sidesteps the specific getaddrinfo behavior that was causing trouble when IPv6 was implicitly preferred or attempted first on those older libc/kernel combinations. It’s a band-aid, yes, but a highly effective one that immediately restores stability.

Is it ideal? No. Ideally, your kernel and libc would handle getaddrinfo gracefully, or your container network would have a fully functional IPv6 stack. But in the real world, you don't always have the luxury of upgrading entire OS kernels on production clusters overnight, especially when legacy systems are involved. This fix is targeted, effective, and requires minimal changes to your application code.

The frustration here is real. Hours spent debugging network, Docker, Consul, and Node.js code, only to find a single environment variable resolves a deeply nested interaction issue. Remember this one. It'll save you when you hit that wall again.