Quick Summary: Node.js app on Docker Swarm hitting EAI_AGAIN after uptime? This guide reveals the obscure iptables bug causing intermittent DNS failures on Ubunt...

Alright, listen up. You're staring at logs, pulling hair, asking why your Node.js app is intermittently puking EAI_AGAIN for external services. Not just any errors: the ones that only pop up after 12-24 hours of uptime, exclusively on your Docker Swarm overlay network, and disappear with a simple container restart, only to reappear. Welcome to the infuriating intersection of Docker, systemd-resolved, and a ghost iptables rule. You've checked everything obvious: memory, CPU, Node.js logs. Nothing. The problem persists, intermittently, insidiously.

This isn't about sub-millisecond latency; it's about outright DNS lookup failures. Before we dive, confirm your environment matches this particular hell:

Category	Specifics
Operating System	Ubuntu 20.04 LTS+, Debian 11+. Any distro with systemd-resolved.
Container Runtime	Docker Engine 19.03+, Docker Swarm, overlay networks.
Node.js Versions	Node.js 14.x, 16.x, 18.x (using glibc's getaddrinfo).
Networking Component	systemd-resolved active on host, managing /etc/resolv.conf.
The Culprit	A legacy iptables REDIRECT rule on the host.

The pattern is crucial: slow-burn degradation, not immediate failure. This screams state, cache, or time-dependent accumulation. The EAI_AGAIN points to DNS. Why after hours? Why only from containers on the overlay network?

Inside your container, /etc/resolv.conf points to 127.0.0.11—Docker's internal DNS. Docker forwards these to the host's systemd-resolved (127.0.0.53). All standard. But when the error hits, tcpdump -i any port 53 -n on the host shows queries leaving the container's interface, hitting the Docker bridge, then vanishing. No response from 127.0.0.53. No outbound queries. A black hole.

The Root Cause

Days of debugging, staring at iptables-save output, then it clicked. Not Docker, not Node.js, not systemd-resolved. It was a leftover iptables REDIRECT rule. Inserted by some forgotten VPN, a misconfigured local DNS proxy (like when you were tinkering with local LLM deployments and needed specific network rules), or an experimental network setup. This rule, lurking in a chain (often PREROUTING), captured UDP port 53 traffic.

Docker injects its own iptables rules. But if a custom REDIRECT for UDP port 53 exists in a chain processed *before* Docker’s, it hijacks DNS queries from Docker's internal resolver (127.0.0.11) *before* they reach systemd-resolved (127.0.0.53). This rogue rule often redirects to 127.0.0.1:53. If nothing's listening there, or if the process that *was* listening is defunct or its state has decayed, those queries hit a brick wall and time out, causing EAI_AGAIN. The delay? systemd-resolved and glibc resolver caches. When TTLs expire, fresh lookups hit the iptables trap.

The Fix: Exorcising the Ghost

Cut the Gordian knot. Find and remove that insidious iptables rule. Host-level operation. Backup first.

1. Identify the Rogue Rule:

   Dump iptables config. Look for unexpected REDIRECT rules targeting UDP port 53 in the nat table, chains like PREROUTING or OUTPUT.

   sudo iptables-save > ~/iptables-backup-$(date +%F_%H-%M).txt
   sudo iptables -t nat -L -v -n --line-numbers

   Look for lines like REDIRECT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53 redir ports 53.

2. Backup Configuration (Seriously):

   sudo cp /etc/iptables/rules.v4 /etc/iptables/rules.v4.bak_$(date +%F_%H-%M)
   sudo netfilter-persistent save

3. Remove the Rule:

   Find the chain and line number (e.g., PREROUTING, line 3). Remove it.

   sudo iptables -t nat -D PREROUTING 3

4. Persist the Change:

   Save new configuration. Restart Docker/systemd-resolved.

   sudo netfilter-persistent save
   sudo systemctl restart docker
   sudo systemctl restart systemd-resolved

5. Verify and Monitor:

   Restart Node.js services. Monitor. Error should be gone.

Alternative (Less Ideal) Workarounds

If host iptables is untouchable (bad sign!), consider these:

• Force specific DNS in Docker:

  Edit /etc/docker/daemon.json on each host. Explicitly set public DNS (e.g., Google, Cloudflare). Restart Docker.

  {
    "dns": ["8.8.8.8", "1.1.1.1"]
  }

• Modify Node.js app DNS:

  Use custom https.Agent with Node.js's dns.resolve (bypasses glibc) or a specific DNS package. This fixes the app, not the host issue, making it a band-aid.

  const axios = require('axios');
  const { Agent } = require('https');
  const { Resolver } = require('dns');
  
  const resolver = new Resolver();
  resolver.setServers(['8.8.8.8', '1.1.1.1']);
  
  const agent = new Agent({
    lookup: (hostname, options, callback) => {
      resolver.resolve4(hostname, (err, addresses) => {
        if (err) return callback(err);
        callback(null, addresses[0], 4);
      });
    }
  });
  
  axios.get('https://example.com', { httpsAgent: agent })
    .then(response => console.log(response.data))
    .catch(error => console.error(error));

This issue highlights a critical SRE principle: the layers below often bite hardest. A forgotten iptables rule can unravel a good distributed system. Don't assume. Verify. Keep your host configuration clean.