EMFILE on Node.js in Systemd? It’s Not Your Ulimit, You Idiot.

Are you tearing your hair out? You've got a Node.js application running happily in dev, but in production, under load, it sporadically shits the bed with EMFILE, EADDRNOTAVAIL, or other inexplicable resource exhaustion errors. You’ve checked ulimit -n, confirmed your file descriptor limits are through the roof. You’ve tweaked sysctl. You’ve stared at your code, wondering if your brilliant async architecture is, in fact, a memory-leaking nightmare.

Spoiler: It’s not your ulimit. It's not (directly) your code. It's an obscure, infuriating interaction between systemd, cgroupfs v1, and how Node.js handles internal I/O threads. And it's been silently throttling your otherwise robust application.

The Problem: Intermittent Resource Exhaustion Under Load

Your Node.js service, managed by systemd, shows logs like this:

• Error: EMFILE: too many open files, 'some-socket-operation'
• Error: EADDRNOTAVAIL: Address not available, 'some-network-bind'
• Failed to connect to XYZ service: write EPIPE
• Or just general slowdowns and connection timeouts that resolve after a restart.

These errors often manifest when your application is under significant network load, performing many DNS lookups, or interacting with the file system frequently. The infuriating part? They're not consistent. They pop up, disappear, and make you doubt your sanity.

Environments Where This Error Triggers

This particular beast tends to surface on older Linux distributions and Node.js versions that aggressively leverage libuv's thread pool for blocking I/O operations.

Newer distributions (e.g., Ubuntu 20.04+, CentOS 8+) using systemd 240+ and cgroupfs v2 might be less susceptible or have different default configurations.

You've Done All The 'Right' Things

I know, you've been a good SRE. You’ve meticulously configured your system:

• Checked ulimit -n: It's 65536 or higher for the user running the service.
• Edited /etc/security/limits.conf: Hard and soft limits are set correctly.
• Tuned /etc/sysctl.conf: fs.file-max is large, net.ipv4.ip_local_port_range is wide.
• Verified your service user: It’s not root, it has proper permissions.

And yet, it persists. The intermittent crashes, the phantom resource limits. This is where you stopped looking at ulimit because the problem wasn't a shortage of available file descriptors for your processes. It was a shortage of tasks.

The Root Cause: systemd's DefaultTasksMax and cgroupfs v1

Here’s the deal: On older Linux kernels and systemd versions, particularly those still heavily relying on cgroupfs v1, systemd imposes a limit on the number of tasks a cgroup can create. This limit is called TasksMax, and its default value (via DefaultTasksMax in /etc/systemd/system.conf) is often 512.

What’s a 'task' here? It’s basically a kernel thread. And guess what Node.js does under the hood when it needs to perform blocking I/O (like DNS resolution, many file system operations, some crypto functions, etc.)? It offloads them to libuv's internal thread pool. Each of these operations can spin up a new kernel thread if the pool is busy.

When your Node.js application is under heavy load, concurrently doing many network lookups or disk operations, it can quickly exhaust that default TasksMax=512 limit. Critically, this limit is enforced at the cgroup level, before your process-level ulimit -n even comes into play for file descriptors. The kernel refuses to create new threads/tasks, leading to the same EMFILE or EADDRNOTAVAIL errors you'd see with actual FD exhaustion, even though your FD limits are fine.

It’s an architectural choke point in the interaction between systemd's resource management and libuv's thread pool behavior.

The Fix: Increase TasksMax in Your systemd Service File

The solution is deceptively simple: explicitly raise the TasksMax limit for your Node.js service within its systemd unit file. You can set it to a very high number, or even infinity for services you know are thread-heavy.

Here’s how to do it:

1. Locate your service file: This is typically in /etc/systemd/system/your-service.service or /lib/systemd/system/your-service.service.
2. Edit the service file: Add or modify the TasksMax directive under the [Service] section.

Example modification:

[Unit]
Description=My Awesome Node.js Service

[Service]
ExecStart=/usr/bin/node /opt/my-app/app.js
WorkingDirectory=/opt/my-app
Restart=always
User=nodeuser
Group=nodeuser
Environment=NODE_ENV=production

# THE CRITICAL LINE YOU NEED TO ADD OR MODIFY
TasksMax=infinity

[Install]
WantedBy=multi-user.target
    

A word of caution: Setting TasksMax=infinity means your service can theoretically spawn an unlimited number of threads, potentially exhausting system memory or CPU if your application has a bug. However, for most well-behaved Node.js applications with typical I/O patterns, infinity is perfectly safe as libuv manages its thread pool efficiently. If you're nervous, pick a very large number like 32768 or 65536, which is still orders of magnitude higher than the default 512.

3. Reload systemd daemon: After editing the file, you must tell systemd to re-read its configuration.

sudo systemctl daemon-reload

4. Restart your service: Apply the new limits.

sudo systemctl restart your-service.service

5. Verify (Optional but Recommended): Check the status of your service and monitor its behavior under load. You can also inspect the cgroup limits.

Why Is This So Obscure?

This problem is a nightmare because it sits at the intersection of several complex abstractions: the Node.js runtime, libuv's internal thread management, systemd's resource control groups, and Linux kernel threading models. The error messages (EMFILE) are misleading, pointing you towards file descriptor limits when the true bottleneck is task/thread creation. It's a classic example of abstraction layers leaking confusing errors.

Conclusion

Congratulations. You've just wrestled with one of the more infuriating, poorly documented quirks of running modern applications on older Linux infrastructure. Stop pulling your hair out. Your Node.js application likely wasn't the problem, and your ulimit was fine. It was systemd's silent, misguided attempt at resource governance throttling your application’s ability to create the necessary background threads for I/O.

You're welcome. Now go enjoy your stable production environment.