Quick Summary: Fix Node.js fs.watch 100% CPU spikes on NFSv3 mounts with specific kernel versions. Troubleshoot inotify limits causing constant retries on Ubuntu...

Alright, listen up. If you're seeing a Node.js process suddenly devour CPU like it's its last meal, but only when it's watching files on a specific NFS mount, and you've already torn your hair out checking application logic and infinite loops, then pull up a chair. I've been through this particular hell, and it’s a special kind of infuriating.

The Problem: Unexplained CPU Spikes on NFS-mounted Directories

Picture this: a Node.js microservice, innocently using fs.watch to monitor a configuration file or a dynamic content directory. Deploy it to a dev environment, works perfectly. Push it to a staging environment running on a VM with an NFSv3 share, and BAM. CPU goes from 2% to 100% within minutes, sometimes seconds. The service becomes unresponsive, logs stop flowing, and your pager is probably already screaming.

We're talking about a process that should be nearly idle. No heavy computation. Just watching a directory. We initially suspected everything from bad garbage collection to a logic bomb in the application code. It wasn't either. It was far more insidious, lurking at the intersection of kernel limitations, network file system quirks, and Node.js's low-level file watching implementation.

The Symptoms (And the Rabbit Holes We Ran Down)

• Consistent 100% CPU usage: Not bursty, not transient. Sustained max utilization.
• Process becomes unresponsive: API endpoints time out, health checks fail.
• Only affects specific environments: Dev, local Docker builds, other cloud environments without NFS were fine. Only systems mounting a specific NFSv3 share exhibited the issue.
• strace shows constant inotify_add_watch and getdents64 calls: This was the first real clue. The process was endlessly trying to add watches or re-scan directories.
• No specific error messages in application logs: Just a slow, painful death.

We spent days digging. We profiled Node.js with --prof, looked at flame graphs, analyzed V8 heap dumps. We scaled up the VMs, thinking it was a resource contention issue. We checked network latency to the NFS server, disk I/O on the NFS share. Nothing. It was a phantom. This level of debugging felt reminiscent of chasing down the ghost in the socket, but with a different set of symptoms and a particularly frustrating lack of error messages.

The Trigger Environments

This particular beast reared its ugly head in a very specific combination of factors. Pay close attention, because your mileage may vary if you're even slightly off this matrix.

Operating System	Kernel Version	Node.js Version	File System	Network File System Version	Inotify Limits	Trigger Condition
Ubuntu Server 20.04 LTS	5.4.0-X-generic (specifically pre-5.4.0-65)	14.x (14.17.0 - 14.19.0)	ext4 (on host)	NFSv3	Default (fs.inotify.max_user_watches ~8192, fs.inotify.max_user_instances ~128)	fs.watch on a directory with > 200 files/subdirectories, especially if symlinks are present or new files frequently appear.
CentOS 7 (similar kernel range)	3.10.0-X-el7	14.x (14.17.0 - 14.19.0)	xfs (on host)	NFSv3	Default/Low	Identical conditions to Ubuntu.

The Root Cause

The core issue is a brutal dance between Node.js's internal fs.watch implementation, the Linux kernel's inotify subsystem, and how NFSv3 handles file events and directory listings. Node.js's fs.watch (especially in older 14.x versions) defaults to using inotify on Linux. While inotify is generally efficient, it has severe limitations, particularly when attempting to watch directories on network file systems like NFSv3.

NFSv3 is a notoriously stateless protocol concerning file change notifications. It doesn't natively support real-time change notifications in the same robust way a local file system does. So, when Node.js's fs.watch tries to use inotify on an NFSv3 mount, the kernel has to resort to polling or other less efficient, often unreliable mechanisms to detect changes. Crucially, it might fail to register watches correctly at all, especially if the directory contains many files or subdirectories, or frequently changes.

The problem is exacerbated by low inotify limits (fs.inotify.max_user_watches and fs.inotify.max_user_instances). If the number of files/directories Node.js attempts to watch exceeds these limits – which happens quickly in a moderately sized directory, particularly when Node.js recursively tries to watch subdirectories or individual files within them – inotify_add_watch calls start failing. Instead of falling back to a saner polling mechanism or raising an explicit error (which newer Node.js versions and kernels often do), these specific older Node.js 14.x versions, interacting with pre-5.4.0-65 kernels, get stuck in a furious retry loop. They constantly attempt to add watches that immediately fail, consuming 100% CPU as the process thrashes the kernel with redundant, futile calls.

This endless loop, often with silently failing inotify calls, is a death spiral. Newer Node.js versions (16.x and above) and kernels have significantly improved fallbacks or smarter detection, mitigating this. But for Node 14.x on these specific kernel versions, it’s a showstopper.

The Solution: Force Polling and Raise Inotify Limits

You have two primary options. The first is to upgrade your Node.js version to 16.x or newer, and ideally, your kernel too. This is the correct long-term fix. However, if you're stuck on Node.js 14.x due to legacy dependencies or a painstakingly slow migration path (which, let's be honest, is often the case in enterprise SRE land), you need a workaround.

The workaround involves explicitly telling Node.js's fs.watch to use its polling mechanism instead of relying on inotify for these problematic directories. Additionally, we'll preemptively raise inotify limits to reduce the chance of other processes hitting similar walls, a general good practice in dense container environments. If you're running in Kubernetes or AWS ECS Fargate, ensure these sysctl changes are applied at the host level or via privileged init containers if your orchestrator supports it.

Step-by-Step Fix

1. Adjust Your Node.js fs.watch Call

Modify your Node.js application code to explicitly use polling for the affected directory. This bypasses the problematic inotify interaction with NFSv3.

const fs = require('fs');
const path = require('path');

const DIRECTORY_TO_WATCH = '/mnt/nfs/config_dir'; // Your problematic NFS path

// Force polling for this specific watch
fs.watch(DIRECTORY_TO_WATCH, { recursive: true, persistent: true, interval: 5000 }, (eventType, filename) => {
    console.log(`File change detected: ${filename}, type: ${eventType}`);
    // Your logic to reload config, etc.
});

console.log(`Watching ${DIRECTORY_TO_WATCH} using polling mechanism.`);

// For other watches on local files, you can still use the default
// fs.watch(path.join(__dirname, 'local_config.json'), (eventType, filename) => { /* ... */ });

The crucial part is the interval: 5000 option (or whatever polling interval is acceptable for your use case). This forces Node.js to poll the directory every 5 seconds instead of relying on kernel events, effectively sidestepping the inotify/NFSv3 conflict.

2. Increase Inotify Limits on the Host System

Even with polling, it's good practice to ensure your kernel has sufficient inotify resources, especially if other applications might be using file watching. This won't directly solve the fs.watch on NFS issue if you force polling, but prevents other potential problems down the line.

# Apply these commands on your Linux host (VM or bare metal)
# This increases the maximum number of user watches
sudo sysctl -w fs.inotify.max_user_watches=524288

# This increases the maximum number of user instances
sudo sysctl -w fs.inotify.max_user_instances=1024

# Make these changes persistent across reboots
echo "fs.inotify.max_user_watches=524288" | sudo tee -a /etc/sysctl.conf
echo "fs.inotify.max_user_instances=1024" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

These values (524288 and 1024) are significantly higher than the defaults and provide ample buffer for most applications. Adjust them based on your environment's needs, but don't go overboard if you don't need to.

Why This Works

By forcing Node.js to use polling, you effectively bypass the unreliable inotify integration with NFSv3 on older kernels. The process stops thrashing, the CPU calms down, and your service goes back to being boringly stable. The increased inotify limits are a defensive measure, preventing similar issues from cropping up if another part of your stack (or even another part of your Node.js app using fs.watch on a local filesystem) tries to watch a large number of files.

This isn't an ideal long-term solution. The real fix is upgrading Node.js and the kernel. But for those times when you're stuck in dependency hell, this little trick will save your service and your sanity. You're welcome.