Quick Summary: Stuck with Node.js fs.watch missing events on NFSv4 mounts? This SRE guide uncovers the obscure kernel 5.4.x bug and provides a definitive fix.

Your Node.js service uses fs.watch. It needs to react instantly to file changes in a directory. Deploy it to production, put that directory on an NFSv4 mount, and your service goes deaf. Changes happen, but Node.js is oblivious.

Restart the service, and changes are picked up. Touch the file, and it triggers. What’s going on? This isn't new; it's an old, infuriating ghost from specific older Linux kernels and how NFSv4 interacts with client-side caching and inotify.

This exact scenario has derailed deployments and wasted countless hours. Stop. This isn't your code. This is a system-level interaction bug from hell.

The Symptoms:

• Node.js fs.watch intermittently misses file modifications, creations, or deletions on an NFSv4 mounted directory.
• Changes are detected only after a service restart or manual intervention (e.g., ls or touch).
• No errors logged by Node.js, just a complete lack of events.
• Works perfectly on local filesystems (ext4, XFS).

The Triggering Environments:

This bug usually appears in these combinations. These are the prime suspects.

Operating System	Kernel Version Range	Node.js Version Range	NFS Client/Server
Ubuntu 20.04 LTS	5.4.0-x-generic (e.g., 5.4.0-100 to 5.4.0-160)	14.x, 16.x	NFSv4 Client/Server
CentOS 8 / RHEL 8	4.18.0-x-generic (e.g., 4.18.0-300 to 4.18.0-400)	14.x, 16.x	NFSv4 Client/Server
Debian 10 (Buster)	4.19.x-x-amd64	14.x	NFSv4 Client/Server

The Root Cause

Here’s the ugly truth. This isn’t a Node.js bug. It’s an infuriating dance between the Linux kernel's inotify subsystem, NFSv4 client-side caching, and how libuv (Node.js's I/O layer for fs.watch) interacts with it all.

On local filesystems, inotify hooks directly into the kernel’s VFS for instant changes. On NFS, aggressive client-side caching (attribute, data) reduces network traffic. Good for performance, bad for real-time change detection.

The core problem: a race condition. When a file changes on the NFS server, the client needs notification. But the NFS client often doesn't invalidate its local attribute cache immediately (controlled by acregmin, acregmax, noac). During this window, inotify might receive a stale inode change notification *before* the NFS client's attribute cache updates.

Node.js (via libuv) gets a notification, stats the file, but reads the old state due to client-side caching. It concludes nothing changed and ignores the event. This is rampant in older kernel versions (pre-5.8) where inotify and NFS cache consistency were less robust. The client eventually updates its cache, but too late. This kind of event sensitivity, where nanosecond delays can have catastrophic consequences, is crucial for discussions like "The Nanosecond War: Engineering Ultra-Low Latency Algorithmic Trading".

The Fix: Starve the Cache (Aggressively)

Disable or severely cripple client-side attribute caching for that specific NFS mount. This forces the client to always ask the server for the latest file attributes, eliminating the stale cache window. This is direct, brutal, and effective.

You must modify your NFS client mount options. This isn't pretty and introduces I/O overhead, but for critical fs.watch reliability, it’s necessary.

Step-by-Step Resolution:

1. Identify the Mount Point: Find the NFS mount your Node.js app is watching using:

   df -hT | grep nfs

2. Modify /etc/fstab: Edit /etc/fstab with root privileges for persistence.

   sudo vi /etc/fstab

3. Add noac to Mount Options: Find your NFS mount line, e.g.:

   NFS_SERVER:/path/to/share /mnt/my-nfs-app nfs defaults,hard,intr 0 0

   Append noac to existing options. If no options, replace defaults (usually append).

   The Command/Config Override:

   NFS_SERVER:/path/to/share /mnt/my-nfs-app nfs noac,hard,intr 0 0

   If noac is too aggressive, reduce attribute caching times drastically:

   NFS_SERVER:/path/to/share /mnt/my-nfs-app nfs acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,intr 0 0

   noac is generally the more definitive fix as it implies zeroed `ac` options. Use it first.

4. Remount the Filesystem: Save /etc/fstab, then remount.

   sudo mount -o remount /mnt/my-nfs-app

   Verify options: cat /proc/mounts | grep /mnt/my-nfs-app

5. Restart Your Node.js Service: Restart and test. fs.watch should now behave as expected.

Why Not Just Poll?

Ditching fs.watch for polling (setInterval + fs.stat) is inefficient. You constantly hit the network, even when nothing changes. inotify is event-driven; polling is brute-force. For high-performance applications, polling is a last resort. For solutions like those discussed in "Rustless.js: Another Silver Bullet for JavaScript's Existential Crisis?", you'd prioritize fixing the underlying issue over I/O overhead from workarounds.

A Word of Caution: Performance Trade-offs

Disabling attribute caching (noac) *will* increase metadata operations to your NFS server. For read-heavy static data, a slight performance hit is possible. But for directories needing immediate change notification, this overhead is a small price for correctness. Measure, don't guess, but for accurate fs.watch on NFS, correctness typically outweighs minor noac performance impact.

There you have it. A frustrating, obscure problem, its root cause, and the definitive fix. Now get your Node.js apps listening again.