Quick Summary: Node.js fs.watch fails to detect file creations/deletions on bind-mounted NFS shares inside Docker. Uncover an obscure kernel/NFS dentry cache int...

Alright, listen up. You've hit that special hell where your Node.js application, humming along nicely in a Docker container, suddenly develops amnesia for new files on your NFS share. Specifically, fs.watch, or anything built on top of it like nodemon or webpack --watch, just decides that new files or deleted files simply don't exist. It's infuriating. Existing file changes? Maybe. New files? Forget about it. This isn't a simple max_user_watches issue. This is deeper, nastier, and a colossal waste of my weekend.

You're probably tearing your hair out. Your local setup works perfectly. CI/CD? Flawless. But deploy to that specific staging environment with the old kernel and the NFS mount, and boom – your hot-reloading stops, your file processing queue sits idle, or your content deployment pipeline grinds to a halt because it can't see the new assets. It's a silence that screams inefficiency.

The Problem: Node.js Blindness on NFS

The symptom is clear: files are created or deleted on an NFS share, but your Node.js application, watching that directory via fs.watch (or any library using it), receives no corresponding 'change', 'rename', or 'create' events. This is especially prevalent for nested directories. You can modify an existing file, and sometimes that works, but the moment you touch new_file.txt or rm old_file.txt, it's like a tree falling in a forest with no one to hear it.

The Trigger Environments

This particular flavor of hell often manifests under these precise conditions. Other combinations might work, but if you're hitting this wall, check your versions:

Component	Problematic Versions	Notes
Host OS Kernel	Linux 4.15.x - 4.19.x	Ubuntu 18.04, CentOS 7/8 (with specific kernel updates). Newer 5.x kernels often mitigate this.
Node.js Version	12.x, 14.x, 16.x (LTS)	Less version-dependent, more kernel/NFS interaction. Node 18+ might have minor mitigations but still susceptible.
Docker Engine	19.03.x - 20.10.x	Running with --mount type=bind,source=/path/to/nfs,target=/app/data
NFS Server	NFSv3 or NFSv4 (async export, noac)	Specific server-side export options like rw,sync,no_subtree_check,noac can exacerbate.
Client Mount Options	Default hard,intr,noatime	The usual suspects are often not enough to fix this.

The Frustrating Dance of Failed Debugging

You’ve already tried the obvious: bumping fs.inotify.max_user_watches, upgrading Node.js, even trying polling-based watchers. None of it consistently works. You’ve probably confirmed `inotifywait` on the host does see the events, which just adds to the confusion. Why is Docker, or rather, the Node process inside Docker, deaf? This isn't just a simple case of a queue filling up; events are never generated for certain types of directory changes.

The Root Cause

This insidious bug is a cocktail of specific kernel versions, NFS client behavior, and Node.js's underlying libuv implementation. Here’s the punchline: Node.js’s fs.watch (via libuv) relies on inotify, which itself relies on the kernel's Virtual File System (VFS) layer and its directory entry cache (dcache). On older Linux kernels (specifically 4.15.x to 4.19.x), when an NFSv3/v4 share is mounted with aggressive caching settings (like noac on the server, implying client-side attribute caching), the interaction between the NFS client's dentry cache invalidation and inotify’s internal directory scanning logic becomes fundamentally flawed.

When a new file is created on the NFS server, the client's dcache might not immediately invalidate the directory entry for the parent directory. inotify, attempting to detect IN_CREATE or IN_DELETE events for directories, performs an internal readdir operation. On these specific kernel versions, a race condition or a flawed heuristic can occur where inotify's readdir operation, trying to be clever and efficient, queries the d_type of directory entries. If the dcache is stale or inconsistent for NFS entries, inotify might miss the newly created or deleted file entirely during its scan, as it doesn't trigger a full cache refresh. It's like the kernel's brain has a sticky note saying "this directory is empty," even after you've put a new file in it. It's a particularly painful flavor of latency, a kind of nanosecond war lost at the kernel level.

libuv, and by extension Node.js, often tries to optimize file system operations by avoiding extra stat() calls, sometimes by relying on the d_type information provided by the kernel. When this kernel-level interaction is broken for NFS directories, libuv’s attempt at optimization turns into a blind spot, completely missing the actual filesystem changes. It's not Node's fault, not Docker's fault, it's a deep-seated kernel-NFS-VFS interaction bug that only surfaces under these specific, frustrating circumstances.

The Solution: Brutal Dentry Cache Annihilation

Since the problem lies with the NFS client's dentry cache being too "clever" and stale for inotify's needs on older kernels, we need to force it to be dumber, but more consistent. The most reliable, albeit performance-impacting, fix is to completely disable the directory entry lookup cache for your specific NFS mount on the host system where Docker runs.

This means modifying your NFS mount options. You need to add lookupcache=none to the relevant entry in your /etc/fstab or your manual mount command. This tells the NFS client to never cache directory lookups, forcing it to query the server for directory entries every time. This will increase network traffic and latency for directory operations, but it guarantees inotify will always see the freshest state, preventing those infuriating missed events. While this might seem heavy-handed, sometimes you need to sacrifice a bit of sub-millisecond execution latency to ensure correctness.

Step-by-Step Fix:

1. Identify your NFS mount point: Find the line in /etc/fstab that corresponds to your problematic NFS share. It will look something like this:

   nfs.example.com:/exports/data  /mnt/nfs_data  nfs  defaults,hard,intr,noatime  0 0

2. Add lookupcache=none: Modify the options to include lookupcache=none.
3. The Complete, Copy-Pasteable Command for /etc/fstab:

   # Before (example):
   # nfs.example.com:/exports/data  /mnt/nfs_data  nfs  defaults,hard,intr,noatime  0 0
   
   # After (example, ensure you adapt your actual server/path/options):
   nfs.example.com:/exports/data  /mnt/nfs_data  nfs  defaults,hard,intr,noatime,lookupcache=none  0 0

   Alternatively, if you're mounting manually or through a script:

   sudo mount -o remount,lookupcache=none /mnt/nfs_data

   Or for a fresh mount:

   sudo mount -t nfs -o defaults,hard,intr,noatime,lookupcache=none nfs.example.com:/exports/data /mnt/nfs_data

4. Re-mount the NFS share: After updating /etc/fstab, either reboot your host system (the simplest, but often not an option) or remount the share:

   sudo umount /mnt/nfs_data
   sudo mount /mnt/nfs_data

   Verify it worked with cat /proc/mounts | grep /mnt/nfs_data and look for lookupcache=none in the options.

5. Restart your Docker container: Ensure your Node.js application container is restarted to pick up the changes in the underlying mount behavior.

This fix is a blunt instrument, yes, but it forces the kernel to bypass the problematic caching heuristics that were causing inotify to miss those critical directory events. Your Node.js applications should now reliably detect all file creation and deletion events on the NFS share, bringing sanity back to your development and deployment workflows. It's ugly, it impacts performance slightly, but it works. And sometimes, "it works" is all you need to hear when you're this deep in the trenches.