Quick Summary: Battling Node.js fs.watch reliability issues on Linux 5.10+ kernels with OverlayFS in Docker containers? Uncover the root cause of stale caches an...

Alright, listen up. You’ve been here. Your Node.js dev server isn’t hot-reloading. Your CI builds are mysteriously using stale assets, ignoring that latest critical commit. You’ve restarted Docker a dozen times. You’ve kicked your monitor. You’re losing your damn mind.

I’ve seen this insidious little gremlin too many times. It's not your code. It's not your Node.js version (probably). It's a subtle, infuriating interaction between Node’s filesystem watching mechanisms, Docker’s storage backend, and specific Linux kernel versions. Specifically, OverlayFS. And it's a silent killer of developer velocity.

The Ghost in the Machine: Symptoms

This isn't a total failure, which makes it far worse. It's an intermittent, frustrating problem. You’ll notice:

• New files go unnoticed: You create a new component, save it, and your dev server just... doesn't see it. The next time you restart, it's there.
• Rapid modifications are lost: You save a file, then quickly save again (maybe a linter auto-saves, or you hit Cmd+S twice). The second save's changes are often missed.
• Stale assets in CI: Your Dockerized build steps produce artifacts that are based on older versions of your source code, even though the latest code is mounted.
• It's container-specific: If you run npm run dev directly on your host machine, everything works perfectly. But inside the container? Chaos.

Environments Where This Beast Thrives

This issue is particularly prevalent in the following combinations. Note the Linux kernel version and the Docker storage driver:

Host OS/Kernel	Docker Storage Driver	Node.js Versions Affected	Observed Behavior
Ubuntu 20.04 (Kernel >= 5.10)	Overlay2	14.x, 16.x, 18.x, 20.x	Intermittent missed IN_CREATE and rapid IN_MODIFY events.
Debian 11 (Kernel >= 5.10)	Overlay2	14.x, 16.x, 18.x, 20.x	Similar to Ubuntu, particularly with deeply nested volume mounts.
Fedora 34+ (Kernel >= 5.10)	Overlay2	14.x, 16.x, 18.x, 20.x	Manifests in CI pipelines using temporary build containers.
CentOS Stream 8/9 (Kernel >= 5.10)	Overlay2	14.x, 16.x, 18.x, 20.x	Can lead to unexpected caching issues in development workflows.

You’ve probably already checked your /etc/sysctl.d/99-sysctl.conf for fs.inotify.max_user_watches and max_user_instances. You’ve probably bumped them up to absurd numbers. Didn't help, did it? Because that's not the problem. This isn't about limits; it's about events getting swallowed.

How to Confirm You're Hit

1. Inside your container, run df -Th. Look for overlay as the filesystem type for your mounted volume. If it's there, you're a prime candidate.
2. Run a minimal watcher script: Create a file named watcher.js with this content:

const fs = require('fs');
const path = require('path');

const dirToWatch = process.argv[2] || '.';
console.log(`Watching directory: ${path.resolve(dirToWatch)}`);

fs.watch(dirToWatch, { recursive: true }, (eventType, filename) => {
  console.log(`Event: ${eventType}, File: ${filename}`);
});

console.log('Watcher started. Try creating/modifying files.');

3. Start it: node watcher.js /app/src (or wherever your code is).
4. In another terminal, inside the container:
   • touch /app/src/test.txt (should log create)
   • echo "hello" >> /app/src/test.txt (should log change)
   • rm /app/src/test.txt (should log rename or change/unlink)
   • THE TEST: touch /app/src/new-file.txt && echo "content" > /app/src/new-file.txt (do this quickly). Observe if both create and change events are reliably logged. Often, only create (or nothing) shows up.

The Root Cause

Here’s the deal: modern Linux kernels (specifically 5.10 and newer) introduced some optimizations or changes in how OverlayFS handles inode numbers and metadata updates, particularly for the 'upper' layer where changes are written. When you're using Docker's overlay2 storage driver, your bind-mounted volumes are often presented via OverlayFS within the container filesystem structure.

Node.js's fs.watch (and libraries like Chokidar that depend on it) uses inotify under the hood. Inotify monitors changes to files and directories using unique inode numbers. The problem arises when a file is created and then modified extremely rapidly (e.g., a file is created by a build tool, immediately written to, or an editor auto-saves). In these specific scenarios, the OverlayFS layer might momentarily present an inconsistent state or coalesce/miss distinct inotify events (IN_CREATE followed by IN_MODIFY for the same inode path) before the watch descriptor can properly track the subsequent modification event. It's a race condition at the kernel-VFS-inotify interface, exacerbated by how OverlayFS tracks changes to files that didn't exist in the lower layer.

It’s not just Node.js that suffers from these kinds of obscure file system interactions. Other low-level kernel nuances can similarly trip up applications, like the Alpine's Silent DNS Killer: The ndots:1 Trap, demonstrating that containerization introduces its own set of fascinating challenges.

The Solution: Polling. Yes, Polling.

You hate to hear it. I hate to say it. But for reliable file watching in these specific Docker/OverlayFS environments, you often have to revert to polling. It’s resource-intensive, but it’s robust.

If you're using Chokidar (which you probably are if you're doing anything serious with file watching in Node), you need to explicitly enable polling. This bypasses the faulty inotify mechanism and instead periodically checks the filesystem for changes. Yes, it's inefficient, but it works.

The Copy-Pasteable Fix for Chokidar

Modify your Chokidar configuration. If you're using a framework, find where it configures Chokidar (e.g., Webpack dev server, Next.js, Vite).

const chokidar = require('chokidar');

const watcher = chokidar.watch('/app/src', {
  ignored: /node_modules/,
  persistent: true,
  ignoreInitial: true,
  // THE CRITICAL LINE:
  usePolling: true, 
  interval: 100, // Check every 100ms
  binaryInterval: 300 // For binary files, check less frequently if needed
});

watcher
  .on('add', path => console.log(`File ${path} has been added`))
  .on('change', path => console.log(`File ${path} has been changed`))
  .on('unlink', path => console.log(`File ${path} has been removed`));

console.log('Chokidar watcher started with polling.');

Set interval to something sensible. Too low, and you'll hammer your CPU and disk. Too high, and you'll introduce unacceptable latency for dev workflows. 100-300ms is usually a good starting point.

Why This Matters

Reliable development environments are not a luxury; they are a necessity for productivity. Chasing down issues like this feels like a waste of engineering time, but it’s crucial for maintaining hyperscale systems where every minute of developer friction adds up. You can't achieve high uptime and rapid iteration if your core tooling is fundamentally broken.

Don't let this phantom watcher consume your soul. Enable polling, get your hot-reloading back, and move on to actually shipping features. You’re welcome.