Quick Summary: Troubleshoot Node.js `fs.watch` and `chokidar` silent failures on EKS Bottlerocket nodes using NFS/EFS. Uncover `inotify` limits and fix silent co...

Alright, listen up. You've got a Node.js application running happily on your local machine, picking up config changes, hot-reloading like a champ. Then you deploy it to Kubernetes on EKS, using Bottlerocket OS, mounting your shared config or content via NFS (EFS, whatever). Suddenly, your application is blind. Config changes? Ignored. Hot-reloads? Non-existent. No errors in the logs, just… nothing. Welcome to the infuriating world of silently failing file watchers.

This isn't some obscure Node.js bug, not directly. It’s a gnarly interaction between specific kernel versions, Node.js's underlying libuv, and the default limits imposed by Linux's inotify subsystem. When it happens, your app just stops seeing filesystem events. No crash, no loud error. Just silence.

The Symptom: Your App Is Deaf to Change

The core symptom is simple: anything relying on file system watching (like fs.watch, chokidar, webpack's watch mode, or dynamic config loaders) just… stops working after a while, or never starts working reliably under load. You change a file, and your application never reacts. It's stable, not crashing, but functionally crippled.

You'll pull your hair out checking network mounts, permissions, even Node.js versions. You'll blame EFS latency. You'll wonder if it's the same EIO issue we debugged with fs.readdir on NFSv4, but you won't see an EIO. This is worse: it's a ghost in the machine.

The Environments Where This Fester

This problem is particular. It typically surfaces in environments with high file watch activity and specific kernel configurations, especially within container orchestration platforms.

Component	Affected Versions/Configurations	Notes
Host OS Kernel	Linux Kernel < 5.15 (especially older Bottlerocket OS versions like 1.7.x - 1.9.x)	Newer kernels (5.15+) often have higher defaults or better handling.
Node.js Runtime	Node.js 16.x, 18.x, 20.x	The specific Node.js version is less critical than libuv's interaction with the kernel.
Filesystem	NFSv4/EFS mounts, other network filesystems	Aggravated by these, but can occur on local filesystems with enough watches.
Container Orchestration	Kubernetes (EKS specifically on Bottlerocket)	Defaults on managed node groups are often conservative.

You’re essentially hitting the wall of kernel resource limits without explicit error reporting to the application layer. Not great.

The Root Cause

The problem is the Linux kernel's inotify watch limits. Specifically, fs.inotify.max_user_watches and fs.inotify.max_user_instances. Each process (or rather, each user, but within a container, it's typically one process/user) can only register a finite number of directory or file watches. Every time Node.js's fs.watch or a library like chokidar sets up a watch, it consumes one of these resources.

When the application attempts to add a new watch and the limit is already exhausted, the underlying inotify_add_watch syscall fails with ENOSPC. Here's the kicker: libuv (what Node.js uses) and consequently Node.js itself, often don't translate this specific failure into a JavaScript error that gets thrown or emitted. It simply fails to add the watch. Your application thinks it has a watcher in place, but the kernel silently refused it. It's a fundamental architectural oversight in how these resource exhaustion scenarios are communicated across layers.

Network filesystems like NFS or EFS can sometimes exacerbate this because watching deep directory trees, especially large ones, can quickly consume thousands of watch descriptors. If your application or a dependency tries to watch `node_modules` on an EFS volume, you're doomed.

The Fix: Increase inotify Limits

Since the problem is a kernel limit, the solution is to increase those limits on the host operating system. This is not something you can generally do from inside an unprivileged container. You need to modify the sysctl parameters on the underlying Kubernetes worker nodes.

For Bottlerocket OS on EKS, this means you'll need to use a DaemonSet with privileged permissions or apply an appropriate Bottlerocket HostContainer. The goal is to set fs.inotify.max_user_watches and fs.inotify.max_user_instances to sufficiently high values.

A good starting point for a busy Node.js app, especially one using chokidar on a large shared filesystem, would be 262144 for watches and 1024 for instances. These numbers might seem high, but for applications that aggressively watch file trees, they are necessary. We've seen setups require even higher for large enterprise applications. Remember, a single user can have multiple watch instances.

If you're dealing with hundreds of microservices or large monolithic apps, this is where engineering humility comes into play when scaling distributed systems. Don't assume defaults will hold. Know your kernel.

Here’s how you'd typically apply this via a privileged Kubernetes DaemonSet:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: inotify-limiter
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: inotify-limiter
  template:
    metadata:
      labels:
        name: inotify-limiter
    spec:
      hostNetwork: true # Not strictly required, but often used for DaemonSets that touch host settings
      containers:
      - name: inotify-limiter-container
        image: ubuntu:latest # Or any lightweight image with sysctl
        command: ["/bin/sh", "-c"]
        args:
          - |
            echo "Increasing inotify limits..."
            sysctl -w fs.inotify.max_user_watches=524288
            sysctl -w fs.inotify.max_user_instances=2048
            # Apply changes permanently for future reboots if not using a host container config for Bottlerocket
            echo "fs.inotify.max_user_watches=524288" >> /etc/sysctl.conf
            echo "fs.inotify.max_user_instances=2048" >> /etc/sysctl.conf
            sysctl -p
            echo "Inotify limits set. Sleeping indefinitely."
            sleep infinity
        securityContext:
          privileged: true # CRITICAL: This DaemonSet needs privileged access to modify host sysctl
        volumeMounts:
          - name: sysctl-conf
            mountPath: /etc/sysctl.conf
      volumes:
        - name: sysctl-conf
          hostPath:
            path: /etc/sysctl.conf
            type: FileOrCreate # Ensure the file exists if it doesn't
      nodeSelector:
        kubernetes.io/os: linux # Target Linux nodes

Warning: Running privileged DaemonSets requires extreme caution. Ensure you understand the security implications. Alternatively, for Bottlerocket, consider using a HostContainer with appropriate configurations, which is the recommended Bottlerocket way to manage host settings.

Verification

After applying the fix, ensure your applications are restarted on the affected nodes. Then, verify the limits from within a pod on the node:

kubectl exec -it <your-pod-name> -- cat /proc/sys/fs/inotify/max_user_watches
kubectl exec -it <your-pod-name> -- cat /proc/sys/fs/inotify/max_user_instances

These commands should now reflect the higher values. Your application should start detecting file changes again without complaint. If you're still seeing issues, check dmesg on the host itself for any other inotify related errors or other resource exhaustion messages. This particular bug taught us that silent failures are the worst kind. Good luck out there, you'll need it.