Quick Summary: Stuck with `EIO` or `ENOSPC` from Node.js `fs.readdir` on NFSv4 mounts? This guide reveals the obscure kernel, Node.js, and NFS config clash causi...

Alright, another day, another seemingly impossible Node.js filesystem error. You've seen the EIO: I/O error or ENOSPC: No space left on device popping up from fs.readdir calls. Your disks have space. Your inodes are fine. Everything looks okay. But your Node.js application, which relies heavily on directory traversal, is throwing a fit, especially in your staging or production environments. Sound familiar? Good. You're in the right place.

This isn't your garden-variety file permission error. This is a subtle, insidious beast that only manifests under specific conditions. You'll see fs.readdir either hang indefinitely, return incomplete results, or bail out with an EIO or, bafflingly, ENOSPC. The kicker? It's intermittent, making it a nightmare to reproduce reliably outside of heavy load. It often hits when you're traversing directories with tens of thousands, or even hundreds of thousands, of files, particularly if these directories are deeply nested or contain a mix of very old and new files. Your application performance tanks, processing queues back up, and your alerting system goes bananas.

This specific flavour of hell primarily impacts Node.js applications running on older Linux kernels over NFSv4. We've seen it hit these combinations hard:

Operating System	Kernel Version	Node.js Version Range	Filesystem
CentOS 7 / RHEL 7	3.10.0-xxx.el7	14.x, 16.x	NFSv4
Ubuntu 18.04 LTS	4.15.0-xxx-generic	14.x, 16.x	NFSv4

Note: Node.js 18.x and above, or newer kernels (e.g., CentOS 8 / RHEL 8 with 4.18.x+, Ubuntu 20.04+ with 5.4.x+) are far less susceptible, if at all. This is critical context.

Don't waste time checking disk space; you already know it's there. Don't immediately suspect application logic; while that's always a good sanity check, if this just started happening in a stable app, it's probably not it. And for god's sake, don't just blindly increase ulimit -n thinking it's too many open files. Those are dead ends that will cost you hours. This problem is deeper.

The Root Cause

This is where it gets nasty. The root cause isn't a Node.js bug, per se. It's a fundamental interaction flaw between older Linux kernel's NFS client implementation, the readdir system call, and the way Node.js (via libuv) interacts with the kernel's virtual filesystem (VFS) layer. Specifically, older NFSv4 client implementations on certain kernels have an archaic, fixed-size buffer for handling readdir responses. When a directory contains a massive number of files, or filenames are particularly long, the NFS server sends back a readdirplus response that simply overflows the client's internal rpc.max_payload_size or related RPC buffer. The client kernel then sees this overflow and, instead of gracefully handling it or requesting smaller chunks (which newer kernels do), it just bails out with an EIO or, confusingly, ENOSPC, because it effectively 'ran out of space' in its internal processing buffer, not on the disk itself. Node.js then propagates this low-level kernel error. It’s a silent killer, much like the issues we've seen with Node.js fs.watch and NFSv3 polling hell, but for readdir.

There's a kernel parameter that can alleviate this specific pain. You need to increase the maximum RPC payload size the NFS client is willing to handle. This isn't just about rsize/wsize mount options; those control data transfer size. This is about the metadata payload for directory listings. This fix makes the NFS client able to process larger directory listing responses without overflowing its internal buffer.

# Apply this on the NFS CLIENT machine experiencing the Node.js issue.
# This increases the maximum payload size for NFS RPC requests.
# The default is often 1MB (1048576). We're bumping it to 4MB or 8MB.
# This should be done carefully and monitored.

echo "options sunrpc tcp_max_payload_size=8388608" | sudo tee /etc/modprobe.d/sunrpc.conf
echo "options sunrpc udp_max_payload_size=8388608" | sudo tee -a /etc/modprobe.d/sunrpc.conf
sudo modprobe -r sunrpc
sudo modprobe sunrpc

# For immediate effect without reboot (might require unmounting/remounting NFS shares):
# On some kernels, you might also need to explicitly set rsize/wsize on mount:
# sudo mount -o remount,rsize=8192,wsize=8192 /your/nfs/mountpoint
# However, the modprobe change is usually sufficient for readdir.

# Verify the change (this might be tricky as it's often an internal kernel value).
# You can sometimes see it reflected indirectly in /proc/sys/sunrpc/tcp_max_payload_size
# or by checking module parameters with `modinfo sunrpc`.
# The actual effect is within the RPC layer.

Here’s how to apply this critical fix:

1. Identify Affected Clients: Pinpoint the Node.js application servers exhibiting this EIO/ENOSPC behavior over NFSv4.
2. Backup Existing Configuration: Before touching anything, back up /etc/modprobe.d/sunrpc.conf if it exists, or just note its absence.
3. Apply the modprobe Configuration: Execute the echo "options..." commands as shown above. This creates or appends to a modprobe configuration file, instructing the kernel to load the sunrpc module with a larger tcp_max_payload_size and udp_max_payload_size. We recommend 8MB (8388608 bytes) as a starting point.
4. Reload the sunrpc Module: Run sudo modprobe -r sunrpc followed by sudo modprobe sunrpc. This will unload and reload the RPC module with the new options. WARNING: This might briefly interrupt NFS traffic on the client. Plan for a maintenance window.
5. Test Thoroughly: After applying the fix, restart your Node.js application and push some traffic through. Monitor logs closely for recurring EIO or ENOSPC errors. Run your heaviest directory traversal jobs.

By increasing tcp_max_payload_size for the sunrpc module, you're essentially telling the NFS client's RPC layer, 'Hey, when you get those large readdirplus responses from the server, you've got more internal buffer space to work with. Don't choke and throw an I/O error.' This parameter is often overlooked because rsize/wsize are the more common NFS tuning parameters. But for metadata-heavy operations like readdir, especially on older kernels, this RPC payload size becomes the bottleneck. While your shiny new Next.js or Remix app might be blazing fast on local SSDs, network filesystems introduce a whole different league of pain, often at these obscure kernel interaction layers.

This isn't a silver bullet for all NFS woes, but it's a critical, often-missed piece of the puzzle for fs.readdir issues on specific Node.js/Linux/NFSv4 configurations. Always keep your kernels updated, and understand that network filesystems, while convenient, introduce complex failure modes that local filesystems simply don't have. Dig into those kernel docs, because sometimes, the bug isn't in your code or even your immediate infrastructure, but in a decades-old default value hidden deep within a module parameter.