Quick Summary: Debugging an obscure Node.js UDP packet loss issue on Ubuntu 20.04 with SO_REUSEPORT under high CPU load in cgroupv1 environments. Uncover the ker...

You’ve seen it. That inexplicable, infuriating UDP packet loss defying all conventional debugging. Your application logs are silent. netstat -su shows no global receive errors. Yet, your service, critical for real-time data, is dropping messages. And it only happens when CPU utilization spikes on shared hosts.

I’ve wasted weeks on this particular hell. It’s not your code. Not a firewall. It’s a nasty interaction: Node.js, Linux kernel network stack, SO_REUSEPORT, and legacy cgroup CPU scheduling under contention. Buckle up.

The Problem: Phantom UDP Drops

You have multiple Node.js instances, perhaps in Docker, all binding to the same UDP port via dgram.createSocket({ type: 'udp4', reusePort: true }). This setup is common for distributed fan-out/fan-in. Everything's fine, until your host hits 80%+ CPU. Then, critical UDP packets just… vanish. We're talking 10-30% loss, intermittent, unpredictable, and invisible to standard network diagnostics.

Affected Environments

This phantom drop is specific, primarily occurring in these conditions:

Operating System	Kernel Version Range	Node.js Version Range	Container Runtime
Ubuntu 20.04 LTS	5.4.0-x to 5.4.0-100	12.x LTS, 14.x LTS, 16.x LTS	Docker (cgroupv1 default)
Debian 10	4.19.x to 5.0.x	10.x LTS, 12.x LTS	LXC, Docker (cgroupv1)

Common thread: older LTS distros, kernels around 5.x, and crucially, cgroupv1 for CPU management. Cgroupv2 users likely dodged this.

Initial (Futile) Troubleshooting

You probably tried:

• Increasing net.core.rmem_max. (Packets aren't reaching socket buffers.)
• Checking NIC drivers. (Rarely the primary cause.)
• Blaming Node.js event loop slowness. (A factor, but not the root.)
• Running tcpdump. (You'll see packets at the interface, not necessarily delivered to your app.)

Deep Dive Diagnostics

Go kernel-level. Forget application metrics.

1. dropwatch: Hooks kernel tracepoints to show exact drop locations. You'll likely see drops related to NAPI budgets or RX queue overflows *before* socket layers.
2. perf top -e net:net_dev_queue_xmit: Look for network processing hotspots. Observe high CPU steal time for network interrupts when your processes are thrashing CPU.

The Root Cause

The problem is an interaction between the Linux kernel's NAPI budget, SO_REUSEPORT load balancing, and cgroupv1 CPU scheduling under severe contention. With multiple Node.js instances using SO_REUSEPORT, the kernel distributes packets to their respective socket receive queues. Under heavy CPU pressure in a cgroupv1 environment, the kernel's scheduler throttles processes. NIC interrupts (packet arrival notification) and NAPI polling (pulling packets from NIC ring buffer) demand CPU cycles. If the CPU scheduler starves these network processing tasks within the cgroup, packets accumulate in the NIC's ring buffer. They eventually overflow and drop *before* reaching a socket's queue. For critical, low-latency applications, microsecond delays can be catastrophic.

The NAPI polling budget (net.core.netdev_budget) might be exhausted, or REUSEPORT distribution fails to assign to a "ready" socket because the target Node.js process isn't scheduled to consume packets. Node.js's single-threaded event loop exacerbates this; if busy, it won't poll the kernel for network events, leading to queue build-up. This loss is distinct from application-level drops and often invisible to netstat because packets never reached a socket's buffer.

This situation is a brutal reminder: latency remains an unforgiving metric, especially in high-performance or real-time environments.

The Fix: Prioritize Network Interrupts with cgroup Tweaks

The solution isn’t boosting Node.js. It's telling the kernel: "Network processing is paramount, even under CPU pressure." You need to guarantee CPU for network interrupt handling.

1. Optional: Pin NICs to CPU Cores: Use irqbalance or manual `/proc/irq` affinity to dedicate cores to NIC interrupts for critical services.
2. Crucial: Adjust cgroup CPU `sched_rt_runtime_us`: Allocate a small, guaranteed CPU slice for real-time tasks within your application's cgroup. This ensures the kernel handles network interrupts related to your app, even under contention.

First, identify your container's cgroup (e.g., /sys/fs/cgroup/cpu/docker/<container_id>). Then:

# Navigate to the CPU cgroup for your container/service
# Example: /sys/fs/cgroup/cpu/system.slice/docker-<container_id>.scope

# Set the CPU period (100ms)
echo 100000 > /path/to/your/cgroup/cpu.rt_period_us

# Allocate 10ms of real-time runtime per period (10% of CPU slice for RT tasks)
echo 10000 > /path/to/your/cgroup/cpu.rt_runtime_us

Explanation: `cpu.rt_period_us` and `cpu.rt_runtime_us` define real-time CPU scheduling within a cgroup. Setting `cpu.rt_runtime_us` reserves a small CPU time slice for real-time tasks. Kernel network processing (NAPI, interrupt handling for SO_REUSEPORT) is critical for packet delivery and can be considered real-time. This creates a dedicated CPU budget for these kernel functions, preventing starvation by user-space contention. Use cautiously; excessive `rt_runtime_us` can cause instability. 10ms out of 100ms (10%) is a safe starting point.

Prevention & Takeaways

• Upgrade to cgroupv2: Migrating often mitigates these issues due to its robust CPU scheduler.
• Isolate Critical Services: Don't co-locate high-PPS UDP services on CPU-contended hosts. Consider dedicated cores.
• Monitor Kernel Metrics: Use dropwatch, perf, and /proc/net/dev. Application logs are often too late for network problems.
• Know Your Kernel: Understand its interaction with network drivers, scheduling, and cgroups.

This problem shows SRE isn't just code or deployments. It's deep stack understanding, down to kernel interactions. Debug with system observability, not just application logs. And a lot of frustration.