Quick Summary: Debugging a rare Node.js native addon crash involving mmap'd memory on Linux, triggered by kernel memory compaction and V8 GC. Fix sporadic SIGSEG...

You’ve seen it. That cold dread in your gut when the pager screams at 3 AM. A production Node.js service, supposedly rock-solid, has crashed. Not gracefully. Not with a clean stack trace pointing to some rookie coding error. No, this is a SIGSEGV. Or worse, a Bus error. Always sporadic. Always under load. And always, always when you’re least prepared. I’ve been there. This particular demon took weeks, copious amounts of coffee, and a deep dive into kernel internals to exorcise. You’re welcome.

The culprit? A nasty interaction between Node.js native addons using mmap, V8’s garbage collector, and specific behaviors of the Linux kernel’s memory management, particularly Transparent Huge Pages (THP) and memory compaction. It’s a silent killer, an elusive bug that makes you question your sanity. You're trying to achieve sub-millisecond performance and you're getting slammed by memory ghosts.

The Symptoms: A Ghost in the Machine

Your application uses a native addon (NAPI or FFI) that leverages mmap(2) to allocate large, shared memory regions. Perhaps it's for inter-process communication, a custom data structure residing off-heap, or to load massive ML model weights directly into memory. Everything seems fine during development. Then, in production, under sustained high memory pressure or specific data access patterns, the process abruptly dies.

• Error Message: Typically SIGSEGV (Segmentation Fault) or Bus error.
• Location: Stack traces are often useless, pointing to low-level V8 internal functions (e.g., during a GC sweep or mark phase), or sometimes directly into the native addon code.
• Reproducibility: Infuriatingly intermittent. Might happen after minutes, hours, or even days of uptime. Often correlates with system-wide memory pressure or sudden spikes in data processing.

The Trigger Environment

This isn't a universal bug. It’s a specific confluence of factors. Before you tear your hair out, confirm you're running in one of these lovely conditions:

Operating System	Kernel Version Range	Node.js Version Range	Contributing Factors
Linux (Any Distribution)	4.9.x to 5.10.x (especially 5.x)	12.x, 14.x, 16.x	Transparent Huge Pages (THP) enabled (default on many systems), NUMA balancing active, high memory pressure.
Linux (Any Distribution)	5.11.x to 6.x	16.x, 18.x, 20.x	Aggressive memory compaction, specific cgroup v2 memory limits.

Initial Misdirections (Don't Waste Your Time Here)

I’ve seen engineers spend days chasing ghosts here. Don't be that engineer. Common pitfalls like obvious bad pointers in C++ or FFI/NAPI type mismatches usually lead to deterministic, easier-to-diagnose crashes. This one is different.

The Deep Dive: Unmasking the Culprit

Alright, let's get dirty. You need to verify if your system is exhibiting the exact conditions that trigger this specific nightmare.

Step 1: Confirm Native Addon mmap Usage

First, be absolutely sure your native addon is directly calling mmap(2). This is the lynchpin. If it's using standard heap allocation (malloc/new), you're barking up the wrong tree. Look for code like mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0) or similar.

Step 2: Check Kernel Memory Configuration

This is critical. Is THP enabled? Is NUMA active? Memory compaction aggressive? These are the kernel-level behaviors that create the perfect storm for this bug.

cat /sys/kernel/mm/transparent_hugepage/enabled
# Expected output: [always] madvise never (or similar showing 'always' is active)

cat /sys/kernel/mm/transparent_hugepage/defrag
# Expected output: [always] defer madvise never (or similar showing 'always' is active)

# For NUMA systems:
numactl --hardware
# Look for multiple nodes. If active, NUMA balancing could be in play.

# Check dmesg for compaction events (especially under load or after a crash):
dmesg | grep "compaction"
# Look for "Direct compaction" or "kswapd" activity that frequently reclaims or moves pages.

If THP is [always] and you're seeing frequent compaction messages, you're on the right track.

Step 3: Analyze Core Dumps (If You Can Get Them)

Configure your system to generate core dumps (ulimit -c unlimited). When a crash occurs, use gdb:

gdb -c core /path/to/your/node/executable
bt full
info registers

Look for the program counter (rip on x86_64) pointing to an address within a V8 internal function, or an address that appears to be within your mmap'd region but causes a fault. The faulting address itself might appear 'valid' but the underlying physical page is gone or moved.

The Root Cause

Here’s the deal: When your native addon uses mmap, it directly asks the kernel for memory. Node.js (and V8) might get a pointer to this memory. While V8 is generally good at treating external memory as, well, external, sometimes pointers to these regions *can* become internalized or part of an internal scan during a full GC cycle if not explicitly managed. The insidious part starts when Linux’s memory management kicks in. THP aims to reduce TLB misses by using larger pages, and memory compaction (especially under pressure or with NUMA balancing) will actively move physical pages around to create contiguous blocks. If your mmap'd region is not 'locked' in physical memory, the kernel is free to move its underlying physical pages or even unmap them entirely if it decides they are unused. If V8 then tries to access an address that, virtually, *should* be valid but physically points to garbage, or worse, has been unmapped or moved, you get a SIGSEGV or Bus error. The virtual address mapping stays, but the physical backing changes or vanishes, creating a "mirage" that only a memory access attempt can reveal. This is similar to how unhandled concurrent access issues can arise in shared memory systems, which we've discussed in our deep dive into Node.js worker_threads deadlocks with native addons.

The Solution: Lock It Down

The fix is to explicitly tell the kernel: "Hands off this memory!" You need to prevent the kernel from moving or swapping out the physical pages backing your mmap'd region. The way to do this is using mlock(2) or specifying MAP_LOCKED in your mmap call.

You’ll need to modify your C++ native addon code. This is a crucial configuration change.

#include <sys/mman.h> // For mmap, mlock
#include <node_api.h> // For NAPI interaction (if applicable)
#include <errno.h>    // For errno

// Example: Modified mmap call in your native addon
void* allocateLockedMemory(size_t size) {
    void* addr = mmap(
        NULL,
        size,
        PROT_READ | PROT_WRITE,
        MAP_PRIVATE | MAP_ANONYMOUS | MAP_LOCKED, // <-- CRITICAL CHANGE HERE: MAP_LOCKED
        -1,
        0
    );

    if (addr == MAP_FAILED) {
        // Handle error, e.g., log errno, throw NAPI exception
        fprintf(stderr, "mmap failed with errno: %d\n", errno);
        return nullptr;
    }

    // You can also use mlock() after mmap() if MAP_LOCKED isn't available
    // or you need more fine-grained control:
    // if (mlock(addr, size) == -1) {
    //     fprintf(stderr, "mlock failed with errno: %d\n", errno);
    //     munmap(addr, size); // Clean up mmap if mlock fails
    //     return nullptr;
    // }

    return addr;
}

// Remember to munlock and munmap when done!
void freeLockedMemory(void* addr, size_t size) {
    // if (munlock(addr, size) == -1) {
    //     fprintf(stderr, "munlock failed with errno: %d\n", errno);
    // }
    if (munmap(addr, size) == -1) {
        fprintf(stderr, "munmap failed with errno: %d\n", errno);
    }
}

A Word of Caution: Using MAP_LOCKED or mlock consumes system memory that cannot be swapped out. Use it judiciously. Only lock memory that absolutely needs to be physically resident and unmovable. Ensure you have enough physical RAM for your application's total locked memory footprint, plus the rest of the system. Excessive use can lead to OOM conditions elsewhere or reduce overall system performance.

Final Thoughts

This bug is a testament to the complexities of modern systems, where high-level runtimes interact with low-level kernel mechanisms. Pinning memory prevents the kernel from playing its page-moving games, thus eliminating the race condition that causes V8 to fault on a "mirage" of memory. Implement this change, monitor your systems closely, and finally, get some sleep. You’ve earned it.