How Jane Street Saves Logs from a Dying Linux Kernel #Linux #SystemsEngineering #JaneStreet #nvidia

How Jane Street Saves Logs from a Dying Linux Kernel When a Linux machine crashes, the very tools you need to diagnose the failure usually go down with it. Here is how Jane Street engineer Jacob Root solved the ultimate observability challenge: building a kernel log shipper that can survive when disk, network, and storage stacks fail completely. Key Engineering Tactics Used: mlockall() to pin executable pages into RAM so missing disk access can't crash the binary. Reading directly from /dev/kmsg instead of relying on standard disk-based utilities. Bypassing disk-based DNS and TLS certificates by caching IPs and blasting raw UDP packets. Using Linux pstore to dump compressed, newest-first crash buffers into motherboard NVRAM/firmware during a total kernel panic. The result? Catching a silent, intermittent null-pointer dereference inside an NVIDIA driver triggered by Chrome. "Anything that reports a failure has to outlive it." #Linux #SystemsEngineering #Programming #SoftwareEngineering #JaneStreet #KernelPanic #DevOps #TechShorts