llama.cpp Just Made 125B MoE Models 4× Faster!

llama.cpp's new MoE expert cache took a 125B model from 9.7 to 37.2 tokens a second on one 24 GB GPU. Here's where that 4× really comes from, and which cards get slower. The 4× is from lich-77's experimental expert-cache branch (llama.cpp issue #29949), measured on an RX 7900 XTX running Qwen3.8-Flash-Next. Its baseline was -ncmoe 48, which keeps every one of the model's 24,576 experts in system RAM, so the card started out holding none of them. The cache that actually shipped (PR #29887, release 0.7.0, flag --moe-cache-mib) posts 2.2× on an RTX 5090 against llama.cpp's default --fit. On the same machine, the experimental branch was 1.55× faster than the merged one, and pinned memory plus 48 routing round trips per token explain most of that gap. Whether this speeds up your local LLM depends on two things: how much of the expert pile your spare VRAM can hold, which sets your hit rate, and how fast your PCIe slot moves the misses. Below about an 85% hit rate the cache costs you speed. An 8 GB RTX 4060 laptop went from 23.4 to 15 tokens a second, a 12 GB RTX 3060 from 32 to 12, and on a 12 GB RTX 4070 SUPER plain llama.cpp beat every cache size. On a 32 GB RTX 5090 in a full x16 slot it roughly doubled decode. We also cover the Windows pinned-memory fix from October 16th (37.76 to 62.73 tokens a second), what the cache does to prompt speed and speculative decoding, how Strata compares on Qwen3.8-Flash-Next, and a 15-minute test that gives you your own number. CHAPTERS 0:00 9.7 tokens a second on a 24 GB card 3:25 What an MoE model is 6:10 Why the GPU sits idle 7:43 The old fix: whole layers 9:12 How the expert cache works 12:13 Where the 4× actually comes from 13:44 1.55×: pinned memory and 48 waits 16:20 8 and 12 GB cards get slower 20:28 Your PCIe slot 23:03 Prompts, drafts and loops 26:40 The 15-minute test 29:20 Strata vs llama.cpp 31:15 Who wins, who loses SOURCES lich-77's GPU-resident LRU cache (the 4× run), issue #29949: https://github.com/ggml-org/llama.cpp/issues/29949 Merged expert cache, PR #29887 (RTX 4090/5090 table, owner reports in the thread): https://github.com/ggml-org/llama.cpp/pull/29887 Multi-GPU cache and the prompt-speed table, PR #30112: https://github.com/ggml-org/llama.cpp/pull/30112 Pinned memory on Windows, PR #30449: https://github.com/ggml-org/llama.cpp/pull/30449 Release v0.7.0: https://github.com/ggml-org/llama.cpp/releases/tag/v0.7.0 First build with the cache, b11480: https://github.com/ggml-org/llama.cpp/releases/tag/b11480 Qwen3.8-Flash-Next model card: https://huggingface.co/Qwen/Qwen3.8-Flash-Next r/LocalLLaMA PR thread (RTX 5090 and RTX 3060 runs): https://www.reddit.com/r/LocalLLaMA/comments/1x03xkc/ RTX 4060 laptop runs: https://www.reddit.com/r/LocalLLaMA/comments/1x0wyge/ Pinned RAM, 48 syncs per token, Strata same-PC test: https://www.reddit.com/r/LocalLLM/comments/1wz88oq/ 80B on 16 GB cache sweep: https://www.reddit.com/r/llamacpp/comments/1x84r67/ NoCodeAbode, 12 GB RTX 4070 SUPER test: https://www.youtube.com/watch?v=BSEDJFnvdqE Strata README (RTX 5070 speeds): https://github.com/Niko1221/Strata #llamacpp #LocalLLM #LocalAI