I Tested Every Best Local AI For GPU (8GB TO 24GB VRAM)

Access ChatLLM Here - https://chatllm.abacus.ai/kai Two 2-bit files of the same model, Qwen 3.8 27B, at the same size: one fixed 16 of 20 real bugs, the other fixed 7, and the worse one was the bigger file. That's why this isn't another tier list. On 8GB to 24GB of VRAM the model name matters less than which file you download and whether it fits in your GPU's memory. We go card by card. On 8GB, your system RAM is the real tier: a mixture-of-experts model like Qwen 3.6 35B A3B runs from RAM at 23 tokens a second on a 6GB RTX 2060 with 32GB of DDR4. On 12GB, ISTA's GSQ-RCO build beats a bigger 2-bit file of the same model. On 16GB, the default Q4_K_M download doesn't fit once Windows takes its share. And on 24GB, there's room to spare, plus what NVIDIA's new Nemotron 3.5 Bolt 74B "24GB" actually leaves you. This video is sponsored by ChatLLM by Abacus AI. Test your local model against GPT-6 Astra, Claude Fable 5.1, Opus 5.5, Gemini 4, DeepSeek V4.1 and Kimi K3 in one window, plus image and video generation, slide decks and app building, for $10 a month: https://chatllm.abacus.ai CHAPTERS 0:00 Two and a half hours for one page 1:16 The rule every tier list uses 1:52 What the comment sections said 2:29 Why spilling into system RAM is so slow 3:58 Mixture of experts: the second rule 4:53 8GB: your RAM is the real tier 6:18 Sponsor: ChatLLM by Abacus AI 8:02 12GB: whose 2-bit file you download 10:05 16GB: the default Q4_K_M trap 11:43 24GB: room to spare, then Nemotron 3.5 Bolt 13:27 Tokens per second vs minutes per task 14:43 What to download for your card WHAT TO DOWNLOAD - 8GB: with 32GB of system RAM, Qwen 3.6 35B A3B with the experts in RAM. Without it, Gemma 4 12B (Q3_K_M, 5.69GB) for chat. - 12GB: Qwen 3.8 27B, ISTA GSQ-RCO. 3-bit if you have room for context, 2-bit (IQ2_XS, 8.42GB) if you don't. - 16GB: ISTA GSQ-RCO IQ3_S (11.77GB), not the 16.46GB Q4_K_M. Squeeze the KV cache before you shrink the model. - 24GB: plain Q4_K_M with long context. Give Bolt a few weeks of real-world reports. Whatever the size, check who made the file before you check how many bits it has. MODELS AND FILES (Hugging Face) - Qwen 3.8 27B, ISTA GSQ-RCO: https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF - Qwen 3.8 27B, Unsloth: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF - Qwen 3.8 27B, AtomicChat: https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF - Qwen 3.6 35B A3B: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF - Gemma 4 12B: https://huggingface.co/unsloth/gemma-4-12b-it-GGUF - LFM2.5 24B A2B: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B - Nemotron 3.5 Bolt 74B-A9B 24GB: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF SOURCES - PatchEval, 16/20 vs 7/20 on one RTX 3060 (@DogukanUrker): https://x.com/DogukanUrker/status/2094734713071997041 - LFM2.5 24B A2B, 87 tok/s and 8/20 (@DogukanUrker): https://x.com/DogukanUrker/status/2110062837963903485 · https://x.com/DogukanUrker/status/2110258774468448607 - Qwen 3.8 27B on an RTX 4060 8GB (Red Stapler): https://www.youtube.com/watch?v=ye50BbXEczo - Qwen 3.6 35B A3B on an RTX 2060 6GB: https://www.reddit.com/r/LocalLLaMA/comments/1x27au9/ - Windows VRAM overhead (VramRaccoon): https://www.youtube.com/watch?v=WzQM-zt3SY4 - Qwen 3.8 27B on 12GB, 16 of 64 layers need KV cache: https://www.reddit.com/r/LocalLLaMA/comments/1wzha67/ - KV cache quant vs model quant, 189 benchmarks: https://www.reddit.com/r/LocalLLaMA/comments/1x6fwcd/ - RTX 3090 with MTP, 31.0 → 41.3 tok/s: https://github.com/sudoingX/qwen38-mtp - Nemotron 3.5 Bolt release (NVIDIA): https://blogs.nvidia.com/blog/local-ai-rtx-spark-laptops-software-updates/ - Bolt on Artificial Analysis: https://artificialanalysis.ai/models/nemotron-3-5-bolt-74b-a9b - Bolt release thread and early tests: https://www.reddit.com/r/LocalLLaMA/comments/1x6o31q/ - 3090 Founders Edition VRAM temperatures: https://www.reddit.com/r/LocalLLaMA/comments/1x4z19k/ All speeds are self-reported by the people linked. File sizes are decimal GB from Hugging Face; a "24GB" card holds 25.77 decimal GB. MORE ON RUNNING AI ON YOUR OWN GPU - Best Local AI Models For Every VRAM Tier: https://youtu.be/mn7CD0JjVpw - I Tested Every Qwen3.8-27B Quant: Here's the Best One For Your GPU: https://youtu.be/hg26j0erUSo - Is 8GB VRAM enough for 27B models? Tested: https://youtu.be/P1K36q76Uzk - Is 16GB All You Need for Serious Local LLM?: https://youtu.be/i9mRnbdDYxk - How I Got Frontier Performance on 16GB VRAM (Qwen 3.8 27B): https://youtu.be/h_KqS4GtU3I - Every Ways to Get 32GB VRAM for Local AI at Full Context: https://youtu.be/uavLRbbfM94 - Every Ways to Get 128GB VRAM for Local AI at Full Context: https://youtu.be/NA9D49kQFuU - I Tested Every Local AI Model So You Don't Have To: https://youtu.be/C9M6iUFtVB4 - DeepSeek Just Shrunk the KV Cache 437×: https://youtu.be/n-_e9bROGHo #LocalLLM #LocalAI #Qwen