https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF
https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF
NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally. Featuring native 2-bit quantization, it fits within 24GB of VRAM while running alongside other applications. Optimized for intelligence and speed per gigabyte of VRAM, it strikes an optimal balance between performance and capability: running faster than most mainstream dense models within the same VRAM footprint, while delivering higher intelligence than most mainstream MoE models of comparable memory usage.
The model is ready for both commercial and non-commercial use.
Model Summary
Total Parameters
74B (9B active parameters)
Architecture
Hybrid Mamba-2–Transformer LatentMoE with C-RADIO v4-H vision encoder
Precision
IQ2 (2-bit quantization via Quantization-Aware Training (QAT))
Context Length
Up to 256K tokens
Input / Output
Text, Image, Video / Text
Deployment
24GB packaged model; supports deployment via vLLM on NVIDIA Ampere (A100), Hopper (H100-80GB), or Blackwell architectures
Speculative Decoding
DFlash-based LiLiCorr draft model; supports 7 speculative tokens in the accompanying Bolt vLLM recipe
Reasoning Mode
Chat template controlled (enable_thinking=True/False), inherited from the Super 3.5 VL client
Recommended Sampling Parameters
Text: temperature 1.0, top_p 0.95 (inherited from the Super 3.5 VL client)
Best Suited For
Running everyday agentic tasks locally
License
OpenMDW-1.1
Model Overview
Model Developer: NVIDIA Corporation
Development Period: December 2025 – May 2026
Data Freshness:
Post-training data cutoff is May 2026.
What is Nemotron?
NVIDIA Nemotron™ is a family of open models providing open weights, training data, and recipes that deliver leading efficiency and precision for building specialized AI agents. NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, featuring native 2-bit quantization to fit within 24GB of VRAM and run alongside other applications. Optimized for intelligence and speed per gigabyte of VRAM, it strikes an optimal balance between speed and intelligence: faster than most mainstream dense models within the same VRAM footprint, and smarter than mainstream MoE models of comparable memory usage.
The model is ready for both commercial and non-commercial use.
bolt!
Top Discussions
jacek2023 (OP) (⬆114): BF16 vs 2-bit (read before you start flaming)
Thiom (⬆69): Deliberately avoiding a comparison with Qwen3.8 27B? Genuinely asking, btw, because Qwen3.8 27B at Q4 fits easily into 24GB of VRAM and completely smokes Nemotron.
Artic_Temperature203 (⬆38): 3.8 27B isn’t an MoE architecture. I’d love to see the comparison too, but I don’t think they’re intentionally being misleading here.
Sea-Wind-2216 (⬆60): Why are they comparing MoE to dense models in the first place?
ForsookComparison (⬆35): A 74B parameter model taking up only 24GB of disk space after QAT is genuinely incredible. For some people, this looks very promising to retire Qwen3.6-35B.
nickm_27 (⬆23): Great stuff, glad to see another model that isn’t just completely codemaxxed. Looking forward to seeing if it can replace my older models in my workflow.
MerePotato (⬆27): On consumer hardware, it’s the best model out there aside from Qwen 3.8 27B—the problem is that 3.8 27B makes everything else look underwhelming. But for users with less than 24GB VRAM, this might be a better fit than 3.8 27B; after all, cramming 3.8 27B in requires a hefty quantization penalty, whereas this model allows offloading a fair number of layers to system RAM alongside spec decoding without taking too much of a speed hit.
Negative-Web8619 (⬆35): Why not compare it to 3.8 35b? smh