Loading...
Loading...
Free 550B (55B active MoE) flagship reasoning model with large context
Take Nemotron 3 Ultra apart, layer by layer. The 3D teardown below shows its 108 transformer layers (0 dense + 108 MoE), GQA attention, and expert routing, with values from the public model config.
NVIDIA Research is the research arm of NVIDIA Corporation, the Santa Clara-based computing company founded in 1993; the research team itself was founded in 2006 by chief scientist David Kirk and now spans more than 200 scientists working on AI, graphics, high-performance computing and autonomous vehicles. Its Nemotron line of open models restarted its numbering in 2025: Nemotron-4 (2024) was followed by Nemotron Nano 2 (2025) and then the Nemotron 3 family of Nano, Super and Ultra models, all released with open weights.
more about NVIDIA Research →8192-dim vectors over a 131,072-token vocabulary. The wide hidden dim supports the large expert capacity below.
A deep stack that interleaves Mamba2 SSM blocks with attention + MoE blocks. The SSM layers give linear-time state mixing over long contexts (262K in config, up to 1M when served with a longer max length); the attention layers do precise retrieval. (The scene renders the full stack as transformer slabs.)
64 query heads share just 2 KV heads via GQA, an aggressive KV compression that complements the Mamba layers. Used on the attention subset of the hybrid stack.
A router scores 512 experts per token and activates 22 plus a shared expert, spending 55B of 550B per token. The very high expert count (and 108 layers) is what reaches the 550B total.
Final norm then projection to 131,072 logits. Nemotron 3 Ultra is tuned for agentic reasoning over very long contexts.