Loading...
Loading...
Free 12B multimodal reasoning model with hybrid architecture for OCR, chart reasoning, and vision tasks
Take Nemotron Nano 12B v2 VL apart, layer by layer. The 3D teardown below shows its 62 transformer layers and GQA attention, with values from the public model config.
NVIDIA Research is the research arm of NVIDIA Corporation, the Santa Clara-based computing company founded in 1993; the research team itself was founded in 2006 by chief scientist David Kirk and now spans more than 200 scientists working on AI, graphics, high-performance computing and autonomous vehicles. Its Nemotron line of open models restarted its numbering in 2025: Nemotron-4 (2024) was followed by Nemotron Nano 2 (2025) and then the Nemotron 3 family of Nano, Super and Ultra models, all released with open weights.
more about NVIDIA Research →5120-dim vectors over a 132,096-token vocabulary. A vision encoder turns images and video frames into tokens that join this same stream.
Not a uniform transformer: 28 Mamba-2 blocks, 28 MLP blocks and only 6 attention blocks, interleaved. Mamba-2 mixes tokens through a fixed-size state, so most of the stack costs the same however long the prompt is. The scene draws every block as attention + FFN; read this note for the real layout.
Only 6 of the 62 blocks attend: 40 query heads share 8 KV heads. These few layers handle precise lookups the Mamba-2 state cannot, and they are the only blocks that grow a KV cache, which is why it stays so small.
28 standalone MLP blocks with a 20,480 intermediate dim and Squared ReLU activation. No experts: all 12.6B params are active for every token.
Final RMSNorm then 131,072 logits. The VL variant routes image and text tokens through this same head for unified multimodal output.