Loading...
Loading...
Open-source reasoning model rivalling o1, 671B params (37B active), transparent thinking tokens
Take DeepSeek R1 (0528) apart, layer by layer. The 3D teardown below shows its 61 transformer layers (3 dense + 58 MoE), MLA attention, and expert routing, with values from the public model config.
DeepSeek is a Chinese AI lab founded in July 2023 in Hangzhou by Liang Wenfeng. It was funded solely by High-Flyer, the quantitative hedge fund Liang co-founded, until a $7 billion Series A in May 2026, and grew out of GPU capacity originally stockpiled for trading research. The lab releases its V, R and V4 model lines as open weights under the MIT licence, with the V4 Pro and Flash tiers succeeded by V4.1 Flash in September 2026.
more about DeepSeek →Each input token id maps to a 7168-dim vector from a 129,280-token vocabulary. Identical base to V3; R1 inherits the same tokenizer and embedding table.
R1 shares V3's pre-norm block exactly: RMSNorm, attention, residual add, RMSNorm, FFN, residual add. The first 3 layers are dense; the remaining 58 are MoE. The reasoning behaviour comes entirely from RL post-training, not architecture changes.
128 query heads share a compressed 512-dim KV latent, the same MLA as V3. This keeps the KV cache small for long reasoning chains that span many thousands of tokens.
A router scores 256 experts per token and activates 8 plus 1 shared. R1 spends 37B of its 671B parameters per token, the same sparse routing as V3.
After a final RMSNorm, the hidden state projects to 129,280 logits. R1 was trained with GRPO reinforcement learning to emit <think> reasoning traces before the final answer, improving math, code, and logic benchmarks without changing the architecture.