Loading...
Loading...
The official release of DeepSeek V4 Pro, a 1.7T MoE superseding the April preview with greatly enhanced agentic capabilities, 1M context, and optional reasoning effort
Take DeepSeek V4 Pro 0813 apart, layer by layer. The 3D teardown below shows its 61 transformer layers (0 dense + 61 MoE), MLA attention, and expert routing, with values from the public model config.
DeepSeek is a Chinese AI lab founded in July 2023 in Hangzhou by Liang Wenfeng. It was funded solely by High-Flyer, the quantitative hedge fund Liang co-founded, until a $7 billion Series A in May 2026, and grew out of GPU capacity originally stockpiled for trading research. The lab releases its V, R and V4 model lines as open weights under the MIT licence, with the V4 Pro and Flash tiers succeeded by V4.1 Flash in September 2026.
more about DeepSeek →7168-dim vectors over a 129,280-token vocabulary, the same width as V3 and unchanged from the 0423 preview.
Pre-norm blocks with Manifold-Constrained Hyper-Connections (mHC) replacing plain residual adds. The first 3 layers use static Hash-MoE routing, the remaining 58 use learned MoE routing. A DSpark speculative decoding module is attached at layers 58-60 (not drawn).
Compressed Sparse Attention and Heavily Compressed Attention with a 128-token local sliding window: 128 query heads onto a single 512-dim compressed KV head with a 64-dim RoPE part. The sparse indexer (64 heads, top-k 1024) makes the 1M context tractable. (Rendered here as the MLA block.)
384 routed experts, 6 active per token, plus 1 shared expert, each 3072 wide. The router scores with sqrt(softplus) affinity. Pro activates 49B parameters per token; the 1.65T checkpoint is the 1.6T backbone plus the DSpark draft module.
Per-head RMSNorm then a projection to 129,280 logits. DSpark drafts several tokens semi-autoregressively and the main model verifies them; 0813 pairs that with the agentic post-training that separates it from the preview.