Loading...
Loading...
Flagship 1.6T MoE (49B active) with hybrid attention, 1M context, and high/xhigh reasoning for full-codebase analysis and long-horizon agents
Take DeepSeek V4 Pro 0423 (preview) apart, layer by layer. The 3D teardown below shows its 61 transformer layers (0 dense + 61 MoE), MLA attention, and expert routing, with values from the public model config.
DeepSeek is a Chinese AI lab founded in July 2023 in Hangzhou by Liang Wenfeng. It was funded solely by High-Flyer, the quantitative hedge fund Liang co-founded, until a $7 billion Series A in May 2026, and grew out of GPU capacity originally stockpiled for trading research. The lab releases its V, R and V4 model lines as open weights under the MIT licence, with the V4 Pro and Flash tiers succeeded by V4.1 Flash in September 2026.
more about DeepSeek →7168-dim vectors over a 129,280-token vocabulary, the same width as V3; V4 spends its extra parameter budget on expert capacity, not width.
Pre-norm blocks with Manifold-Constrained Hyper-Connections (mHC) replacing plain residual adds. The first 3 layers use static Hash-MoE routing (content-independent), the remaining 58 use learned MoE routing.
Replaces MLA with a hybrid of Compressed Sparse Attention and Heavily Compressed Attention, plus a 128-token local sliding window: 128 query heads onto a single 512-dim compressed KV head with a 64-dim RoPE part. This is what makes the 1-million-token context tractable. (Rendered here as the MLA block; the real mechanism is CSA+HCA.)
384 routed experts, 6 active per token, plus 1 shared expert, each 3072 wide. The router scores with sqrt(softplus) affinity. V4 Pro activates 49B of its 1.6T parameters per token.
Per-head RMSNorm then a projection to 129,280 logits. V4 Pro targets agentic and long-document workloads that genuinely use the full 1M context.