Loading...
Loading...
Multimodal 552B MoE (8B active prefill / 16B decode) with 1M context and optional reasoning effort, the V4.1 refresh of the fast, high-throughput Flash line for coding and agent workflows
Take DeepSeek V4.1 Flash apart, layer by layer. The 3D teardown below shows its 40 transformer layers (0 dense + 40 MoE), MLA attention, and expert routing, with values from the public model config.
DeepSeek is a Chinese AI lab founded in July 2023 in Hangzhou by Liang Wenfeng. It was funded solely by High-Flyer, the quantitative hedge fund Liang co-founded, until a $7 billion Series A in May 2026, and grew out of GPU capacity originally stockpiled for trading research. The lab releases its V, R and V4 model lines as open weights under the MIT licence, with the V4 Pro and Flash tiers succeeded by V4.1 Flash in September 2026.
more about DeepSeek →5120-dim vectors over a 129,280-token vocabulary. Image tokens from a 32-layer DeepSeek-ViT encoder join the same stream (not drawn). Encoder blocks 2 and 15 also look up Engram conditional memory: 196B parameters of token n-gram tables (n-grams up to 4, 8 heads of 256 dims) read sparsely per token, outside the 552B backbone.
40 layers in two halves. The decoder's global KV cache is projected from the final encoder hidden states instead of each decoder layer's own states, which is why prefill activates only 8B parameters and decode 16B. Single-Pass mHC carries 4 parallel residual streams around every attention and MoE sublayer. The scene draws one 40-layer stack; the encoder / decoder split and the CED bridge are not shown.
64 query heads attend to a single 512-dim compressed KV head with a 64-dim RoPE part. The first two layers are SWA-only; the other 38 pair a 128-token sliding window with Compressed Sparse Attention 2, whose Full / Reindex / Reuse modes share main KV across layers (sources at layers 2, 8, 14, 20) and reuse top-k indices (32 indexer heads, top-k 512). Global KV is 890 bytes per token with FP4 caching (3.125 KiB at bf16), not counting the bounded sliding-window cache. (Rendered as the MLA block.)
384 routed experts plus 1 shared, 6 routed active per token, expert dim 2304, sqrt(softplus) routing. Of the 552B backbone, 8B is active during prefill and 16B during decode.
The 4 mHC residual streams are mixed back to one, then RMSNorm and a projection to 129,280 logits. Three MTP layers and DSpark speculative decoding (target layers 37-39 with 128 draft experts, 3 active) speed up generation; neither is drawn.