Loading...
Loading...
Flagship 2.8T sparse hybrid MoE (largest open-source model ever): a 3:1 Kimi Delta Attention and Gated MLA stack with 1M context, built for long-horizon coding and agent workloads
Take Kimi K3 apart, layer by layer. The 3D teardown below shows its 93 transformer layers (1 dense + 92 MoE), MLA attention, and expert routing, with values from the public model config.
Founded in March 2023 and headquartered in Beijing, Moonshot AI develops the Kimi series of foundation models with backing from Alibaba and Tencent. Its Kimi Chat assistant, launched in October 2023, was one of China's first consumer long-context LLM products, and the K2 line (July 2025 onward) shipped 1T-parameter open-weight MoE models under a modified MIT licence. Kimi K3, released in July 2026, is a 2.8T-parameter open-weight multimodal model with a 1M-token context.
more about Moonshot AI →7168-dim vectors over the same 163,840-token vocabulary as Kimi K2. K3 is natively multimodal; this scene shows the text decoder only.
Three Kimi Delta Attention layers for every Gated MLA layer (69 KDA + 24 MLA). Block Attention Residuals group the stack into blocks of 12, letting each layer draw on a learned mix of earlier block outputs instead of one running residual.
KDA is linear attention with a fixed-size recurrent state and a short convolution, so most layers keep no growing KV cache. Every fourth layer, plus the final one, is Gated MLA (96 heads, 512-dim KV latent, output gate) and no positional encoding. Only those 24 layers cache per token, about 27 KiB at bf16 across a 1M-token context window.
A sigmoid router picks 16 of 896 experts per token. Routed experts work in a 3584-dim latent projected down from the 7168-dim stream and RMSNorm-ed on the way back up, while 2 shared experts run at full width. Only 104B of 2.8T parameters fire per token.
Final RMSNorm then an untied projection to 163,840 logits. The FFNs use SiTU, a bounded gate (a tanh-capped sigmoid) in place of SwiGLU.