Inkling

6144-dim vectors from a 201,024-row embedding (padded; the output head keeps only the first 200,058 ids), RMS-normalized before the first layer. Inkling is natively multimodal, with images and audio projected into the same hidden space; this scene shows the text decoder only.

Five sliding-window layers for every global layer (55 + 11), with every sixth layer global. The first 2 layers use a dense FFN, the other 64 are MoE. Each layer adds four short convolutions (kernel 4): on the keys, on the values, and on the attention and MLP outputs before they rejoin the residual stream.

Sliding-window layers attend over 512 tokens with 64 query / 16 KV heads; global layers see the full context with 64 query / 8 KV heads (head_dim 128). Queries and keys are RMS-normalized, and instead of RoPE each head adds a learned relative-position bias mixed from 16 distance profiles, which the launch post says extrapolates better to long sequences.

A sigmoid router with an aux-loss-free balancing bias picks 6 of 256 routed experts (FFN 3072); 2 shared experts run on every token and are weighted jointly with the chosen ones. Only 41B of 975B parameters fire per token.

Final RMSNorm, a muP divide by 24, then an untied projection to 201,024 logits, sliced to the 200,058 real tokens.