Kimi-Linear-48B-A3B-Base
Kimi Linear Base is a 48-billion-parameter mixture-of-experts checkpoint for attention research and further training. It activates about three billion parameters and supports a million-token context.
Recurrent memory with global attention
The architecture alternates three Kimi Delta Attention layers with one global Multi-Head Latent Attention layer. KDA extends Gated DeltaNet with finer-grained gating. This combination keeps recurrent state while retaining periodic access through global attention.Kimi Linear architecture and inference instructions
Why the attention kernel matters
The paper describes a specialized chunkwise algorithm for diagonal-plus-low-rank transitions. It releases the kernel, serving implementation and both training-stage checkpoints. The architecture result therefore includes implementation work, not just a change to the parameter count.Kimi Linear: original attention architecture report
Separate architecture research from chat quality
Practical guidance. Use the base checkpoint when you need to continue training or compare attention designs. Keep the training stage fixed in model comparisons: a base checkpoint and an instruction-tuned assistant answer different evaluation questions.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Launch date | October 2025 |
| Access | Downloadable |
| Model identifiers | moonshotai/Kimi-Linear-48B-A3B-Base |
| Context window | 1,048,576 tokens Published checkpoint context limit; input and generated tokens share this budget. Kimi-Linear-48B-A3B-Base model card |
| Parameters | 48 billion total; 3 billion active per token.Kimi-Linear-48B-A3B-Base model card |
| Weight license | MIT.Kimi-Linear-48B-A3B-Base model card |
| Training data volume | 5.7 trillion tokens.Kimi-Linear-48B-A3B-Base model card |
Related models
- Kimi-Linear-48B-A3B-Instruct
Moonshot AI · Kimi Linear
Instruction-tuned hybrid-attention model designed to reduce memory use in long conversations.
Launch date October 2025Downloadable - Moonlight-16B-A3B
Moonshot AI · Moonlight
Pretrained language model used to study Muon optimization at 16 billion total parameters.
Launch date February 2025Downloadable