Moonlight-16B-A3B
Moonlight-16B-A3B is the base checkpoint from Moonshot's Muon training research. It uses a mixture of experts and was trained on 5.7 trillion tokens.
An experiment in training efficiency
The research adapts Muon to larger language models by adding weight decay and controlling the scale of parameter updates. It compares compute-efficient training against AdamW. This is a result about training, not a promise that every Moonlight inference will be twice as fast.Muon is Scalable for LLM Training: original report
Architecture compatibility and research artifacts
The release uses the DeepSeek-V3 implementation interface. That code compatibility does not establish that it was initialized from DeepSeek weights. Moonshot also supplies distributed Muon code and a report about intermediate training checkpoints. Those training artifacts belong to the same experiment.Moonlight research code and checkpoint information
Use the base checkpoint for adaptation
Practical guidance. Choose the base release when you need to study pretraining or train a task-specific adaptation. For a conversation interface, compare the separate Instruct checkpoint before adding your own instruction training.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Launch date | February 2025 |
| Access | Downloadable |
| Model identifiers | moonshotai/Moonlight-16B-A3B |
| Context window | 8,192 tokens Published checkpoint context limit; input and generated tokens share this budget. Moonlight base checkpoint configuration |
| Parameters | 16 billion total; approximately 3 billion active per token.Moonlight-16B-A3B model card |
| Weight license | MIT.Moonlight-16B-A3B model card |
| Training stage | Pretrained base model; the instruction-tuned release is separate.Moonlight-16B-A3B model card |
Related models
- Kimi-VL-A3B-Instruct
Moonshot AI · Kimi VL
Vision-language model for images, documents and video, built from Moonlight and the MoonViT encoder.
Launch date April 2025Downloadable - Moonlight-16B-A3B-Instruct
Moonshot AI · Moonlight
Instruction-tuned Moonlight release for conversation and task prompts, with an 8,192-token context.
Launch date February 2025Downloadable