LLM-jp-3.1 8×13B Instruct4
LLM-jp-3.1 8×13B Instruct4 starts from the corresponding LLM-jp-3 checkpoint. The family adds 400 billion mid-training tokens, with instruction-response data mixed into the pretraining corpus.
Research release
Instruct4 adds two stages of DPO
After supervised fine-tuning, the project generated multiple answers and scored them with Qwen2.5-32B-Instruct. It used those preferences for DPO in two stages. The second stage also adds the Japanese safety preference dataset ac-self-inst.LLM-jp-3.1 8×13B Instruct4 official model card
Compare alignment at the same size
Practical guidance. Keep the expert routing and serving configuration fixed when comparing Instruct3 and Instruct4. Their response differences cannot be attributed to parameter count alone.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-3.1-8x13b-instruct4 |
| Context window | 4,096 tokens Maximum sequence length stated in the model card. LLM-jp-3.1 8×13B Instruct4 official model card |
| Architecture | 40 layers; eight experts with two selected per token.LLM-jp-3.1 8×13B Instruct4 official model card |
| Size | 73,162,828,800 total; 22,200,806,400 active parameters.LLM-jp-3.1 8×13B Instruct4 official model card |
| License | Apache 2.0.LLM-jp-3.1 8×13B Instruct4 official model card |
| Pretraining total | 2.5 trillion tokens, including the 400-billion-token mid-training stage.LLM-jp-3.1 8×13B Instruct4 official model card |
Related models
- LLM-jp-3 8×13B Instruct3
LLM-jp / NII LLMC · LLM-jp-3 MoE
LLM-jp-3 8×13B Instruct3 is an eight-expert language model with two experts selected per token.
Launch date not confirmedResearch releaseDownloadable - LLM-jp-3.1 8×13B Base
LLM-jp / NII LLMC · LLM-jp-3.1
LLM-jp-3.1 8×13B Base adds instruction pretraining to LLM-jp-3.
Launch date not confirmedResearch releaseDownloadable