LLM-jp-3.1 8×13B Base
LLM-jp-3.1 8×13B Base starts from the corresponding LLM-jp-3 checkpoint. The family adds 400 billion mid-training tokens, with instruction-response data mixed into the pretraining corpus.
Research release
Instruction pretraining changes the base model
About 90 billion tokens of instruction-response data were prepared for the 400-billion-token mixed stage. This is additional pretraining, not the separate Instruct4 SFT and DPO release.LLM-jp-3.1 8×13B Base official model card
Keep the mid-trained checkpoint distinct
Practical guidance. The 8×13B label is an expert configuration, not a 104B total parameter count. Use the published total and active counts for deployment planning.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-3.1-8x13b |
| Context window | 4,096 tokens Maximum sequence length stated in the model card. LLM-jp-3.1 8×13B Base official model card |
| Architecture | 40 layers; eight experts with two selected per token.LLM-jp-3.1 8×13B Base official model card |
| Size | 73,162,828,800 total; 22,200,806,400 active parameters.LLM-jp-3.1 8×13B Base official model card |
| License | Apache 2.0.LLM-jp-3.1 8×13B Base official model card |
| Pretraining total | 2.5 trillion tokens, including the 400-billion-token mid-training stage.LLM-jp-3.1 8×13B Base official model card |
Related models
- LLM-jp-3 8×13B Base
LLM-jp / NII LLMC · LLM-jp-3 MoE
LLM-jp-3 8×13B Base is an eight-expert language model with two experts selected per token.
Launch date not confirmedResearch releaseDownloadable - LLM-jp-3.1 8×13B Instruct4
LLM-jp / NII LLMC · LLM-jp-3.1
LLM-jp-3.1 8×13B Instruct4 adds instruction pretraining to LLM-jp-3 and SFT plus two-stage preference alignment.
Launch date not confirmedResearch releaseDownloadable