LLM-jp-3.1 13B Instruct4
LLM-jp-3.1 13B Instruct4 starts from the corresponding LLM-jp-3 checkpoint. The family adds 400 billion mid-training tokens, with instruction-response data mixed into the pretraining corpus.
Research release
Instruct4 adds two stages of DPO
After supervised fine-tuning, the project generated multiple answers and scored them with Qwen2.5-32B-Instruct. It used those preferences for DPO in two stages. The second stage also adds the Japanese safety preference dataset ac-self-inst.LLM-jp-3.1 13B Instruct4 official model card
Compare alignment at the same size
Practical guidance. Use the base and Instruct4 pair to separate the effect of instruction pretraining from later response alignment. A helpfulness score alone will not describe refusal behavior.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-3.1-13b-instruct4 |
| Context window | 4,096 tokens Maximum sequence length stated in the model card. LLM-jp-3.1 13B Instruct4 official model card |
| Architecture | 40 layers; hidden size 5,120; 40 attention heads.LLM-jp-3.1 13B Instruct4 official model card |
| Size | 13,706,931,200 parameters.LLM-jp-3.1 13B Instruct4 official model card |
| License | Apache 2.0.LLM-jp-3.1 13B Instruct4 official model card |
| Pretraining total | 2.5 trillion tokens, including the 400-billion-token mid-training stage.LLM-jp-3.1 13B Instruct4 official model card |
Related models
- LLM-jp-3 13B Instruct3
LLM-jp / NII LLMC · LLM-jp-3
LLM-jp-3 13B Instruct3 is a preference-aligned instruction release with 4,096-token context and Apache 2.0 weights.
Launch date not confirmedResearch releaseDownloadable - LLM-jp-3.1 13B Base
LLM-jp / NII LLMC · LLM-jp-3.1
LLM-jp-3.1 13B Base adds instruction pretraining to LLM-jp-3.
Launch date not confirmedResearch releaseDownloadable