LLM-jp-3.1 13B Base
LLM-jp-3.1 13B Base starts from the corresponding LLM-jp-3 checkpoint. The family adds 400 billion mid-training tokens, with instruction-response data mixed into the pretraining corpus.
Research release
Instruction pretraining changes the base model
About 90 billion tokens of instruction-response data were prepared for the 400-billion-token mixed stage. This is additional pretraining, not the separate Instruct4 SFT and DPO release.LLM-jp-3.1 13B Base official model card
Keep the mid-trained checkpoint distinct
Practical guidance. Use this model to study instruction pretraining without also changing to an expert architecture. Keep the dataset and post-training method fixed when comparing generations.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-3.1-13b |
| Context window | 4,096 tokens Maximum sequence length stated in the model card. LLM-jp-3.1 13B Base official model card |
| Architecture | 40 layers; hidden size 5,120; 40 attention heads.LLM-jp-3.1 13B Base official model card |
| Size | 13,706,931,200 parameters.LLM-jp-3.1 13B Base official model card |
| License | Apache 2.0.LLM-jp-3.1 13B Base official model card |
| Pretraining total | 2.5 trillion tokens, including the 400-billion-token mid-training stage.LLM-jp-3.1 13B Base official model card |
Related models
- LLM-jp-3 13B Base
LLM-jp / NII LLMC · LLM-jp-3
LLM-jp-3 13B Base is a pretrained text model with 4,096-token context and Apache 2.0 weights.
Launch date not confirmedResearch releaseDownloadable - LLM-jp-3.1 13B Instruct4
LLM-jp / NII LLMC · LLM-jp-3.1
LLM-jp-3.1 13B Instruct4 adds instruction pretraining to LLM-jp-3 and SFT plus two-stage preference alignment.
Launch date not confirmedResearch releaseDownloadable