LLM-jp / NII LLMCLLM-jp-3 MoE
LLM-jp-3 8×13B Instruct3
LLM-jp-3 8×13B Instruct3 has 73,162,828,800 total parameters and 22,200,806,400 active parameters. Its 40-layer architecture retains a 4,096-token context.
Research release
Preference tuning is a separate release
The Instruct3 card adds preference training with aya-ja-evol-inst and ac-self-inst. Those datasets address helpfulness and safety after supervised tuning.LLM-jp-3 8×13B Instruct3 official model card
Measure preference changes at fixed architecture
Practical guidance. For 8×13B, check helpful answers and refusal behavior separately. The presence of a safety dataset does not establish safe behavior for every new prompt.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-3-8x13b-instruct3 |
| Context window | 4,096 tokens Maximum sequence length stated in the model card. LLM-jp-3 8×13B Instruct3 official model card |
| Architecture | 40 layers; hidden size 5120; eight experts, two active.LLM-jp-3 8×13B Instruct3 official model card |
| License | Apache 2.0.LLM-jp-3 8×13B Instruct3 official model card |
| Pretraining | 2.1 trillion seen tokens reported in the model card.LLM-jp-3 8×13B Instruct3 official model card |
| Ancestry | Drop-Upcycling from LLM-jp-3 13B; expert reinitialization ratio 0.5.LLM-jp-3 MoE training and release |
Related models
- LLM-jp-3 8×13B Base
LLM-jp / NII LLMC · LLM-jp-3 MoE
LLM-jp-3 8×13B Base is an eight-expert language model with two experts selected per token.
Launch date not confirmedResearch releaseDownloadable