LLM-jp / NII LLMCLLM-jp-3 MoE
LLM-jp-3 8×1.8B Instruct3
LLM-jp-3 8×1.8B Instruct3 has 9,266,362,368 total parameters and 2,924,279,808 active parameters. Its 24-layer architecture retains a 4,096-token context.
Research release
Preference tuning is a separate release
The Instruct3 card adds preference training with aya-ja-evol-inst and ac-self-inst. Those datasets address helpfulness and safety after supervised tuning.LLM-jp-3 8×1.8B Instruct3 official model card
Measure preference changes at fixed architecture
Practical guidance. For 8×1.8B, check helpful answers and refusal behavior separately. The presence of a safety dataset does not establish safe behavior for every new prompt.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-3-8x1.8b-instruct3 |
| Context window | 4,096 tokens Maximum sequence length stated in the model card. LLM-jp-3 8×1.8B Instruct3 official model card |
| Architecture | 24 layers; hidden size 2048; eight experts, two active.LLM-jp-3 8×1.8B Instruct3 official model card |
| License | Apache 2.0.LLM-jp-3 8×1.8B Instruct3 official model card |
| Pretraining | 2.1 trillion seen tokens reported in the model card.LLM-jp-3 8×1.8B Instruct3 official model card |
| Ancestry | Drop-Upcycling from LLM-jp-3 1.8B; expert reinitialization ratio 0.5.LLM-jp-3 MoE training and release |
Related models
- LLM-jp-3 8×1.8B Base
LLM-jp / NII LLMC · LLM-jp-3 MoE
LLM-jp-3 8×1.8B Base is an eight-expert language model with two experts selected per token.
Launch date not confirmedResearch releaseDownloadable