LLM-jp-3 8×1.8B Base
LLM-jp-3 8×1.8B Base has 9,266,362,368 total parameters and 2,924,279,808 active parameters. Its 24-layer architecture retains a 4,096-token context.
Research release
The expert count does not equal total size
The model combines shared transformer components with expert feed-forward layers. The card supplies both total and active parameter counts, so multiplying eight by the nominal dense size would give the wrong model size.LLM-jp-3 8×1.8B Base official model card
Plan storage and computation separately
Practical guidance. This 8×1.8B expert configuration can be compared with dense releases using the same corpus. Memory still needs to hold the complete expert model, even though routing uses only part of it per token.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-3-8x1.8b |
| Context window | 4,096 tokens Maximum sequence length stated in the model card. LLM-jp-3 8×1.8B Base official model card |
| Architecture | 24 layers; hidden size 2048; eight experts, two active.LLM-jp-3 8×1.8B Base official model card |
| License | Apache 2.0.LLM-jp-3 8×1.8B Base official model card |
| Pretraining | 2.1 trillion seen tokens reported in the model card.LLM-jp-3 8×1.8B Base official model card |
| Ancestry | Drop-Upcycling from LLM-jp-3 1.8B; expert reinitialization ratio 0.5.LLM-jp-3 MoE training and release |
Related models
- LLM-jp-3 8×1.8B Instruct3
LLM-jp / NII LLMC · LLM-jp-3 MoE
LLM-jp-3 8×1.8B Instruct3 is an eight-expert language model with two experts selected per token.
Launch date not confirmedResearch releaseDownloadable