LLM-jp-3 8×13B Base
LLM-jp-3 8×13B Base has 73,162,828,800 total parameters and 22,200,806,400 active parameters. Its 40-layer architecture retains a 4,096-token context.
Research release
The expert count does not equal total size
The model combines shared transformer components with expert feed-forward layers. The card supplies both total and active parameter counts, so multiplying eight by the nominal dense size would give the wrong model size.LLM-jp-3 8×13B Base official model card
Plan storage and computation separately
Practical guidance. This 8×13B expert configuration can be compared with dense releases using the same corpus. Memory still needs to hold the complete expert model, even though routing uses only part of it per token.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-3-8x13b |
| Context window | 4,096 tokens Maximum sequence length stated in the model card. LLM-jp-3 8×13B Base official model card |
| Architecture | 40 layers; hidden size 5120; eight experts, two active.LLM-jp-3 8×13B Base official model card |
| License | Apache 2.0.LLM-jp-3 8×13B Base official model card |
| Pretraining | 2.1 trillion seen tokens reported in the model card.LLM-jp-3 8×13B Base official model card |
| Ancestry | Drop-Upcycling from LLM-jp-3 13B; expert reinitialization ratio 0.5.LLM-jp-3 MoE training and release |
Related models
- LLM-jp-3 8×13B Instruct3
LLM-jp / NII LLMC · LLM-jp-3 MoE
LLM-jp-3 8×13B Instruct3 is an eight-expert language model with two experts selected per token.
Launch date not confirmedResearch releaseDownloadable