LLM-jp-3 1.8B Base
LLM-jp-3 1.8B Base is the 1.8B pretrained text model in LLM-jp's generation-3 size series. The underlying model was exposed to 2.1 trillion pretraining tokens.
Research release
The size changes the architecture
The 24-layer architecture has hidden width 2,048 and 16 attention heads. Pretraining includes Japanese web and archive material, English Dolma sources and code from The Stack.LLM-jp-3 1.8B Base official model card
Use the size as an experimental control
Practical guidance. This size offers a useful boundary between the sub-billion models and the larger dense releases. Keep corpus and tuning choices fixed in scaling comparisons.
Sparse autoencoders support separate research
The project also publishes TopK sparse autoencoders trained on LLM-jp-3 1.8B. The named checkpoint series uses the prefix llm-jp-3-1.8b-sae-l12-k32-16x and suffixes c10, c100, c1000, c10000, c100000 and c988240. These are interpretability components, not replacement text-generation checkpoints.LLM-jp-3 1.8B sparse autoencoder component
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-3-1.8b |
| Context window | 4,096 tokens Maximum sequence length stated in the model card. LLM-jp-3 1.8B Base official model card |
| Architecture | 24 layers; hidden size 2,048; 16 attention heads.LLM-jp-3 1.8B Base official model card |
| Parameters | 1,867,614,208 total, from the card's embedding and non-embedding counts.LLM-jp-3 1.8B Base official model card |
| License | Apache 2.0; research weights.LLM-jp-3 1.8B Base official model card |
Related models
- LLM-jp-3 1.8B Instruct3
LLM-jp / NII LLMC ยท LLM-jp-3
LLM-jp-3 1.8B Instruct3 is a preference-aligned instruction release with 4,096-token context and Apache 2.0 weights.
Launch date not confirmedResearch releaseDownloadable