LLM-jp / NII LLMCLLM-jp-3
LLM-jp-3 440M Base
LLM-jp-3 440M Base is the 440M pretrained text model in LLM-jp's generation-3 size series. The underlying model was exposed to 2.1 trillion pretraining tokens.
Research release
The size changes the architecture
The 16-layer model increases hidden width to 1,024. Pretraining includes Japanese web and archive material, English Dolma sources and code from The Stack.LLM-jp-3 440M Base official model card
Use the size as an experimental control
Practical guidance. This size is useful when the 150M model is too narrow for an adaptation study. Compare task accuracy before increasing generation limits.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-3-440m |
| Context window | 4,096 tokens Maximum sequence length stated in the model card. LLM-jp-3 440M Base official model card |
| Architecture | 16 layers; hidden size 1,024; 8 attention heads.LLM-jp-3 440M Base official model card |
| Parameters | 447,052,800 total, from the card's embedding and non-embedding counts.LLM-jp-3 440M Base official model card |
| License | Apache 2.0; research weights.LLM-jp-3 440M Base official model card |
Related models
- LLM-jp-3 440M Instruct3
LLM-jp / NII LLMC ยท LLM-jp-3
LLM-jp-3 440M Instruct3 is a preference-aligned instruction release with 4,096-token context and Apache 2.0 weights.
Launch date not confirmedResearch releaseDownloadable