LLM-jp-3 150M Base
LLM-jp-3 150M Base is the 150M pretrained text model in LLM-jp's generation-3 size series. The underlying model was exposed to 2.1 trillion pretraining tokens.
Research release
The size changes the architecture
At this size, embeddings account for about two-thirds of the published parameter total. Pretraining includes Japanese web and archive material, English Dolma sources and code from The Stack.LLM-jp-3 150M Base official model card
Use the size as an experimental control
Practical guidance. Use the smallest checkpoint for controlled training and inference studies. Its existence is not evidence that it can replace a larger conversational model.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-3-150m |
| Context window | 4,096 tokens Maximum sequence length stated in the model card. LLM-jp-3 150M Base official model card |
| Architecture | 12 layers; hidden size 512; 8 attention heads.LLM-jp-3 150M Base official model card |
| Parameters | 152,219,136 total, from the card's embedding and non-embedding counts.LLM-jp-3 150M Base official model card |
| License | Apache 2.0; research weights.LLM-jp-3 150M Base official model card |
Related models
- LLM-jp-3 150M Instruct3
LLM-jp / NII LLMC ยท LLM-jp-3
LLM-jp-3 150M Instruct3 is a preference-aligned instruction release with 4,096-token context and Apache 2.0 weights.
Launch date not confirmedResearch releaseDownloadable