LLM-jp 1.3B v1.0 Base
LLM-jp 1.3B v1.0 is an early Japanese-English pretrained model. It includes source-code data and uses the project's 50,570-entry mixed vocabulary.
Research release
The final stage added selected corpus data
The pretraining sequence used ten non-overlapping corpus folds, then another 27 billion tokens selected from the same data sources. Those sources include Japanese mC4, English Pile data, Wikipedia and The Stack.LLM-jp 1.3B v1.0 Base official model card
Preserve the original tokenizer in reproductions
Practical guidance. Use this compact historical checkpoint to reproduce early LLM-jp work. Newer tokenizer vocabularies and chat templates are not interchangeable with this release.
Megatron-DeepSpeed files and earlier checkpoint
The mdsfmt repository is the training-format distribution. The itr87430 checkpoint precedes the final 27-billion-token stage and represents about 270 billion seen tokens. It remains grouped here as a training checkpoint, not a new model family.LLM-jp 1.3B v1.0 Megatron-DeepSpeed checkpoints
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-1.3b-v1.0llm-jp-1.3b-v1.0-mdsfmtllm-jp-1.3b-v1.0-mdsfmt-itr87430 |
| Context window | 2,048 tokens Maximum sequence length stated in the model card. LLM-jp 1.3B v1.0 Base official model card |
| Architecture | 24 layers; hidden size 2048; 16 attention heads.LLM-jp 1.3B v1.0 Base official model card |
| License | Apache 2.0.LLM-jp 1.3B v1.0 Base official model card |
| Training | 300 billion tokens; Megatron-DeepSpeed on the mdx platform.LLM-jp 1.3B v1.0 Base official model card |
Related models
- LLM-jp-3 1.8B Base
LLM-jp / NII LLMC ยท LLM-jp-3
LLM-jp-3 1.8B Base is a pretrained text model with 4,096-token context and Apache 2.0 weights.
Launch date not confirmedResearch releaseDownloadable