LLM-jp 13B v1.0 Base
LLM-jp 13B v1.0 is an early Japanese-English pretrained model. It includes source-code data and uses the project's 50,570-entry mixed vocabulary.
Research release
The final stage added selected corpus data
The pretraining sequence used ten non-overlapping corpus folds, then another 27 billion tokens selected from the same data sources. Those sources include Japanese mC4, English Pile data, Wikipedia and The Stack.LLM-jp 13B v1.0 Base official model card
Preserve the original tokenizer in reproductions
Practical guidance. Use this checkpoint as the base for the v1 instruction models. A modern 13B checkpoint would not reproduce the training ancestry of those releases.
Megatron-DeepSpeed files and earlier checkpoint
The mdsfmt repository is the training-format distribution. The itr87870 checkpoint precedes the final 27-billion-token stage and represents about 270 billion seen tokens. It remains grouped here as a training checkpoint, not a new model family.LLM-jp 13B v1.0 Megatron-DeepSpeed checkpoints
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-13b-v1.0llm-jp-13b-v1.0-mdsfmtllm-jp-13b-v1.0-mdsfmt-itr87870 |
| Context window | 2,048 tokens Maximum sequence length stated in the model card. LLM-jp 13B v1.0 Base official model card |
| Architecture | 40 layers; hidden size 5120; 40 attention heads.LLM-jp 13B v1.0 Base official model card |
| License | Apache 2.0.LLM-jp 13B v1.0 Base official model card |
| Training | 300 billion tokens; Megatron-DeepSpeed on the mdx platform.LLM-jp 13B v1.0 Base official model card |
Related models
- LLM-jp 13B v1.0 Jaster Full
LLM-jp / NII LLMC · LLM-jp v1
LLM-jp 13B v1.0 Jaster Full is a distinct instruction-tuning experiment from the first LLM-jp release.
Launch date not confirmedResearch releaseDownloadable - LLM-jp 13B v2.0 Base
LLM-jp / NII LLMC · LLM-jp v2
The 13B v2.0 base changes the pretraining corpus and vocabulary and extends context to 4,096 tokens.
Launch date not confirmedResearch releaseDownloadable