LLM-jp 13B v2.0 Base
LLM-jp 13B v2.0 Base was pretrained on 256 billion tokens. It uses a revised mixed Japanese, English and code vocabulary rather than the v1 tokenizer.
Research release
The token vocabulary changes between generations
The card describes separate code, English and Japanese vocabulary construction, followed by score re-estimation. Training used Megatron-LM on 128 A100 GPUs. Its Japanese corpus includes Common Crawl rather than the earlier mC4 mixture.LLM-jp 13B v2.0 Base official model card
Do not mix the v1 and v2 tokenizers
Practical guidance. This release changes more than instruction tuning. A fair comparison with v1 needs to account for different tokenization, context length and pretraining data.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-13b-v2.0 |
| Context window | 4,096 tokens Maximum sequence length stated in the model card. LLM-jp 13B v2.0 Base official model card |
| Architecture | 40 layers; hidden size 5,120; 40 attention heads.LLM-jp 13B v2.0 Base official model card |
| Vocabulary | 96,867 tokenizer entries; model vocabulary padded to 97,024.LLM-jp 13B v2.0 Base official model card |
| License | Apache 2.0.LLM-jp 13B v2.0 Base official model card |
Related models
- LLM-jp 13B v1.0 Base
LLM-jp / NII LLMC · LLM-jp v1
The original 13B LLM-jp text model, trained on 300 billion tokens with a 2,048-token context.
Launch date not confirmedResearch releaseDownloadable - LLM-jp 13B v2.0 Instruct
LLM-jp / NII LLMC · LLM-jp v2
LLM-jp 13B v2.0 Instruct is one of three published safety-data tuning comparisons.
Launch date not confirmedResearch releaseDownloadable