LLM-jp ModernBERT Base
LLM-jp ModernBERT Base is a masked-language encoder trained on the 3.4 TB Japanese subset of llm-jp-corpus v4. It uses the project's tokenizer.
Research release
Longer context is a second training stage
The card describes 500,000 first-stage steps and 200,000 second-stage steps. Its examples load a masked-language model, and evaluation covers Japanese semantic similarity, inference and grammatical acceptability.LLM-jp ModernBERT Base official model card
Add the downstream task deliberately
Practical guidance. Use this model as an encoder foundation for further training. Do not assume that masked-language pretraining alone provides a ready-made semantic search or chat endpoint.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | llm-jp-modernbert-base |
| Context window | 8,192 tokens Maximum sequence length stated in the model card. LLM-jp ModernBERT Base official model card |
| Model type | ModernBERT-based encoder; not an autoregressive chat model.LLM-jp ModernBERT Base official model card |
| Training stages | 1,024-token sequences first, then 8,192-token sequences.LLM-jp ModernBERT Base official model card |
| License | Apache 2.0.LLM-jp ModernBERT Base official model card |
Related models
- LayoutLMv3 Japanese Preview
LLM-jp / NII LLMC ยท LLM-jp Document AI
A Japanese document encoder pretrained with page images, text and word positions.
Launch date not confirmedPreviewDownloadable