Finnish ModernBERT Base
Finnish ModernBERT Base is a multilingual masked-language encoder with 143 million parameters. It uses a size-specific vocabulary and a 16,000-token configuration.
Training and configuration
Training covers Finnish, Swedish, English, code, Latin and Northern Sámi. The card reports 410.2 billion tokens. The card discusses 128K context as a theoretical extrapolation. The released configuration specifies 16,000 positions.Finnish ModernBERT Base: model card and weightsFinnish ModernBERT Base: released configuration
Checkpoint for continued training
finnish-modernbert-base-cpt is a downloadable checkpoint from before learning-rate decay. Its card reports 398.7 billion tokens and recommends it for continued pretraining. It is recorded here as a training-stage variant of this model, rather than another final release.Finnish ModernBERT Base: checkpoint for continued pretraining
Practical assessment
Practical guidance. Base adds depth while retaining a hidden width of 768. Assess it on the documents your application will process. The same vector width as another encoder does not make their representations interchangeable.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Access | Downloadable |
| Model identifiers | finnish-modernbert-base |
| Context window | 16,000 tokens max_position_embeddings in this release's configuration; theoretical extrapolation is not recorded as an established input limit. Finnish ModernBERT Base: released configuration |
| License | Apache 2.0.Finnish ModernBERT Base: model card and weights |
| Architecture | 22 layers, hidden width 768 and 12 attention heads.Finnish ModernBERT Base: released configuration |
| Vocabulary | 42,240 entries in the released configuration.Finnish ModernBERT Base: released configuration |
| Training volume | 410.2 billion tokens, as stated by this release's card.Finnish ModernBERT Base: model card and weights |
Related models
- Finnish ModernBERT Base EDU
TurkuNLP · Finnish ModernBERT
Finnish ModernBERT Base EDU provides 16,000-token encoder inputs with 143 million parameters and educational-data annealing.
Launch date not confirmedDownloadable - Finnish ModernBERT Base Short
TurkuNLP · Finnish ModernBERT
Finnish ModernBERT Base Short provides 8,192-token encoder inputs with 209 million parameters.
Launch date not confirmedDownloadable