Finnish ModernBERT Base Short
Finnish ModernBERT Base Short is a multilingual masked-language encoder with 209 million parameters. It uses the family's 128,000-entry vocabulary and an 8,192-token configuration.
Training and configuration
Training covers Finnish, Swedish, English, code, Latin and Northern Sámi. The card reports 365.0 billion tokens. This Short architecture has more vocabulary entries than the corresponding longer-window model.Finnish ModernBERT Base Short: model card and weightsFinnish ModernBERT Base Short: released configuration
Checkpoint for continued training
finnish-modernbert-base-short-cpt is a downloadable checkpoint from before learning-rate decay. Its card reports 360.2 billion tokens and recommends it for continued pretraining. It is recorded here as a training-stage variant of this model, rather than another final release.Finnish ModernBERT Base Short: checkpoint for continued pretraining
Practical assessment
Practical guidance. Base Short may split text differently from Base because their vocabularies differ. Compare both token counts and task results. Do not attribute every change to the input-window length.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Access | Downloadable |
| Model identifiers | finnish-modernbert-base-short |
| Context window | 8,192 tokens max_position_embeddings in this release's configuration; theoretical extrapolation is not recorded as an established input limit. Finnish ModernBERT Base Short: released configuration |
| License | Apache 2.0.Finnish ModernBERT Base Short: model card and weights |
| Architecture | 22 layers, hidden width 768 and 12 attention heads.Finnish ModernBERT Base Short: released configuration |
| Vocabulary | 128,000 entries in the released configuration.Finnish ModernBERT Base Short: released configuration |
| Training volume | 365.0 billion tokens, as stated by this release's card.Finnish ModernBERT Base Short: model card and weights |
Related models
- Finnish ModernBERT Base
TurkuNLP · Finnish ModernBERT
Finnish ModernBERT Base provides 16,000-token encoder inputs with 143 million parameters.
Launch date not confirmedDownloadable - Finnish ModernBERT Base Short EDU
TurkuNLP · Finnish ModernBERT
Finnish ModernBERT Base Short EDU provides 8,192-token encoder inputs with 209 million parameters and educational-data annealing.
Launch date not confirmedDownloadable