Finnish ModernBERT Large Short
Finnish ModernBERT Large Short is a multilingual masked-language encoder with 475 million parameters. It uses the family's 128,000-entry vocabulary and an 8,192-token configuration.
Training and configuration
Training covers Finnish, Swedish, English, code, Latin and Northern Sámi. The card reports 362.2 billion tokens. This Short architecture has more vocabulary entries than the corresponding longer-window model.Finnish ModernBERT Large Short: parent architecture and trainingFinnish ModernBERT Large Short: released configuration
Checkpoint for continued training
finnish-modernbert-large-short-cpt is a downloadable checkpoint from before learning-rate decay. Its card reports 357.3 billion tokens and recommends it for continued pretraining. It is recorded here as a training-stage variant of this model, rather than another final release.Finnish ModernBERT Large Short: checkpoint for continued pretraining
Practical assessment
Practical guidance. Large Short is the parent of the separate MS MARCO sentence encoder. The pretrained encoder does not acquire that retrieval objective merely by pooling its token outputs.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Access | Downloadable |
| Model identifiers | finnish-modernbert-large-short |
| Context window | 8,192 tokens max_position_embeddings in this release's configuration; theoretical extrapolation is not recorded as an established input limit. Finnish ModernBERT Large Short: released configuration |
| License | Apache 2.0.Finnish ModernBERT Large Short: parent architecture and training |
| Architecture | 28 layers, hidden width 1,024 and 16 attention heads.Finnish ModernBERT Large Short: released configuration |
| Vocabulary | 128,000 entries in the released configuration.Finnish ModernBERT Large Short: released configuration |
| Training volume | 362.2 billion tokens, as stated by this release's card.Finnish ModernBERT Large Short: parent architecture and training |
Related models
- Finnish ModernBERT Large
TurkuNLP · Finnish ModernBERT
Finnish ModernBERT Large provides 16,000-token encoder inputs with 401 million parameters.
Launch date not confirmedDownloadable - Finnish ModernBERT Large MS MARCO
TurkuNLP · Finnish ModernBERT
This sentence encoder adapts TurkuNLP's Finnish ModernBERT Large Short for retrieval.
Launch date not confirmedResearch releaseDownloadable - Finnish ModernBERT Large Short EDU
TurkuNLP · Finnish ModernBERT
Finnish ModernBERT Large Short EDU provides 8,192-token encoder inputs with 475 million parameters and educational-data annealing.
Launch date not confirmedDownloadable