Finnish ModernBERT Large
Finnish ModernBERT Large is a multilingual masked-language encoder with 401 million parameters. It uses a size-specific vocabulary and a 16,000-token configuration.
Training and configuration
Training covers Finnish, Swedish, English, code, Latin and Northern Sámi. The card reports 399.2 billion tokens. The card discusses 128K context as a theoretical extrapolation. The released configuration specifies 16,000 positions.Finnish ModernBERT Large: model card and weightsFinnish ModernBERT Large: released configuration
Checkpoint for continued training
finnish-modernbert-large-cpt is a downloadable checkpoint from before learning-rate decay. Its card reports 388.1 billion tokens and recommends it for continued pretraining. It is recorded here as a training-stage variant of this model, rather than another final release.Finnish ModernBERT Large: checkpoint for continued pretraining
Practical assessment
Practical guidance. Large uses wider token representations as well as more layers. Check memory on full documents, not only short sentences. A theoretical extension in context length does not establish reliable retrieval at that length.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Access | Downloadable |
| Model identifiers | finnish-modernbert-large |
| Context window | 16,000 tokens max_position_embeddings in this release's configuration; theoretical extrapolation is not recorded as an established input limit. Finnish ModernBERT Large: released configuration |
| License | Apache 2.0.Finnish ModernBERT Large: model card and weights |
| Architecture | 28 layers, hidden width 1,024 and 16 attention heads.Finnish ModernBERT Large: released configuration |
| Vocabulary | 55,616 entries in the released configuration.Finnish ModernBERT Large: released configuration |
| Training volume | 399.2 billion tokens, as stated by this release's card.Finnish ModernBERT Large: model card and weights |
Related models
- Finnish ModernBERT Large EDU
TurkuNLP · Finnish ModernBERT
Finnish ModernBERT Large EDU provides 16,000-token encoder inputs with 401 million parameters and educational-data annealing.
Launch date not confirmedDownloadable - Finnish ModernBERT Large Short
TurkuNLP · Finnish ModernBERT
Finnish ModernBERT Large Short provides 8,192-token encoder inputs with 475 million parameters.
Launch date not confirmedDownloadable