Finnish ModernBERT Tiny
Finnish ModernBERT Tiny is a multilingual masked-language encoder with 51 million parameters. It uses a size-specific vocabulary and a 16,000-token configuration.
Training and configuration
Training covers Finnish, Swedish, English, code, Latin and Northern Sámi. The card reports 447.9 billion tokens. The card discusses 128K context as a theoretical extrapolation. The released configuration specifies 16,000 positions.Finnish ModernBERT Tiny: model card and weightsFinnish ModernBERT Tiny: released configuration
Checkpoint for continued training
finnish-modernbert-tiny-cpt is a downloadable checkpoint from before learning-rate decay. Its card reports 429.9 billion tokens and recommends it for continued pretraining. It is recorded here as a training-stage variant of this model, rather than another final release.Finnish ModernBERT Tiny: checkpoint for continued pretraining
Practical assessment
Practical guidance. Tiny has the fewest layers in this group. It is a useful starting point when model memory matters. Compare its task errors with Base before increasing capacity, and use the tokenizer released with this exact size.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Access | Downloadable |
| Model identifiers | finnish-modernbert-tiny |
| Context window | 16,000 tokens max_position_embeddings in this release's configuration; theoretical extrapolation is not recorded as an established input limit. Finnish ModernBERT Tiny: released configuration |
| License | Apache 2.0.Finnish ModernBERT Tiny: model card and weights |
| Architecture | 6 layers, hidden width 768 and 12 attention heads.Finnish ModernBERT Tiny: released configuration |
| Vocabulary | 27,264 entries in the released configuration.Finnish ModernBERT Tiny: released configuration |
| Training volume | 447.9 billion tokens, as stated by this release's card.Finnish ModernBERT Tiny: model card and weights |
Related models
- Finnish ModernBERT Tiny EDU
TurkuNLP · Finnish ModernBERT
Finnish ModernBERT Tiny EDU provides 16,000-token encoder inputs with 51 million parameters and educational-data annealing.
Launch date not confirmedDownloadable - Finnish ModernBERT Tiny Short
TurkuNLP · Finnish ModernBERT
Finnish ModernBERT Tiny Short provides 8,192-token encoder inputs with 129 million parameters.
Launch date not confirmedDownloadable