Finnish ModernBERT Large 1024 Annealed
This research checkpoint was annealed before the model's context-extension stage. TurkuNLP recommends it for reproduction and research. Its actual configuration has a 1,024-token position limit.
Research release
Training and configuration
The card identifies this as the last checkpoint before context extension, followed by annealing. Its generic overview table also displays longer context figures. The explicit configuration sets max_position_embeddings to 1,024; that is the limit recorded here.Finnish ModernBERT Large 1024 Annealed: model card and weightsFinnish ModernBERT Large 1024 Annealed: released configuration
Reproduce the short-context experiment
Practical guidance. Use the 1,024-token configuration for reproduction. The card's shared table includes longer-window figures, so copying that table into a deployment limit would misstate this checkpoint.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | finnish-modernbert-large-seq-len-1024-117300-annealed |
| Context window | 1,024 tokens max_position_embeddings in this release's configuration; theoretical extrapolation is not recorded as an established input limit. Finnish ModernBERT Large 1024 Annealed: released configuration |
| License | Apache 2.0.Finnish ModernBERT Large 1024 Annealed: model card and weights |
| Architecture | 28 layers, hidden width 1,024 and 16 attention heads.Finnish ModernBERT Large 1024 Annealed: released configuration |
| Vocabulary | 55,616 entries in the released configuration.Finnish ModernBERT Large 1024 Annealed: released configuration |
| Training volume | 367.5 billion tokens, as stated by this release's card.Finnish ModernBERT Large 1024 Annealed: model card and weights |
Related models
- Finnish ModernBERT Large
TurkuNLP ยท Finnish ModernBERT
Finnish ModernBERT Large provides 16,000-token encoder inputs with 401 million parameters.
Launch date not confirmedDownloadable