Megatron FinBERT 1.3B
TurkuNLP describes this cased Finnish release as a 1.3-billion-parameter Megatron version of FinBERT. It is an encoder, not one of the group's GPT text-generation models.
Width is the main architectural distinction
The configuration uses megatron-bert with a feed-forward width of 8,192. The card gives document-classification results based on 10% of two datasets, but does not supply a complete pretraining account. Those narrow results are not a general ranking.Megatron FinBERT 1.3B: configurationMegatron FinBERT 1.3B: model card
Account for encoder memory
Practical guidance. Plan for the large hidden representations as well as the weight file. Use shorter batches when fine-tuning token labels, and compare against FinBERT Large before assuming that extra width improves the task.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Access | Downloadable |
| Model identifiers | megatron-bert-finnish-cased-1.3B |
| Context window | 512 tokens Position limit in the published configuration. Megatron FinBERT 1.3B: configuration |
| License | Apache 2.0.Megatron FinBERT 1.3B: model card |
| Architecture | 24 layers, hidden width 2,048 and 32 attention heads.Megatron FinBERT 1.3B: configuration |
| Vocabulary | 50,176 entries; BertWordPieceCase tokenizer type.Megatron FinBERT 1.3B: configuration |
Related models
- FinBERT Large Cased
TurkuNLP ยท FinBERT
FinBERT Large Cased increases encoder depth and width while using the same Finnish training data as FinBERT Base.
Launch date not confirmedDownloadable