IndicBERT v3 1B
IndicBERT v3 1B changes Gemma 3 into a bidirectional encoder. It uses masked next-token prediction rather than ordinary BERT masked-token prediction.
Restricted access
Masked next-token prediction shifts the target position
A masked token is predicted from the preceding hidden state. Data collation must preserve that shift. The card says dedicated sentence-embedding and retrieval versions are planned; it does not identify this checkpoint as an already optimized retrieval model.IndicBERT v3 1B official model card
Do not use a standard BERT training recipe unchanged
Practical guidance. Check token positions before downstream tuning. A correctly shaped tensor can still train the wrong objective if the masked target is aligned to the wrong hidden state.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Restricted access |
| Access | Downloadable |
| Model identifiers | IndicBERT-v3-1B |
| Context window | 4,096 tokens Sequence length used throughout training, as stated in the card. IndicBERT v3 1B official model card |
| Base declared in metadata | google/gemma-3-1b-itIndicBERT v3 1B official model card |
| Training | About 35 billion tokens; curriculum stages introduce English, higher-resource languages, lower-resource languages and joint training.IndicBERT v3 1B official model card |
| Terms | The card declares MIT. Gemma ancestry has separate upstream terms; do not treat the metadata as a replacement for them.IndicBERT v3 1B official model card |
Related models
- IndicBERT v2 Masked Language Model
AI4Bharat ยท IndicBERT
Multilingual IndicBERT v2 encoder trained with the masked-language-model objective.
Launch date not confirmedDownloadable