Pythia 12B
Pythia 12B is a base language model in a suite designed for controlled training research. This record describes the current standard-Pile weights.
Research release
Architecture and scale
The largest Pythia network has 36 layers, a hidden width of 5,120 and 40 attention heads. It extends both width and depth beyond the 6.9B model. Released intermediate checkpoints support research into behavior before the final training step.pythia-12b model cardpythia-12b model configuration
Weights and training revisions
The release includes 154 saved training steps. The data order is shared with other sizes in the same suite. The main branch provides the final model; a step revision selects an earlier checkpoint. This base model has not been trained as a conversational assistant.pythia-12b model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pythia-12b |
| Context window | 2,048 tokens Maximum position count in the released model configuration. pythia-12b model configuration |
| Nominal parameters | 12B including embeddingspythia-12b model card |
| Network | 36 layers; hidden width 5,120; 40 attention headspythia-12b model configuration |
| Training data | Standard Pilepythia-12b model card |
| License | Apache 2.0pythia-12b model card |
Related models
- Pythia 12B Deduped
EleutherAI ยท Pythia
A 12B Pythia checkpoint trained on deduplicated Pile text.
Launch date Research releaseDownloadable