Pythia 70M
Pythia 70M is a base language model in a suite designed for controlled training research. This record describes the current standard-Pile weights.
Research release
Architecture and scale
Pythia 70M uses six layers with a hidden width of 512. The model has about 18.92 million non-embedding parameters. It is much smaller than the 160M network, which also doubles the layer count.pythia-70m model cardpythia-70m model configuration
Weights and training revisions
The release includes 154 saved training steps. The data order is shared with other sizes in the same suite. The main branch provides the final model; a step revision selects an earlier checkpoint. This base model has not been trained as a conversational assistant.pythia-70m model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pythia-70m |
| Context window | 2,048 tokens Maximum position count in the released model configuration. pythia-70m model configuration |
| Nominal parameters | 70M including embeddingspythia-70m model card |
| Network | 6 layers; hidden width 512; 8 attention headspythia-70m model configuration |
| Training data | Standard Pilepythia-70m model card |
| License | Apache 2.0pythia-70m model card |
Related models
- Pythia 70M Deduped
EleutherAI ยท Pythia
A 70M Pythia checkpoint trained on deduplicated Pile text.
Launch date Research releaseDownloadable