Pythia 160M
Pythia 160M is a base language model in a suite designed for controlled training research. This record describes the current standard-Pile weights.
Research release
Architecture and scale
The network has 12 layers, a hidden width of 768 and about 85.06 million non-embedding parameters. It offers a deeper architecture than Pythia 70M. A parameter label alone does not capture that depth change.pythia-160m model cardpythia-160m model configuration
Weights and training revisions
The release includes 154 saved training steps. The data order is shared with other sizes in the same suite. The main branch provides the final model; a step revision selects an earlier checkpoint. This base model has not been trained as a conversational assistant.pythia-160m model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pythia-160m |
| Context window | 2,048 tokens Maximum position count in the released model configuration. pythia-160m model configuration |
| Nominal parameters | 160M including embeddingspythia-160m model card |
| Network | 12 layers; hidden width 768; 12 attention headspythia-160m model configuration |
| Training data | Standard Pilepythia-160m model card |
| License | Apache 2.0pythia-160m model card |
Related models
- Pythia 160M Deduped
EleutherAI ยท Pythia
A 160M Pythia checkpoint trained on deduplicated Pile text.
Launch date Research releaseDownloadable