Pythia 410M
Pythia 410M is a base language model in a suite designed for controlled training research. This record describes the current standard-Pile weights.
Research release
Architecture and scale
This size has 24 layers and about 302.31 million non-embedding parameters. Its depth matches Pythia 1.4B, but its hidden state is half as wide. Those models therefore offer a useful comparison of width at fixed depth.pythia-410m model cardpythia-410m model configuration
Weights and training revisions
The release includes 154 saved training steps. The data order is shared with other sizes in the same suite. The main branch provides the final model; a step revision selects an earlier checkpoint. This base model has not been trained as a conversational assistant.pythia-410m model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pythia-410m |
| Context window | 2,048 tokens Maximum position count in the released model configuration. pythia-410m model configuration |
| Nominal parameters | 410M including embeddingspythia-410m model card |
| Network | 24 layers; hidden width 1,024; 16 attention headspythia-410m model configuration |
| Training data | Standard Pilepythia-410m model card |
| License | Apache 2.0pythia-410m model card |
Related models
- Pythia 410M Deduped
EleutherAI ยท Pythia
A 410M Pythia checkpoint trained on deduplicated Pile text.
Launch date Research releaseDownloadable