Pythia 31M
Pythia 31M is a base language model in a suite designed for controlled training research. This record describes the current standard-Pile weights.
Research release
Architecture and scale
This model keeps the six-layer depth of Pythia 14M but increases the hidden width to 256. The non-embedding network has about 4.74 million parameters. Its architecture offers a width comparison without also doubling the number of layers.pythia-31m model cardpythia-31m model configuration
Weights and training revisions
This URL now contains a model trained on the standard Pile. Before the correction, it served the deduplicated model. Older experiments using the same unqualified name can therefore refer to different weights. The displaced model remains available under the deduped name.pythia-31m model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pythia-31m |
| Context window | 2,048 tokens Maximum position count in the released model configuration. pythia-31m model configuration |
| Nominal parameters | 31M including embeddingspythia-31m model card |
| Network | 6 layers; hidden width 256; 8 attention headspythia-31m model configuration |
| Training data | Standard Pilepythia-31m model card |
| License | Apache 2.0pythia-31m model card |
Related models
- Pythia 31M Deduped
EleutherAI ยท Pythia
A 31M Pythia checkpoint trained on deduplicated Pile text.
Launch date July 2023Research releaseDownloadable