Pythia 31M Deduped
Pythia 31M Deduped is the deduplicated-Pile counterpart of Pythia 31M. It preserves the architecture while changing the training data.
Research release
The same depth with a wider hidden state
This checkpoint uses deduplicated training data with six layers and a hidden width of 256. It is wider than Pythia 14M Deduped, but equally deep. That distinction matters when a study compares how scale affects learned representations.pythia-31m-deduped model cardpythia-31m model configuration
Weights and training revisions
These deduplicated weights previously occupied the unqualified model URL. The standard model card documents the later correction and move to this explicit deduped name. The July publication month is supported by a public weight-conversion discussion.pythia-31m model cardpythia-31m-deduped public weight conversion discussion
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | July 2023 |
| Access | Downloadable |
| Model identifiers | EleutherAI/pythia-31m-deduped |
| Context window | 2,048 tokens Maximum position count in the released model configuration. pythia-31m model configuration |
| Nominal parameters | 31M including embeddingspythia-31m-deduped model card |
| Network | 6 layers; hidden width 256; 8 attention headspythia-31m model configuration |
| Training data | Deduplicated Pilepythia-31m-deduped model card |
| License | Apache 2.0pythia-31m-deduped model card |
Related models
- Pythia 31M
EleutherAI ยท Pythia
A six-layer Pythia model that doubles the width of the 14M network.
Launch date Research releaseDownloadable