Pythia 14M Deduped
Pythia 14M Deduped is the deduplicated-Pile counterpart of Pythia 14M. It preserves the architecture while changing the training data.
Research release
A compact control for data comparisons
The deduplicated model pairs a six-layer, 128-wide network with a filtered version of the Pile. At this scale, vocabulary embeddings account for much of the parameter count. Use the matched standard-data release to examine data effects without changing the architecture.pythia-14m-deduped model cardpythia-14m model configuration
Weights and training revisions
These deduplicated weights previously occupied the unqualified model URL. The standard model card documents the later correction and move to this explicit deduped name. The July publication month is supported by a public weight-conversion discussion.pythia-14m model cardpythia-14m-deduped public weight conversion discussion
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | July 2023 |
| Access | Downloadable |
| Model identifiers | EleutherAI/pythia-14m-deduped |
| Context window | 2,048 tokens Maximum position count in the released model configuration. pythia-14m model configuration |
| Nominal parameters | 14M including embeddingspythia-14m-deduped model card |
| Network | 6 layers; hidden width 128; 4 attention headspythia-14m model configuration |
| Training data | Deduplicated Pilepythia-14m-deduped model card |
| License | Apache 2.0pythia-14m-deduped model card |
Related models
- Pythia 14M
EleutherAI ยท Pythia
A tiny six-layer model for studying the effect of training data.
Launch date Research releaseDownloadable