Pythia 70M Deduped
Pythia 70M Deduped is the deduplicated-Pile counterpart of Pythia 70M. It preserves the architecture while changing the training data.
Research release
A small matched deduplication experiment
The six-layer architecture matches standard Pythia 70M. The data is deduplicated before training, while the total training budget remains about 300 billion tokens. Repeated passes over a smaller corpus make training exposure different from the amount of unique text.pythia-70m-deduped model cardpythia-70m model configuration
Weights and training revisions
The release includes 154 saved training steps. The data order is shared with other sizes in the same suite. The main branch provides the final model; a step revision selects an earlier checkpoint. This base model has not been trained as a conversational assistant.pythia-70m-deduped model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pythia-70m-deduped |
| Context window | 2,048 tokens Maximum position count in the released model configuration. pythia-70m model configuration |
| Nominal parameters | 70M including embeddingspythia-70m-deduped model card |
| Network | 6 layers; hidden width 512; 8 attention headspythia-70m model configuration |
| Training data | Deduplicated Pilepythia-70m-deduped model card |
| License | Apache 2.0pythia-70m-deduped model card |
Related models
- Pythia 70M
EleutherAI ยท Pythia
The smallest size in the original eight-size Pythia research suite.
Launch date Research releaseDownloadable