Pythia 160M Deduped
Pythia 160M Deduped is the deduplicated-Pile counterpart of Pythia 160M. It preserves the architecture while changing the training data.
Research release
Deduplicated data at twelve layers
This model uses the same 12-layer, 768-wide architecture as standard Pythia 160M. The downloadable training steps support analysis of when a behavior emerges. Comparisons should use matching token exposure, not merely a familiar checkpoint label.pythia-160m-deduped model cardpythia-160m model configuration
Weights and training revisions
The release includes 154 saved training steps. The data order is shared with other sizes in the same suite. The main branch provides the final model; a step revision selects an earlier checkpoint. This base model has not been trained as a conversational assistant.pythia-160m-deduped model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pythia-160m-deduped |
| Context window | 2,048 tokens Maximum position count in the released model configuration. pythia-160m model configuration |
| Nominal parameters | 160M including embeddingspythia-160m-deduped model card |
| Network | 12 layers; hidden width 768; 12 attention headspythia-160m model configuration |
| Training data | Deduplicated Pilepythia-160m-deduped model card |
| License | Apache 2.0pythia-160m-deduped model card |
Related models
- Pythia 160M
EleutherAI ยท Pythia
A twelve-layer open model with checkpoints across its training run.
Launch date Research releaseDownloadable