Pythia 12B Deduped
Pythia 12B Deduped is the deduplicated-Pile counterpart of Pythia 12B. It preserves the architecture while changing the training data.
Research release
The upper end of the deduplicated suite
Pythia 12B Deduped uses the largest Pythia architecture on deduplicated Pile data. It is an independently trained checkpoint, not a smaller file format of standard Pythia 12B. Its weight files and revision must be selected accordingly.pythia-12b-deduped model cardpythia-12b model configuration
Weights and training revisions
The release includes 154 saved training steps. The data order is shared with other sizes in the same suite. The main branch provides the final model; a step revision selects an earlier checkpoint. This base model has not been trained as a conversational assistant.pythia-12b-deduped model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pythia-12b-deduped |
| Context window | 2,048 tokens Maximum position count in the released model configuration. pythia-12b model configuration |
| Nominal parameters | 12B including embeddingspythia-12b-deduped model card |
| Network | 36 layers; hidden width 5,120; 40 attention headspythia-12b model configuration |
| Training data | Deduplicated Pilepythia-12b-deduped model card |
| License | Apache 2.0pythia-12b-deduped model card |
Related models
- Pythia 12B
EleutherAI ยท Pythia
The largest Pythia size, with 36 layers and open training checkpoints.
Launch date Research releaseDownloadable