Pythia 6.9B Deduped
Pythia 6.9B Deduped is the deduplicated-Pile counterpart of Pythia 6.9B. It preserves the architecture while changing the training data.
Research release
A large matched data comparison
This model pairs the 6.9B architecture with deduplicated Pile training. Its direct control is standard Pythia 6.9B. Neither checkpoint is a chat fine-tune, so instruction following should not be used to infer a difference in the deduplication method.pythia-6.9b-deduped model cardpythia-6.9b model configuration
Weights and training revisions
The release includes 154 saved training steps. The data order is shared with other sizes in the same suite. The main branch provides the final model; a step revision selects an earlier checkpoint. This base model has not been trained as a conversational assistant.pythia-6.9b-deduped model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pythia-6.9b-deduped |
| Context window | 2,048 tokens Maximum position count in the released model configuration. pythia-6.9b model configuration |
| Nominal parameters | 6.9B including embeddingspythia-6.9b-deduped model card |
| Network | 32 layers; hidden width 4,096; 32 attention headspythia-6.9b model configuration |
| Training data | Deduplicated Pilepythia-6.9b-deduped model card |
| License | Apache 2.0pythia-6.9b-deduped model card |
Related models
- Pythia 6.9B
EleutherAI ยท Pythia
A 32-layer open model for research at a six-billion-parameter scale.
Launch date Research releaseDownloadable