Pythia 6.9B
Pythia 6.9B is a base language model in a suite designed for controlled training research. This record describes the current standard-Pile weights.
Research release
Architecture and scale
Pythia 6.9B has 32 layers, a hidden width of 4,096 and 32 attention heads. Its non-embedding parameter count is about 6.44 billion. It shares its layer count with Pythia 2.8B but uses a wider hidden state.pythia-6.9b model cardpythia-6.9b model configuration
Weights and training revisions
The release includes 154 saved training steps. The data order is shared with other sizes in the same suite. The main branch provides the final model; a step revision selects an earlier checkpoint. This base model has not been trained as a conversational assistant.pythia-6.9b model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pythia-6.9b |
| Context window | 2,048 tokens Maximum position count in the released model configuration. pythia-6.9b model configuration |
| Nominal parameters | 6.9B including embeddingspythia-6.9b model card |
| Network | 32 layers; hidden width 4,096; 32 attention headspythia-6.9b model configuration |
| Training data | Standard Pilepythia-6.9b model card |
| License | Apache 2.0pythia-6.9b model card |
Related models
- Pythia 6.9B Deduped
EleutherAI ยท Pythia
A 6.9B Pythia checkpoint trained on deduplicated Pile text.
Launch date Research releaseDownloadable