Pythia 14M
Pythia 14M is a base language model in a suite designed for controlled training research. This record describes the current standard-Pile weights.
Research release
Architecture and scale
The 128-wide network has about 1.19 million non-embedding parameters. Most of its nominal size comes from token embeddings. This makes the model useful for small-scale experiments, but the 14M label does not describe a 14-million-parameter Transformer stack.pythia-14m model cardpythia-14m model configuration
Weights and training revisions
This URL now contains a model trained on the standard Pile. Before the correction, it served the deduplicated model. Older experiments using the same unqualified name can therefore refer to different weights. The displaced model remains available under the deduped name.pythia-14m model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pythia-14m |
| Context window | 2,048 tokens Maximum position count in the released model configuration. pythia-14m model configuration |
| Nominal parameters | 14M including embeddingspythia-14m model card |
| Network | 6 layers; hidden width 128; 4 attention headspythia-14m model configuration |
| Training data | Standard Pilepythia-14m model card |
| License | Apache 2.0pythia-14m model card |
Related models
- Pythia 14M Deduped
EleutherAI ยท Pythia
A 14M Pythia checkpoint trained on deduplicated Pile text.
Launch date July 2023Research releaseDownloadable