GPT-Neo 125M
GPT-Neo 125M is the smallest downloadable GPT-Neo release. It predicts the next token and was trained on the Pile, rather than on a conversational instruction dataset.
Research release
Twelve layers and a narrow hidden state
Its 12 Transformer layers have a hidden width of 768 and 12 attention heads. Attention alternates between full-sequence and local operations. The local window covers 256 tokens; this is different from the 2,048-token model context.gpt-neo-125m model configuration
A small model for training experiments
The 300-billion-token training run is substantial despite the small parameter count. This release is a compact reference for fine-tuning and language-model experiments. Its training objective does not establish reliable instruction following.gpt-neo-125m model cardGPT-Neo release history and archived training code
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | EleutherAI/gpt-neo-125m |
| Context window | 2,048 tokens Maximum position count in the released model configuration. gpt-neo-125m model configuration |
| Parameters | 125 milliongpt-neo-125m model card |
| Training exposure | 300 billion tokens; 572,300 stepsgpt-neo-125m model card |
| License | MITgpt-neo-125m model card |
Related models
- GPT-Neo 1.3B
EleutherAI · GPT-Neo
An early open GPT-style model trained on 380 billion tokens.
Launch date Research releaseDownloadable - GPT-Neo 2.7B
EleutherAI · GPT-Neo
The largest original GPT-Neo checkpoint, with 32 Transformer layers.
Launch date Research releaseDownloadable