GPT-NeoX-20B
GPT-NeoX-20B is a released model, while GPT-NeoX is also the name of the software used to train it. This checkpoint contains more than 20 billion parameters and has downloadable Apache-licensed weights.
Research release
A 44-layer model with rotary positions
The released model uses 44 layers, a hidden width of 6,144 and 64 attention heads. Its architecture uses rotary position embeddings. The configuration limits the original checkpoint to 2,048 token positions.gpt-neox-20b model cardgpt-neox-20b model configuration
Announcement and weight availability differed
The announcement preceded the public weight release by one week. The authors gave donors early access, then made the model public. CoreWeave supplied the training infrastructure. This explains why the announcement and download dates differ.GPT-NeoX-20B announcement and weight release
A base checkpoint for further adaptation
The training objective was next-token prediction on the Pile. The model card treats the checkpoint as a research resource that can be fine-tuned. It does not provide the instruction following or output controls of a finished chat application.gpt-neox-20b model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/gpt-neox-20b |
| Context window | 2,048 tokens Maximum position count in the released model configuration. gpt-neox-20b model configuration |
| Parameters | 20,554,567,680gpt-neox-20b model card |
| License | Apache 2.0gpt-neox-20b model card |
Related models
- Pythia 12B
EleutherAI · Pythia
The largest Pythia size, with 36 layers and open training checkpoints.
Launch date Research releaseDownloadable - GPT-J 6B
EleutherAI · GPT-J
A six-billion-parameter text model built with Mesh Transformer JAX.
Launch date Research releaseDownloadable