GPT-Neo 1.3B
GPT-Neo 1.3B is one of the two models in the initial public GPT-Neo weight release. It is an English text-completion model with downloadable MIT-licensed weights.
Research release
More depth than the 125M model
The configuration specifies 24 layers, a hidden width of 2,048 and 16 attention heads. As in the smaller model, global attention alternates with local attention. The 256-token local window does not replace the full context limit.gpt-neo-1.3b model configuration
A release with a different training budget
This model saw 380 billion training tokens, compared with 420 billion for GPT-Neo 2.7B. A comparison between them therefore changes both model size and training exposure. The original training repository is archived, but the released weights remain downloadable.gpt-neo-1.3b model cardGPT-Neo release history and archived training code
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/gpt-neo-1.3b |
| Context window | 2,048 tokens Maximum position count in the released model configuration. gpt-neo-1.3b model configuration |
| Parameters | 1.3 billiongpt-neo-1.3b model card |
| Training exposure | 380 billion tokens; 362,000 stepsgpt-neo-1.3b model card |
| License | MITgpt-neo-1.3b model card |
Related models
- GPT-Neo 2.7B
EleutherAI · GPT-Neo
The largest original GPT-Neo checkpoint, with 32 Transformer layers.
Launch date Research releaseDownloadable - GPT-Neo 125M
EleutherAI · GPT-Neo
A compact English text model with local and global attention.
Launch date not confirmedResearch releaseDownloadable