GPT-Neo 2.7B
GPT-Neo 2.7B is the larger model in the initial GPT-Neo release. Its Pile training and text-completion objective preceded the later GPT-J and GPT-NeoX models.
Research release
A 32-layer local-and-global attention model
The network has 32 layers, a hidden width of 2,560 and 20 attention heads. Its local attention window spans 256 tokens. The released configuration supports 2,048 token positions.gpt-neo-2.7b model configuration
Keep the model and training software separate
The checkpoint remains available with an MIT license even though the original Mesh TensorFlow training repository is archived. It is a base model. A conversational application needs its own adaptation and output evaluation.gpt-neo-2.7b model cardGPT-Neo release history and archived training code
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/gpt-neo-2.7b |
| Context window | 2,048 tokens Maximum position count in the released model configuration. gpt-neo-2.7b model configuration |
| Parameters | 2.7 billiongpt-neo-2.7b model card |
| Training exposure | 420 billion tokens; 400,000 stepsgpt-neo-2.7b model card |
| License | MITgpt-neo-2.7b model card |
Related models
- GPT-Neo 1.3B
EleutherAI · GPT-Neo
An early open GPT-style model trained on 380 billion tokens.
Launch date Research releaseDownloadable - GPT-Neo 125M
EleutherAI · GPT-Neo
A compact English text model with local and global attention.
Launch date not confirmedResearch releaseDownloadable