GPT-J 6B
GPT-J 6B was released by Ben Wang and Aran Komatsuzaki. It brought a six-billion-parameter English language model to downloadable Apache-licensed weights.
Research release
Rotary positions and a parallel Transformer
The network has 28 layers, a hidden width of 4,096 and 16 attention heads. Rotary position embeddings apply to 64 dimensions in each head. Its tokenizer uses the GPT-2 vocabulary, with the embedding table padded to 50,400 entries.gpt-j-6b model card
JAX training is part of its identity
The authors trained GPT-J with Mesh Transformer JAX. It is a separate checkpoint and implementation from GPT-Neo 2.7B and GPT-NeoX-20B. The model card documents next-token prediction on the Pile. It does not describe an instruction-tuned assistant or a multilingual translation system.gpt-j-6b model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/gpt-j-6b |
| Context window | 2,048 tokens Maximum position count in the released model configuration. gpt-j-6b model configuration |
| Parameters | 6,053,381,344gpt-j-6b model card |
| License | Apache 2.0gpt-j-6b model card |
Related models
- GPT-NeoX-20B
EleutherAI · GPT-NeoX
A 20-billion-parameter Pile model from EleutherAI distributed training research.
Launch date Research releaseDownloadable - GPT-Neo 2.7B
EleutherAI · GPT-Neo
The largest original GPT-Neo checkpoint, with 32 Transformer layers.
Launch date Research releaseDownloadable