Pile-T5 XXL
Pile-T5 XXL is an encoder-decoder model trained from scratch. Downloadable weights and intermediate checkpoints support research into text-to-text learning.
Research release
Encoder and decoder structure
XXL uses 24 encoder and 24 decoder layers with a hidden width of 4,096. It is the largest Pile-T5 size. The architecture keeps an explicit encoder rather than turning every input and output into one decoder stream.pile-t5-xxl model configurationpile-t5-xxl model card
An English base for supervised tasks
The model was trained on the deduplicated Pile with span corruption. Its card describes English-language research and later task adaptation. It does not establish general multilingual performance or an instruction-following interface.pile-t5-xxl model card
Training steps and formats
The repository includes 200 checkpoints spaced 10,000 training steps apart. T5x and Transformers files are distributions of this model. Task-specific fine-tunes are linked by the project and should be identified separately when reporting results.pile-t5-xxl model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pile-t5-xxl |
| Context window | 512 tokens Sequence length used during training; not a documented hard inference limit. pile-t5-xxl model card |
| Parameters | 11,135,426,560pile-t5-xxl model card |
| Training exposure | About 2 trillion tokens; 2 million stepspile-t5-xxl model card |
| License | Apache 2.0pile-t5-xxl model card |
Related models
- Pile-T5 Base
EleutherAI · Pile-T5
A compact encoder-decoder model trained on two trillion Pile tokens.
Launch date Research releaseDownloadable - Pile-T5 Large
EleutherAI · Pile-T5
A 783-million-parameter encoder-decoder model trained from scratch for text-to-text research.
Launch date Research releaseDownloadable - Pile-T5 XL
EleutherAI · Pile-T5
A 2.85-billion-parameter text-to-text model with an open training history.
Launch date Research releaseDownloadable