Pile-T5 Large
Pile-T5 Large is an encoder-decoder model trained from scratch. Downloadable weights and intermediate checkpoints support research into text-to-text learning.
Research release
Encoder and decoder structure
Large uses 24 encoder layers and 24 decoder layers, each with a hidden width of 1,024. It is almost three times the parameter count of Base, but the published results do not show consistent gains.pile-t5-large model configurationpile-t5-large model card
The authors flag an unresolved training issue
The release report says Large underperformed on SuperGLUE and Flan Held In. The authors suspect a training bug. This caveat is specific to the Large run and should accompany comparisons with the other Pile-T5 sizes.Pile-T5 release, training and evaluation report
Training steps and formats
The repository includes 200 checkpoints spaced 10,000 training steps apart. T5x and Transformers files are distributions of this model. Task-specific fine-tunes are linked by the project and should be identified separately when reporting results.pile-t5-large model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pile-t5-large |
| Context window | 512 tokens Sequence length used during training; not a documented hard inference limit. pile-t5-large model card |
| Parameters | 783,173,632pile-t5-large model card |
| Training exposure | About 2 trillion tokens; 2 million stepspile-t5-large model card |
| License | Apache 2.0pile-t5-large model card |
Related models
- Pile-T5 Base
EleutherAI · Pile-T5
A compact encoder-decoder model trained on two trillion Pile tokens.
Launch date Research releaseDownloadable - Pile-T5 XL
EleutherAI · Pile-T5
A 2.85-billion-parameter text-to-text model with an open training history.
Launch date Research releaseDownloadable - Pile-T5 XXL
EleutherAI · Pile-T5
An eleven-billion-parameter encoder-decoder model trained from scratch.
Launch date Research releaseDownloadable