Pile-T5 XL
Pile-T5 XL is an encoder-decoder model trained from scratch. Downloadable weights and intermediate checkpoints support research into text-to-text learning.
Research release
Encoder and decoder structure
XL retains 24 layers in both encoder and decoder but expands hidden width to 2,048. It uses the UMT5 implementation and a Llama tokenizer in Transformers; these choices do not make it a Llama-derived checkpoint.pile-t5-xl model configurationpile-t5-xl model card
Separate pretraining from task adaptation
The released base reconstructs masked spans. The project also links task fine-tunes for SuperGLUE, CodeXGLUE and instruction evaluation. A score obtained after those fine-tuning stages is not a zero-shot score for the base weights.pile-t5-xl model card
Training steps and formats
The repository includes 200 checkpoints spaced 10,000 training steps apart. T5x and Transformers files are distributions of this model. Task-specific fine-tunes are linked by the project and should be identified separately when reporting results.pile-t5-xl model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pile-t5-xl |
| Context window | 512 tokens Sequence length used during training; not a documented hard inference limit. pile-t5-xl model card |
| Parameters | 2,849,804,288pile-t5-xl model card |
| Training exposure | About 2 trillion tokens; 2 million stepspile-t5-xl model card |
| License | Apache 2.0pile-t5-xl model card |
Related models
- Pile-T5 Base
EleutherAI · Pile-T5
A compact encoder-decoder model trained on two trillion Pile tokens.
Launch date Research releaseDownloadable - Pile-T5 Large
EleutherAI · Pile-T5
A 783-million-parameter encoder-decoder model trained from scratch for text-to-text research.
Launch date Research releaseDownloadable - Pile-T5 XXL
EleutherAI · Pile-T5
An eleven-billion-parameter encoder-decoder model trained from scratch.
Launch date Research releaseDownloadable