Pile-T5 Base
Pile-T5 Base is an encoder-decoder model trained from scratch. Downloadable weights and intermediate checkpoints support research into text-to-text learning.
Research release
Encoder and decoder structure
Base uses 12 encoder layers and 12 decoder layers. Its hidden width is 768. Unlike a decoder-only text model, it reads an input sequence through an encoder and generates a separate output sequence.pile-t5-base model configurationpile-t5-base model card
A small model with substantial training
The training objective reconstructs spans removed from input text. Two million training steps expose this model to about two trillion tokens. That objective makes it a base for task adaptation, not a chat assistant.pile-t5-base model card
Training steps and formats
The repository includes 200 checkpoints spaced 10,000 training steps apart. T5x and Transformers files are distributions of this model. Task-specific fine-tunes are linked by the project and should be identified separately when reporting results.pile-t5-base model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/pile-t5-base |
| Context window | 512 tokens Sequence length used during training; not a documented hard inference limit. pile-t5-base model card |
| Parameters | 247,586,304pile-t5-base model card |
| Training exposure | About 2 trillion tokens; 2 million stepspile-t5-base model card |
| License | Apache 2.0pile-t5-base model card |
Related models
- Pile-T5 Large
EleutherAI · Pile-T5
A 783-million-parameter encoder-decoder model trained from scratch for text-to-text research.
Launch date Research releaseDownloadable - Pile-T5 XL
EleutherAI · Pile-T5
A 2.85-billion-parameter text-to-text model with an open training history.
Launch date Research releaseDownloadable - Pile-T5 XXL
EleutherAI · Pile-T5
An eleven-billion-parameter encoder-decoder model trained from scratch.
Launch date Research releaseDownloadable