YaRN Llama 2 13B 128K
This YaRN research release extends Llama 2 13B for 128K-token sequences.
Research release
Training and model behavior
The work continues pretraining on a subset of PG19. Nous hosts a version adapted for Flash Attention 2. The 128K release uses 600 additional training steps and the YaRN position-scaling method.YaRN Llama 2 13B 128K model card
Working with this model
Practical guidance. Use the matching model code and position-scaling settings. Check answer quality at several document lengths; accepting a long input does not establish reliable retrieval throughout it.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | 2023 |
| Access | Downloadable |
| Model identifiers | NousResearch/Yarn-Llama-2-13b-128k |
| Context window | 131,072 tokens YaRN context target; requires the released position-scaling implementation. YaRN Llama 2 13B 128K model card |
| Base model | Meta Llama 2 13BYaRN Llama 2 13B 128K model card |
Related models
A Llama 2 model with continued training for longer context.
Launch date 2023Research releaseDownloadableA Llama 2 model with continued training for longer context.
Launch date 2023Research releaseDownloadableA Llama 2 model with continued training for longer context.
Launch date 2023Research releaseDownloadable