YaRN Llama 2 7B 64K
This YaRN research release extends Llama 2 7B for 64K-token sequences.
Research release
Training and model behavior
The work continues pretraining on a subset of PG19. Nous hosts a version adapted for Flash Attention 2. The 64K release uses 400 additional training steps and the YaRN position-scaling method.YaRN Llama 2 7B 64K model card
Working with this model
Practical guidance. Use the matching model code and position-scaling settings. Check answer quality at several document lengths; accepting a long input does not establish reliable retrieval throughout it.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | 2023 |
| Access | Downloadable |
| Model identifiers | NousResearch/Yarn-Llama-2-7b-64k |
| Context window | 65,536 tokens YaRN context target; requires the released position-scaling implementation. YaRN Llama 2 7B 64K model card |
| Base model | Meta Llama 2 7BYaRN Llama 2 7B 64K model card |
Related models
A Llama 2 model with continued training for longer context.
Launch date 2023Research releaseDownloadableA Llama 2 model with continued training for longer context.
Launch date 2023Research releaseDownloadableA Llama 2 model with continued training for longer context.
Launch date 2023Research releaseDownloadable