Bilingual GPT-NeoX 4B 8K
This release extends the 3.8B bilingual model's sequence length from 2,048 to 8,192 tokens through additional training.
Research release
Long-context training changes the checkpoint
rinna uses RoPE positional interpolation and 1.5B extra tokens from sequences longer than 4,000 tokens. This is a trained long-context variant, not only a changed generation setting.Bilingual GPT-NeoX 4B 8K official model card
Place relevant evidence at different positions
Practical guidance. Compare extraction near the start, middle and end of a long document. Accepting an 8K input does not by itself show that the model uses each part equally well.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | bilingual-gpt-neox-4b-8k |
| Context window | 8,192 tokens Maximum sequence length stated in the model card. Bilingual GPT-NeoX 4B 8K official model card |
| Scale | 3.8B language-model parameters; the 4B name is rounded.Bilingual GPT-NeoX 4B 8K official model card |
| License | MIT for this published release.Bilingual GPT-NeoX 4B 8K official model card |
Related models
- Bilingual GPT-NeoX 4B
rinna · Bilingual GPT-NeoX
An English-Japanese GPT-NeoX language model with 3.8B parameters.
Launch date not confirmedResearch releaseDownloadable - Bilingual GPT-NeoX 4B SFT
rinna · Bilingual GPT-NeoX
The supervised instruction release adapts Bilingual GPT-NeoX 4B using English and Japanese instruction data.
Launch date not confirmedDownloadable