rinnaJapanese GPT-NeoX 3.6B
Japanese GPT-NeoX 3.6B PPO
This model adds PPO reinforcement learning after Japanese GPT-NeoX 3.6B SFT v2. The alignment data uses a Japanese translation of an Anthropic HH-RLHF subset.
Alignment changes generation behavior
rinna reports that this PPO release repeats text more often than SFT v2. Its example uses a repetition penalty of 1.1. The comparison applies the same decoding settings to both checkpoints.Japanese GPT-NeoX 3.6B PPO official model card
Compare repetition as well as helpfulness
Practical guidance. Read a full long answer when comparing the two stages. An improved opening can hide repeated text later. Keep sampling settings fixed before attributing a difference to reinforcement learning.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Access | Downloadable |
| Model identifiers | japanese-gpt-neox-3.6b-instruction-ppo |
| License | MIT.Japanese GPT-NeoX 3.6B PPO official model card |
| Training | CarperAI trlx was used for the PPO stage.Japanese GPT-NeoX 3.6B PPO official model card |
Related models
- Japanese GPT-NeoX 3.6B SFT v2
rinna ยท Japanese GPT-NeoX 3.6B
A second supervised conversation fine-tune of Japanese GPT-NeoX 3.6B.
Launch date not confirmedDownloadable