Hermes Mamba 2.8B SlimPajama
This model applies OpenHermes 2.5 supervised chat training to a 2.8B Mamba base pretrained on Cerebras SlimPajama. It differs from the Pile-based Hermes Mamba release through its pretraining data.
Research release
Keep the two data stages distinct
SlimPajama describes the base pretraining corpus. OpenHermes 2.5 supplies the later chat examples. The separately published cDPO model adds preference training; those weights are not included simply by loading this supervised version.Hermes-mamba-2.8b-slimpj model cardHermes-mamba-2.8b-slimpj model configuration
Use the released chat markers
These adaptations replace the usual ChatML start and end markers with ASCII control characters 0x16 and 0x17. The card describes this choice for RWKV tokenizer compatibility. Literal ChatML tokens are not interchangeable with the published format.Hermes-mamba-2.8b-slimpj model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | EleutherAI/Hermes-mamba-2.8b-slimpj |
| Base architecture | Mamba; SlimPajama-pretrained baseHermes-mamba-2.8b-slimpj model cardMamba state-space architecture and original models |
| Nominal parameters | 2.8BHermes-mamba-2.8b-slimpj model card |
| License | Apache 2.0Hermes-mamba-2.8b-slimpj model card |
Related models
- Hermes Mamba 2.8B
EleutherAI · Hermes Mamba
A chat adaptation of the Pile-trained Mamba 2.8B base.
Launch date not confirmedResearch releaseDownloadable - Hermes Mamba 2.8B SlimPajama cDPO
EleutherAI · Hermes Mamba
A preference-trained variant of the SlimPajama Hermes Mamba model.
Launch date not confirmedResearch releaseDownloadable