Hermes Mamba 2.8B
This adaptation fine-tunes a 2.8-billion-parameter Mamba base on the OpenHermes 2.5 dataset from Nous Research. The base was pretrained on the Pile. The EleutherAI contribution here is the chat adaptation, not the original Mamba architecture.
Research release
State-space inference
The model uses the Mamba selective state-space implementation rather than standard Transformer attention. Its card points to the state-spaces Mamba code for loading. The recurrent design does not establish an unlimited reliable context window.Hermes-mamba-2.8b model cardHermes-mamba-2.8b model configuration
Use the released chat markers
These adaptations replace the usual ChatML start and end markers with ASCII control characters 0x16 and 0x17. The card describes this choice for RWKV tokenizer compatibility. Literal ChatML tokens are not interchangeable with the published format.Hermes-mamba-2.8b model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | EleutherAI/Hermes-mamba-2.8b |
| Base architecture | Mamba; Pile-pretrained baseHermes-mamba-2.8b model cardMamba state-space architecture and original models |
| Nominal parameters | 2.8BHermes-mamba-2.8b model card |
| License | Apache 2.0Hermes-mamba-2.8b model card |
Related models
- Hermes Mamba 2.8B SlimPajama
EleutherAI · Hermes Mamba
A Mamba chat model whose base was trained on SlimPajama.
Launch date not confirmedResearch releaseDownloadable - Hermes Mamba 2.8B SlimPajama cDPO
EleutherAI · Hermes Mamba
A preference-trained variant of the SlimPajama Hermes Mamba model.
Launch date not confirmedResearch releaseDownloadable