Comma v0.1 2T
Comma v0.1 2T is a jointly developed Common Pile base model. Its training data draws on public-domain and openly licensed English text and code.
Research release
Training stages and final weights
The main stage used 1.93 trillion tokens, followed by 75.5 billion cool-down tokens. The final weights average ten cool-down checkpoints. Both stages used batches of 8.3 million tokens. Training ran on 512 AMD MI300A GPUs.comma-v0.1-2t model card
What differs between the Comma releases
The two-trillion-token release increases training exposure rather than parameter count. It remains a seven-billion-parameter base model. Comparing it with Comma 1T helps study the effect of longer training on the same broad data mixture.comma-v0.1-2t model card
Open licensing has a documented limit
The authors warn that incorrect metadata and license laundering can contaminate the data. They do not guarantee that every training text has an open license. The model has no instruction or safety alignment, and its language coverage centers on English and code.comma-v0.1-2t model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | common-pile/comma-v0.1-2t |
| Context window | 16,384 tokens Maximum position count in the released model configuration. comma-v0.1-2t model configuration |
| Nominal parameters | 7 billioncomma-v0.1-2t model card |
| License | Apache 2.0comma-v0.1-2t model card |
| Publisher | Common Pile collaborationCommon Pile and Comma release announcement |
Related models
- Comma v0.1 1T
EleutherAI ยท Comma
A seven-billion-parameter Common Pile model trained for one trillion tokens.
Launch date Research releaseDownloadable