Comma v0.1 1T
Comma v0.1 1T is a jointly developed Common Pile base model. Its training data draws on public-domain and openly licensed English text and code.
Research release
Training stages and final weights
The first stage used 965 billion tokens with a cosine learning-rate schedule. A further 35 billion high-quality tokens formed a cool-down stage. The released weights average ten checkpoints from that final stage. Training used 64 Nvidia H100 GPUs.comma-v0.1-1t model card
What differs between the Comma releases
The one-trillion-token run provides a lower-training-budget comparison with Comma 2T. Both use a Llama 3 architecture, but the project trained these weights from scratch. They are not adaptations of Meta Llama 3 weights.comma-v0.1-1t model card
Open licensing has a documented limit
The authors warn that incorrect metadata and license laundering can contaminate the data. They do not guarantee that every training text has an open license. The model has no instruction or safety alignment, and its language coverage centers on English and code.comma-v0.1-1t model card
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | common-pile/comma-v0.1-1t |
| Context window | 16,384 tokens Maximum position count in the released model configuration. comma-v0.1-1t model configuration |
| Nominal parameters | 7 billioncomma-v0.1-1t model card |
| License | Apache 2.0comma-v0.1-1t model card |
| Publisher | Common Pile collaborationCommon Pile and Comma release announcement |
Related models
- Comma v0.1 2T
EleutherAI ยท Comma
A seven-billion-parameter Common Pile model trained for two trillion tokens.
Launch date Research releaseDownloadable