Llemma 7B
Llemma 7B specializes a Code Llama base through continued pretraining on mathematical text and code. EleutherAI and academic collaborators released the model weights.
Research release
Further training for mathematics
The smaller Llemma release continues training from Code Llama 7B. It received 200 billion tokens of further training, more than the larger Llemma model. Parameter size alone therefore does not describe the training comparison.llemma_7b model cardllemma_7b model configuration
A mixture of mathematics and code
Proof-Pile-2 combines mathematical web text, scientific papers and code. The project examines written mathematics, tool use and formal theorem proving. The base model is not an instruction fine-tune, and generated calculations or proofs still need verification.llemma_7b model cardLlemma release and mathematics training
Keep the upstream license in view
The weights retain the Llama 2 license because the model derives from Code Llama. The released model configuration specifies 4,096 positions. A larger limit claimed for another Code Llama distribution should not be assumed for this checkpoint.llemma_7b model cardllemma_7b model configuration
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/llemma_7b |
| Context window | 4,096 tokens Maximum position count in the released model configuration. llemma_7b model configuration |
| Nominal parameters | 7Bllemma_7b model card |
| Base model | Meta Code Llama 7Bllemma_7b model card |
| Additional training tokens | 200 billionllemma_7b model card |
| License | Llama 2llemma_7b model card |
Related models
- Llemma 34B
EleutherAI · Llemma
A 34B mathematical language model adapted from Meta Code Llama.
Launch date Research releaseDownloadable An instruction-tuned Llemma adaptation with a nonstandard end token.
Launch date not confirmedResearch releaseDownloadable