Llemma 34B
Llemma 34B specializes a Code Llama base through continued pretraining on mathematical text and code. EleutherAI and academic collaborators released the model weights.
Research release
Further training for mathematics
The larger release starts from Code Llama 34B and receives 50 billion tokens of further training. Its released configuration uses 48 layers and grouped-query attention with eight key/value heads.llemma_34b model cardllemma_34b model configuration
A mixture of mathematics and code
Proof-Pile-2 combines mathematical web text, scientific papers and code. The project examines written mathematics, tool use and formal theorem proving. The base model is not an instruction fine-tune, and generated calculations or proofs still need verification.llemma_34b model cardLlemma release and mathematics training
Keep the upstream license in view
The weights retain the Llama 2 license because the model derives from Code Llama. The released model configuration specifies 4,096 positions. A larger limit claimed for another Code Llama distribution should not be assumed for this checkpoint.llemma_34b model cardllemma_34b model configuration
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Launch date | |
| Access | Downloadable |
| Model identifiers | EleutherAI/llemma_34b |
| Context window | 4,096 tokens Maximum position count in the released model configuration. llemma_34b model configuration |
| Nominal parameters | 34Bllemma_34b model card |
| Base model | Meta Code Llama 34Bllemma_34b model card |
| Additional training tokens | 50 billionllemma_34b model card |
| License | Llama 2llemma_34b model card |
Related models
- Llemma 7B
EleutherAI · Llemma
A 7B mathematical language model adapted from Meta Code Llama.
Launch date Research releaseDownloadable An instruction-tuned Llemma adaptation with a nonstandard end token.
Launch date not confirmedResearch releaseDownloadable