Optimal Sparsity Math Models
Optimal Sparsity Math is an experimental checkpoint family released with the authors' sparsity study. These are distinct trained configurations grouped for research comparison.
Research release
Each identifier records an architecture configuration
The family belongs to the Optimal Sparsity paper and its author-maintained training code. It is a controlled research release rather than a single general assistant.Optimal Sparsity Math Models official model card
Read the variant name
In each identifier, d is hidden width, E is the number of experts, and k is the number selected. The last two fields give total and active parameters. For example, d512-E8-k2-320M-A170M and d2048-E128-k2-52.2B-A1.5B represent different model sizes and routing costs.Optimal Sparsity author repository and identifier format
Keep compute and accuracy separate
Practical guidance. Use the sweep to study how expert sparsity affects reasoning under a compute budget. Picking the lowest training loss alone would miss the question the study investigates.
The study separates memorization and reasoning
The authors vary total parameters, active parameters and routing under fixed compute budgets. Their paper reports that equal pretraining loss does not guarantee equal reasoning accuracy. It also studies reinforcement post-training and additional inference computation.Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | optimal-sparsity-math-d512-E8-k2-320M-A170Moptimal-sparsity-math-d512-E16-k2-520M-A170Moptimal-sparsity-math-d2048-E8-k2-3.9B-A1.5Boptimal-sparsity-math-d2048-E16-k2-7.1B-A1.5Boptimal-sparsity-math-d2048-E32-k2-13.6B-A1.5Boptimal-sparsity-math-d2048-E16-k16-7.1B-A7.1Boptimal-sparsity-math-d2048-E8-k8-3.9B-A3.9Boptimal-sparsity-math-d2048-E64-k2-26.4B-A1.5Boptimal-sparsity-math-d2048-E32-k16-13.6B-A7.1Boptimal-sparsity-math-d2048-E16-k8-7.1B-A3.9Boptimal-sparsity-math-d1024-E16-k16-1.9B-A1.9Boptimal-sparsity-math-d2048-E32-k8-13.6B-A3.9Boptimal-sparsity-math-d2048-E64-k16-26.4B-A7.1Boptimal-sparsity-math-d1024-E32-k16-3.5B-A1.9Boptimal-sparsity-math-d2048-E128-k2-52.2B-A1.5Boptimal-sparsity-math-d1024-E64-k16-6.7B-A1.9Boptimal-sparsity-math-d2048-E64-k8-26.4B-A3.9Boptimal-sparsity-math-d1024-E128-k16-13.2B-A1.9Boptimal-sparsity-math-d512-E32-k2-920M-A170Moptimal-sparsity-math-d1024-E8-k2-1.1B-A470Moptimal-sparsity-math-d512-E64-k2-1.7B-A170Moptimal-sparsity-math-d2048-E128-k16-52.2B-A7.1Boptimal-sparsity-math-d512-E8-k4-320M-A220Moptimal-sparsity-math-d1024-E16-k2-1.9B-A470Moptimal-sparsity-math-d512-E16-k4-520M-A220Moptimal-sparsity-math-d512-E8-k8-320M-A320Moptimal-sparsity-math-d512-E16-k8-520M-A320Moptimal-sparsity-math-d512-E16-k16-520M-A520Moptimal-sparsity-math-d512-E32-k4-920M-A220Moptimal-sparsity-math-d512-E128-k2-3.3B-A170Moptimal-sparsity-math-d512-E32-k8-920M-A320Moptimal-sparsity-math-d512-E32-k16-920M-A520Moptimal-sparsity-math-d1024-E256-k16-26.0B-A1.9Boptimal-sparsity-math-d512-E64-k4-1.7B-A220Moptimal-sparsity-math-d1024-E32-k2-3.5B-A470Moptimal-sparsity-math-d512-E64-k8-1.7B-A320Moptimal-sparsity-math-d512-E64-k16-1.7B-A520Moptimal-sparsity-math-d2048-E128-k8-52.2B-A3.9Boptimal-sparsity-math-d512-E128-k4-3.3B-A220Moptimal-sparsity-math-d512-E256-k2-6.6B-A170Moptimal-sparsity-math-d512-E128-k8-3.3B-A320Moptimal-sparsity-math-d512-E128-k16-3.3B-A520Moptimal-sparsity-math-d1024-E64-k2-6.7B-A470Moptimal-sparsity-math-d512-E256-k4-6.6B-A220Moptimal-sparsity-math-d512-E256-k8-6.6B-A320Moptimal-sparsity-math-d512-E256-k16-6.6B-A520Moptimal-sparsity-math-d1024-E128-k2-13.2B-A470Moptimal-sparsity-math-d1024-E256-k2-26.0B-A470Moptimal-sparsity-math-d1024-E8-k4-1.1B-A670Moptimal-sparsity-math-d1024-E128-k4-13.2B-A670Moptimal-sparsity-math-d1024-E16-k4-1.9B-A670Moptimal-sparsity-math-d2048-E8-k4-3.9B-A2.3Boptimal-sparsity-math-d1024-E32-k4-3.5B-A670Moptimal-sparsity-math-d1024-E8-k8-1.1B-A1.1Boptimal-sparsity-math-d1024-E256-k4-26.0B-A670Moptimal-sparsity-math-d1024-E16-k8-1.9B-A1.1Boptimal-sparsity-math-d2048-E16-k4-7.1B-A2.3Boptimal-sparsity-math-d1024-E64-k4-6.7B-A670Moptimal-sparsity-math-d1024-E32-k8-3.5B-A1.1Boptimal-sparsity-math-d2048-E32-k4-13.6B-A2.3Boptimal-sparsity-math-d1024-E64-k8-6.7B-A1.1Boptimal-sparsity-math-d2048-E64-k4-26.4B-A2.3Boptimal-sparsity-math-d2048-E128-k4-52.2B-A2.3Boptimal-sparsity-math-d1024-E128-k8-13.2B-A1.1Boptimal-sparsity-math-d1024-E256-k8-26.0B-A1.1B |
| Access | Downloadable model checkpoints; Apache 2.0 model cards.Optimal Sparsity Math Models official model card |
| Coverage | 65 published identifiers are grouped in this article.Optimal Sparsity Math Models official model card |
Related models
- Drop-Upcycling Research Models
LLM-jp / NII LLMC · Drop-Upcycling
Published dense and expert-model experiments for studying partial weight reinitialization during upcycling.
Launch date not confirmedResearch releaseDownloadable - Optimal Sparsity Code Models
LLM-jp / NII LLMC · Optimal Sparsity
The code branch of a controlled expert-model research sweep, with exact variant identifiers kept together.
Launch date not confirmedResearch releaseDownloadable