Drop-Upcycling Research Models
The Drop-Upcycling model collection preserves the dense baselines, expert variants and specialized BTX experts used in the authors' experiments.
Research release
The named variants are experimental controls
The collection lists FS, BTX, NU, RNU and DU variants. DU and RNU names record reinitialization settings such as 0.5 or 1.0. BTX's Japanese, English and code expert checkpoints are also available. The model index maps them to the paper's experiment tables.Drop-Upcycling Research Models official model card
Compare one training change at a time
Practical guidance. Use a matching dense size, corpus and training budget when comparing these variants. The collection is designed to expose training choices, so treating every checkpoint as an interchangeable chatbot would hide its value.
Part of the pretrained weights is reinitialized
Drop-Upcycling retains knowledge from a dense model while statistically reinitializing part of the expert weights. The authors publish the code, training data, checkpoints and logs so the construction methods can be compared.Drop-Upcycling research paper
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | FS-8x1.5BBTX-8x1.5BNU-8x1.5BFS-8x3.7BRNU-0.5-8x1.5BFS-8x152MBTX-8x152MNU-8x152MRNU-0.5-8x152MDU-0.5-8x1.5BDU-0.5-8x152MDU-1.0-8x152MDense-152MDU-0.5-8x3.7BDense-1.5BDense-3.7BDU-1.0-8x1.5BDense-13BDense-btx-code-expert-1.5BDense-btx-english-expert-1.5BDense-btx-japanese-expert-1.5BDense-btx-code-expert-152MDense-btx-japanese-expert-152MDense-btx-english-expert-152M |
| License | Apache 2.0.Drop-Upcycling Research Models official model card |
| Coverage | 24 checkpoint identifiers, grouped by the research experiment.Drop-Upcycling Research Models official model card |
| Published sizes | Dense 152M, 1.5B, 3.7B and 13B; corresponding eight-expert experiments where listed.Drop-Upcycling Research Models official model card |
Related models
- LLM-jp-3 8×1.8B Base
LLM-jp / NII LLMC · LLM-jp-3 MoE
LLM-jp-3 8×1.8B Base is an eight-expert language model with two experts selected per token.
Launch date not confirmedResearch releaseDownloadable - LLM-jp-3 8×13B Base
LLM-jp / NII LLMC · LLM-jp-3 MoE
LLM-jp-3 8×13B Base is an eight-expert language model with two experts selected per token.
Launch date not confirmedResearch releaseDownloadable - Optimal Sparsity Math Models
LLM-jp / NII LLMC · Optimal Sparsity
The math branch of a controlled expert-model research sweep, with exact variant identifiers kept together.
Launch date not confirmedResearch releaseDownloadable