Sakana AIFwPKM
fwpkm-l12-fa-pkm6-fwpkm2_10-pswa0.9
This research checkpoint has 12 layers and combines full attention with static and fast-weight product-key memory.
Research release
The checkpoint selects a specific memory architecture
Every layer uses full attention. The two fast-weight memory layers supplement the static layer.fwpkm-l12-fa-pkm6-fwpkm2_10-pswa0.9 released configuration
Keep memory state explicit in an experiment
Practical guidance. Record when memory is reset and how many times a document is read. Compare equal protocols; extra passes can change results without changing the checkpoint.
The pSWA suffix records a training change
During training, full-attention layers used a 512-position window with probability 0.9. This encouraged the model to use fast-weight memory.Fast-weight Product Key Memory paper, February 2026 revision
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | SakanaAI/fwpkm-l12-fa-pkm6-fwpkm2_10-pswa0.9 |
| Architecture | Qwen3NextMemForCausalLM; hidden width 768.fwpkm-l12-fa-pkm6-fwpkm2_10-pswa0.9 released configuration |
| Static-memory layer indices | 6.fwpkm-l12-fa-pkm6-fwpkm2_10-pswa0.9 released configuration |
| Fast-weight layer indices | 2, 10.fwpkm-l12-fa-pkm6-fwpkm2_10-pswa0.9 released configuration |
| Weight terms | The reviewed repository provides weights but no explicit checkpoint license.fwpkm-l12-fa-pkm6-fwpkm2_10-pswa0.9 weights and file listing |