Sakana AIFwPKM
fwpkm-l12-gdn_swa-pkm2_6_10
This research checkpoint has 12 layers and combines gated deltanet with sliding-window attention with static product-key memory.
Research release
The checkpoint selects a specific memory architecture
Its layer sequence uses three linear-attention layers followed by one sliding-attention layer. This pattern repeats across the model.fwpkm-l12-gdn_swa-pkm2_6_10 released configuration
Keep memory state explicit in an experiment
Practical guidance. Use this checkpoint as a static-memory comparison for the corresponding fast-weight model. Keep training and inference settings aligned so the memory change can be assessed.
Specifications & access
| Detail | Value and conditions |
|---|---|
| Status | Research release |
| Access | Downloadable |
| Model identifiers | SakanaAI/fwpkm-l12-gdn_swa-pkm2_6_10 |
| Architecture | Qwen3NextMemForCausalLM; hidden width 768.fwpkm-l12-gdn_swa-pkm2_6_10 released configuration |
| Static-memory layer indices | 2, 6, 10.fwpkm-l12-gdn_swa-pkm2_6_10 released configuration |
| Fast-weight layer indices | None in this baseline.fwpkm-l12-gdn_swa-pkm2_6_10 released configuration |
| Weight terms | The reviewed repository provides weights but no explicit checkpoint license.fwpkm-l12-gdn_swa-pkm2_6_10 weights and file listing |
| Sliding window | 512 positionsfwpkm-l12-gdn_swa-pkm2_6_10 released configuration |