Gated Delta Networks: Improving Mamba2 with Delta Rule

Mamba2 has gating in order to let the matrix state $S_t$ forget things

DeltaNet was the $S_t$ as an associative key-value store update

Gated DeltaNet just combines both

That’s all. Also they intersperse sliding window attention layers