Gated Delta Networks: Improving Mamba2 with Delta Rule

Mamba2 has gating in order to let the matrix state \(S_t\) forget things

DeltaNet was the \(S_t\) as an associative key-value store update

Gated DeltaNet just combines both

That’s all. Also they intersperse sliding window attention layers