What are you looking for?

Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

AI·Sparse Delta Memory scales the hidden state of gated linear RNNs to orders of magnitude higher capacity via sparse addressing, improving long-context recall and in-context learning while keeping compute efficient.

Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

AI·The paper compares softmax attention with recurrent linear-attention architectures including DeltaNet and Gated DeltaNet, analyzing expressivity, memory decay, training throughput, and introducing cross-layer value routing.