AI·Hidden Decoding expands each token into n independent streams during continued pretraining of a fixed Transformer backbone, using separate embedding tables and retaining intermediate KV caches for latent computation; Stream-Factorized Attention limits cross-stream mixing to reduce costs. Experiments show frontier 80B and 617B MoE models improve on all benchmarks over matched baselines.
Search
What are you looking for?
7 results for “attention”
AI·LLM-based ASR in regulated domains like banking is limited by privacy and real-speech collection costs; synthetic TTS data is a cost-effective substitute, but acoustic mismatch hinders supervised fine-tuning (SFT). Group Relative Policy Optimization (GRPO) applied solely to synthetic speech reduces WER by 40% relative to SFT (36.71% to 22.09%) and by 45% in the SFT-then-GRPO sequence, by improving behavioral calibration and audio attention rather than representations.
AI·Jet-Long is a tuning-free zero-shot method for efficient long-context extension in LLMs that uses dynamic bifocal RoPE with local faithful and adaptively rescaled long-range windows.
AI·Sparse Delta Memory scales the hidden state of gated linear RNNs to orders of magnitude higher capacity via sparse addressing, improving long-context recall and in-context learning while keeping compute efficient.
AI·The paper compares softmax attention with recurrent linear-attention architectures including DeltaNet and Gated DeltaNet, analyzing expressivity, memory decay, training throughput, and introducing cross-layer value routing.
Accurate breast cancer classification from mammography requires effective integration of complementary information from craniocaudal (CC) and mediolateral oblique (MLO) views, which provide a more complete characterization of breast abnormalities. However, existing multi-view learning approaches typically rely on feature-level aggregation or single-stage cross-attention, which can entangle view-specific and shared representations and restrict interaction to limited network depths. To address th…
Multimodal large language models (MLLMs) generate responses autoregressively, integrating visual and linguistic information in an evolving context. Prior work on interpretability has focused on individual layers and circuits (where), leaving the token-level dynamics of multimodal computation during generation (when) underexplored. We address this gap and study attention shifts as per semantic role; tracking model attention to image, text, instruction, and previously generated tokens, One Token …