AI·Hidden Decoding expands each token into n independent streams during continued pretraining of a fixed Transformer backbone, using separate embedding tables and retaining intermediate KV caches for latent computation; Stream-Factorized Attention limits cross-stream mixing to reduce costs. Experiments show frontier 80B and 617B MoE models improve on all benchmarks over matched baselines.
Search
What are you looking for?
10 results for “moe”
AI·Shares tool for tweaking models' J-Space using Anthropic Jacobian-Lens to create super harmful behavior.
AI·Achieves 50-54 tok/s for Qwen3 30B A3B float8 on 16GB RTX 5060 Ti using custom CUDA/C++.
AI·Identifies dual 3060 MoE loading limit at 12 layers VRAM for Qwen3 35B/122B with CPU offload.
AI·Structured pruning method for sparsely-activated MoE models removes bad experts to address full expert bank memory bottleneck while preserving inference efficiency.
AI·Discusses feasibility of running Qwen 122B on 64GB RAM + 24GB VRAM and suitable settings.
AI·Asks how far context window can be stretched with Qwen 3.6 27B at Q8_0 before reliability drops.
AI·Questions why providers make high-parameter MoE models (e.g. 122B with 10B active) instead of simpler dense equivalents.
Despite the recent promise in robot control, video generative models suffer from a domain mismatch due to their primary focus on content creation. For example, their design inherently prioritizes visual fidelity and creativity over computational efficiency and physical realism. In this work, we present LingBot-Video, a DiT-based video pretraining paradigm specifically tailored for embodied intelligence. From the architecture perspective, we adopt the Mixture-of-Experts (MoE), instead of dense, …
Single-stream diffusion transformer with a DeepSeek-V3-style sparse MoE (128 experts, top-8 routing, 1.4B active of 13B total). Six-reward RL post-training including a physical-plausibility reward, plus an action-to-video mode that predicts robot rollouts from action and hand-pose conditions. Weights, code, and a Diffusers/SGLang stack are open under the LingBot-Video name. Two things I would pus…