AI·Hidden Decoding expands each token into n independent streams during continued pretraining of a fixed Transformer backbone, using separate embedding tables and retaining intermediate KV caches for latent computation; Stream-Factorized Attention limits cross-stream mixing to reduce costs. Experiments show frontier 80B and 617B MoE models improve on all benchmarks over matched baselines.
Search
What are you looking for?
1 result for “continued-pretraining”