Kimi K3：开放前沿智能技术报告

TECHNICALREPORT OFKIMIK3 Kimi Team ABSTRACT We introduce Kimi K3, a 2.8T parameter Mixture-of-Experts model with 104 billion activatedparameters, native vision capabilities, and a 1-million-token context window. Kimi K3 is built onKimi Delta Attention [64] and Attention Residuals [58], which improve information flow acrosssequence length and model depth. Together with Stable LatentMoE, which effectively activates16 of 896 routed experts per token, and refined training and data recipes, these advances yield anapproximately2.5×improvement in overall scaling efficiency over Kimi K2 [59]. Post-traininghighlights reinforcement learning across general, agentic, and coding domains and multiple reasoning-effort levels, enabling compositional generalization and robust long-horizon execution. At 2.8T scale,Kimi K3 is supported by infrastructure advances in multiple areas: algorithm–system co-design forKDA, perfectly balanced expert-parallel training with efficient memory management, million-tokenagentic RL with persistent rollout and sandbox states, and deployment innovations. Extensive evaluations show that Kimi K3 achieves frontier-level performance across long-horizoncoding, agentic, knowledge, reasoning, and vision tasks. While its overall performance still trails themost powerful proprietary models, namely Claude Fable 5 and GPT-5.6 Sol, Kimi K3 consistentlyoutperforms other open and proprietary models evaluated in our suite. We release the full Kimi K3model weights to facilitate future research and accelerate the broader deployment and adoption offrontier intelligence.1 1Introduction For much of the development of Large Language Models (LLMs), scaling meant investing more computation beforedeployment by training larger models on more data [55, 46]. The rise of reasoning models has established test-timecomputation as a second axis of scaling: OpenAI’s o-series scales reinforcement learning and test-time reasoning [85,84]; Anthropic’s extended-thinking models allocate adaptive thinking budgets and interleave reasoning with tool use [6,7]; DeepSeek-R1 [41] and Kimi K1.5 [119] show that large-scale reinforcement learning can elicit sophisticatedreasoning behaviors from strong pre-trained models; and Kimi K2.5 Agent Swarm [60] further extends test-timescaling from sequential reasoning to parallel agent coordination. These advances have made test-time scaling a centralfocus of frontier research. However, while the open-source model ecosystem has advanced rapidly on the secondaxis, it has progressed slowly on the first: many recent models remain within or slightly above the 1T-class parameterregime [146, 29, 136, 121]. As increasingly sophisticated reasoning and agentic reinforcement learning methodsare applied to pre-trained foundations of similar scale, open-source progress risks converging while the gap to thestrongest proprietary systems widens. With Kimi K3, we pursue both scaling axes together to the frontier: scaling thepre-trained foundation to unprecedented 3T-class parameters while scaling reinforcement learning, reasoning effort, andlong-horizon interaction at 1M context length. We introduce Kimi K3, a native multimodal Mixture-of-Experts model with 2.8 trillion total parameters, 104 billionactivated parameters, and a context window of up to one million tokens. Its architecture scales information flowacross sequence length, network depth, and model width. Kimi Delta Attention (KDA) [64] provides efficient long-sequence mixing, with periodically interleaved Gated MLA layers preserving global interaction. Attention Residuals(AttnRes) [58] allows each layer to selectively attend to representations from all preceding layers. Stable LatentMoEexpands the routed expert space to 896 experts, with 16 activated per token, while normalization, SiTU-GLU, andQuantile Balancing stabilize optimization at extreme sparsity. These architectural advances, combined with refined dataand training recipes, yield an approximately2.5×improvement in overall scaling efficiency over Kimi K2 [59]. We pair this pre-training foundation with post-training designed explicitly for 1M context test-time scaling. KimiK3 undergoes reinforcement learning across long-horizon coding, general agents, general reasoning and knowledgetasks, each spanning multiple reasoning-effort levels. Training environments include verifiable search and professionalknowledge work, software engineering and kernel optimization, multimodal reasoning with vision-in-the-loop tooluse, persistent assistant workflows, web development, and autonomous execution tasks. These environments train ageneral loop of reasoning, acting, observing, verifying, and adapting, often over hundreds or thousands of tool calls andmillions of accumulated context tokens. Domain- and effort-specialized policies are consolidated into a unified modelthrough multi-teacher on-policy distillation [76, 135, 29]. Realizing this regime requires infrastructure that scales with architecture comple