自动驾驶的世界模型综述

Tuo Feng, Wenguan Wang,Senior Member, IEEE, Yi Yang,Senior Member, IEEE Abstract—Recent breakthroughs in autonomous driving have revolutionized the way vehicles perceive and interact with theirsurroundings. In particular, world models have emerged as a linchpin technology, offering high-fidelity representations of the drivingenvironment that integrate multi-sensor data, semantic cues, and temporal dynamics. Such models unify perception, prediction, andplanning, thereby enabling autonomous systems to make rapid, informed decisions under complex and often unpredictable conditions.Research trends span diverse areas, including 4D occupancy prediction and generative data synthesis, all of which bolster sceneunderstanding and trajectory forecasting. Notably, recent works exploit large-scale pretraining and advanced self-supervised learningto scale up models’ capacity for rare-event simulation and real-time interaction. In addressing key challenges – ranging from domain Index Terms—Autonomous Driving, World Models, Self-Supervised Learning, Behavior Planning, Generative Approaches 1INTRODUCTION posing millisecond-level responsiveness to address unex-pected obstacles or anomalous behaviors in the driving en- 1.1OverviewT HEbecome a global focal point in both scientific research vironment [8], [9]. Equally pivotal is the system’s resiliencein extreme or long-tail scenarios (e.g., severe weather, con- simultaneously reduce traffic accidents, alleviate conges-tion, and enhance mobility for diverse societal groups [1].Current statistics underscore that human error remains theprincipal cause of accidents on the road [2], indicatingthatminimizing human intervention could significantly Within this context, constructing robust and stableworldmodelshas emerged as a foundational element. The no-tion of a world model involves creating a high-fidelityrepresentation of the driving environment – encompassingstatic structures (e.g., roads, buildings) and dynamic enti-ties (e.g., vehicles, pedestrians) [3], [8]. A comprehensiveworld model continuously captures semantic and geometricinformation while updating these representations in real-time, thereby informing downstream tasks such as physi- lower the incidence of traffic-related fatalities and injuries.Beyond safety, economic factors (e.g., reducing congestion cal world prediction [12], [13]. Recent advances integrate Despitethese compelling incentives,achieving high-level autonomy demands overcoming substantial technicalhurdles. Foremost among these is perceiving and under-arXiv:2501.11260v1 [cs.RO] 20 Jan 2025 standing dynamic traffic scenarios, which requires fusingheterogeneoussensor streams(e.g.,LiDAR,radar,cam-eras)into a cohesive environmental representation[4],[5].From complex urban layouts to high-speed high-ways, autonomous vehicles must rapidly assimilate multi- In turn, these robust world models leverage environ-mental representations to optimize the behavior planningof intelligent agents, providing the keystone for safer andmoreefficient autonomous driving applications.By en-abling proactive trajectory optimization, real-time hazarddetection, and adaptive route planning, they directly mit- Existingsurveys on world models that involve au-tonomous driving can generally be classified into two cat- egories. The mainstream category focuses on describinggeneral world models that find applications across mul-tiple fields [20]–[22], with autonomous driving being justone of the specific areas. The second category [23], [24],concentrates on the application of world models within theautonomous driving sector, and attempts to summarize thecurrent state of the field. There are only a few existingsurveys on world models in autonomous driving, they tendto broadly categorize these studies and often focus solelyon world simulation or lack discussions on the interaction 1.3Structure A summary of the structure of this paper can be found inFig. 1, which is presented as follows: Sec. 1 introduces thesignificance of world models in autonomous driving andoutlines the societal and technical challenges they address.Sec. 2 provides a comprehensive background on the for-mulation and core tasks of world models in autonomousdriving, specifically focusing on the future prediction of thephysical world and behavior planning for intelligent agents.Sec. 3 details the taxonomy of methods: Sec. 3.1 delvesinto methods for future prediction of the physical world,discussing physical world evolution of dynamic objects andstatic entities. Sec. 3.2 discusses advanced behavior planningapproaches that emphasize the generation of safe, effec-tive driving strategies. Sec. 3.3 investigates the interactiverelationship between future prediction and behavior plan-ning, highlighting collaborative optimization techniques for 1.2Contributions Guided by the principle that the world model is centralto the understanding of dynamic scenes, this survey aimsto provide a comprehensive, structured review o