当AI构建AI：AI研发自动化研讨会成果报告

Executive Summary For decades, scientists have speculated about the possibility of machines that canimprove themselves. Today, artificial intelligence (AI) systems are increasingly integralparts of the research pipeline at leading AI companies. Some observers see this asevidence that fully automated AIresearch and development (R&D)is on the way,potentially leading to a rapid acceleration of AI capabilities and impaired ability forhumans to understand and control AI. Others see the use of AI for research as amundane extension of existing software tools. This Workshop Report shares findings and conclusions from an expert workshop CSEThosted in July 2025. The workshop covered a range of issues related to automation ofAIR&D.In this report,‘AI R&D’ refers to scientific and engineering work that improvesthe capabilities of AI systems and ‘AI R&Dautomation’ refersbroadlyto any use of AIthat accelerates progress in AI R&D. Key takeaways from the workshop were as follows: 1.Increasingly automated AI R&D is a potential source of major strategicsurprise.While experts disagree on likelihood, scenarios are possible in whichAI R&D becomes highly automated, the pace of AI R&D acceleratesdramatically, and the resulting systems pose extreme risks. This warrantspreparatory action now. 2.Frontier AI companies are already using AI to accelerate AI R&D, and usageis increasing as AI models get more advanced.New models are often usedinternally to advance AI R&D before they are released to the public. 3.Experts’ views differ on how rapid and impactful AI R&D automation is likelyto be.Even if the use of AI in AI R&D continues to increase, there is noconsensus on whether AI progress is more likely to accelerate or plateau.What’s more, because different views are associated with different assumptionsabout how AI R&D works, new data onhow AI R&D automation is progressingin practice may be insufficient to resolve conflicting perspectives. It thus may bedifficult to either detect or rule out extreme ‘intelligence explosion’ scenarios inadvance. 4.Despite challenges in interpreting new evidence, better access to indicatorsof progress in AI R&D automationwould be valuable.Existing empiricalevidence, including existing benchmark evaluations, is insufficient for measuring, Center for Security and Emerging Technology |1 understanding, and forecasting the trajectory of automated AI R&D. Moresystematic collection of existing indicators—as well as developing ways ofgathering new indicators—could provide a significantly clearer picture. 5.Thoughtfully designed transparency efforts could improve access to valuableempirical information about AI R&D automation, which at present is almostfully dependent on patchy, voluntary releases of information fromcompanies.While some early transparency mandates on frontier AIdevelopment have recently been enacted, they do not focus on indicators ofprogress in AI R&D automation. Policymakers have a range of options for howto increase visibility of these indicators. The full report elaborates on these takeaways, including providing examples of howfrontier AI companies are using AI for R&D, delving into experts’ differing views andassumptions, suggesting priority indicators to track, and laying out policy options andimplications. Table of Contents Executive Summary....................................................................................................................1Background and Motivation.....................................................................................................4What Is AI R&D Automation?.........................................................................................................4Why Does AI R&D Automation Matter?.....................................................................................5The Present and Future of Automating AI R&D..................................................................8What Does AI R&D Consist Of?....................................................................................................8How Is AI Being Used for AI R&D So Far?.................................................................................9How Far Could AI R&D Automation Go?.................................................................................10How Does Automating AI R&D Lead to Real-World Impacts?........................................15Indicators to Watch For...........................................................................................................17Metrics for Broad AI Capabilities................................................................................................17Benchmarks for AI Capabilities Specific to AI R&D..............................................................18Signs of How Automated AI R&D Is Progressing Inside AI Companies........................19Summary of Indicators................................................................................................................