What is recursive self-improvement? 'Worst idea in the history of humanity,' physics professor says

Key Points:

• AI developers are approaching recursive self-improvement (RSI), where AI models autonomously improve themselves and create successor models, potentially accelerating advances in science and medicine but also raising significant risks.
• Anthropic reported that its AI model Claude now leads 26% of its research and development tasks under human supervision, marking progress toward more autonomous AI-driven model improvement.
• Experts note that some level of RSI has existed for years, with AI assisting in developing newer models, but recent advances suggest labs are nearing fully autonomous RSI, which raises fears of uncontrollable superintelligence.
• Major AI companies differ in their approaches: OpenAI aims for an automated AI researcher by 2028 but cautions about safety and human control, while Elon Musk predicts near-full automation by 2027, and Microsoft emphasizes "humanist superintelligence" with strict limits.
• Calls to slow AI development to ensure safety measures keep pace have gained traction, with some companies like Anthropic willing to pause if others do so verifiably, but uncertainty remains about safely achieving fully autonomous RSI.