Musings on Recursive Self-Improvement

Hard takeoffs are... hard

When thinking about recursive self-improvement, there are two things to separate out: whether we’re talking about models (and scaffolds) improving more rapidly, or the wider societal/economic impacts of this recursive dynamic - i.e., stuff that depends mostly on deployments, adoption, and other bottlenecks. The lines are sometimes blurred in the discourse, and I want to make this separation clearer here. I also suspect many of these barriers are more structurally resistant to being dissolved by better capabilities than people often assume.

The models will get better

On the former: yes, we are already using AI to filter data, write better code, build experimental setups, and so on. These enhancements can make the human-led process of developing and researching models more efficient. As a result, I expect the time required to achieve a given increase in model capability to fall over time. Since we want good models, this is good news. But there are a few things worth noting here:

First, there are huge costs in doing all this. Even if much of the model development process is automated, frontier training still requires massive amounts of capital and compute. So far labs have deep reserves or investors willing to fund losses in expectation of future gains. But over any longer horizon, these costs still need to be justified economically, and real-world deployment remains one important way this happens. In a way, this is analogous to what Dario was telling Dwarkesh: “Even though a part of my brain wonders if it’s going to keep growing 10x, I can’t buy $1 trillion a year of compute in 2027. If I’m just off by a year at that rate of growth, or if the growth rate is 5x a year instead of 10x a year, then you go bankrupt.”

Second, when 95% of a process is automated, the remaining 5% can act as an important speed limiter: this can be taste, creativity, bureaucracy, anything that may require ‘human time’. Executives in large organisations are more cognizant of these than researchers who only see their immediate aperture. Now, economic history warns us against assuming new technology will simply replicate legacy pipelines perfectly. Just as the Indian pharmaceutical industry in the 1990s bypassed Western R&D bottlenecks by inventing vastly cheaper manufacturing processes, AI might circumvent current human bottlenecks by inventing entirely new ‘menus of production.’ But even the Indian pharma revolution took years of trial, regulatory navigation, and institutional adaptation before it reshaped the industry. Routing around bottlenecks is itself a deployment problem, not an overnight breakthrough.

Third, even if model improvement accelerates sharply, that alone is not sufficient. Aggregate capability gains only matter insofar as they can be identified, productized, and translated into real-world use. I think a lot of people continue to underestimate the importance of deployments across society, which matter for both (a) justifying training costs; and (b) generating useful data to improve models that customers want to continue using; (c) understanding the strengths and limitations of an existing generation of models in real world settings; and (d) getting the transformative changes you want to see in the world.

Finally, there may also be diminishing returns within any given paradigm, since this can be observed nearly everywhere. It is unclear where and when these hit, but my intuition is that current systems are especially strong at exploiting paradigms that already exist, particularly in domains with dense, legible feedback where RL and synthetic data can productively extend the loop. That can still produce extremely fast progress. But I am less sure that these same systems smoothly generalize from paradigm-exploitation to paradigm-generation, by which I mean the creation of genuinely new abstractions and their productive reintegration into the training and research loop.

More narrowly, I do not think it is yet obvious that we have entered an actual ‘intelligence explosion’, as opposed to simply extending the AI-assisted development loop that was already underway over the past few years. Crucially though, the diminishing returns I’m describing are returns within the current paradigm. One could argue that RSI itself is the mechanism by which you escape this — that sufficiently capable systems identify entirely new abstractions and innovative architectural approaches. I expect a version of that in the coming decade, but initially still through a cyborgian dynamic where researchers leverage increasingly capable agents to crack problems that neither humans nor models would solve alone. In any event, the broader argument does not depend on where exactly that ceiling lies.

When people talk about recursive self-improvement, they sometimes acknowledge these frictions but then treat them as secondary, or assume that sufficiently capable systems can route around most of them via internal deployments and accelerated R&D. I think this is often overstated: these bottlenecks do not disappear just because model development speeds up. They are structural, not incidental, and they push strongly against the more explosive versions of the RSI story.

The inconvenience of deployment

On the deployment side, things get even more complicated. Deploying models into the world is not just a ‘nice to have’ thing that labs do out of charity. Labs have strong incentives to see these systems deployed, permitted, adopted, and integrated across the economy. Over time, this is one major way the scale of frontier spending gets justified. And in parallel, you need to go through the court cases, the regulatory burdens, the legal compliance, the weird adoption dynamics, the integration into legacy systems, the cultural adjustments, the political headwinds, everything! There are all sorts of reasons why deployment takes time and I think people are too quick to just wave these away with some handwavy remark about ‘competitive pressures’. This is less a point about narrow model self-improvement than about industrial diffusion: even if models improve quickly, the automation of the economy still has to run through deployment bottlenecks.

When people talk about recursive self-improvement and then talk about society being unrecognizably transformed at a very fast speed, they’re not talking about models developing, but essentially about the entire economy self-improving, where every physical and human constraint disappears. I think it’s uncontroversial to claim that getting to this point will take time. Even if you get much better robots in the coming years, which I expect will happen, getting humans completely out of the physical and digital economy loop is a pretty damn high bar. And even in such a world, you still do not get a ‘hard takeoff’, because so much remains tethered to human time still.

This points to a general issue in a lot of AI thinking: the concepts of consumption and demand are often muddled, and the focus is solely on the supply of capabilities. To make sense of this, we need to clearly separate economic demand (the rate at which human, and ultimately AI, consumers buy, adopt, and integrate products day-to-day) from final utility (the ultimate human purpose or directive that gives this economic activity a reason to exist).

For some time, I expect the economy’s ability to absorb, integrate, and productively deploy these systems to remain an important constraint, although not forever. Viewed through the macroeconomic lens of Say’s Law and capital deepening, it’s true that immediate consumer spending doesn’t necessarily have to be a hard speed limit per se. If AI triggers massive technological deflation, the economy could in principle equilibrate by reinvesting excess surplus into highly capital-intensive processes: essentially, machines building data centers and robots for other machines. This means an ‘Agent-to-Agent’ (A2A) economy can grow incredibly fast without waiting for humans to consume final products today.

Yet, even if this automated A2A loop takes hold someday, it remains fiercely tethered to final utility. Conditional on systems remaining broadly aligned and instruction-following (which is my current assumption), AIs will not be consuming for their ‘own’ sake: they do not possess intrinsic utility, and they do not build server farms for their own amusement. They are doing so purely a extensions of what a human principal somewhere in space and time ultimately desires. It’s also worth noting that this does not require perfect alignment: human economies have always operated with all sorts of principal-agent problems and we manage these through institutional design, incentives, monitoring, and redundancy, not by solving them in the abstract or by relying on an ‘aligned vs misaligned’ dichotomy.

Imagine a human gives an AI system a top-level directive: invent and mass produce a cure for Alzheimer’s. An autonomous A2A supply chain spins up: Agent A (the R&D lab) realizes it needs 100x more compute. Agent A pays Agent B to build a massive new data center. Agent B pays Agent C to mine the silicon, copper, and steel required. Agent C pays Agent D to build a fusion reactor to power the mining equipment. In this scenario, 99.9% of the economic activity is A2A; trillions of dollars are moving, and massive physical infrastructure is being built. No human had to buy a final product, click an ad, or culturally adapt to to keep this massive industrial boom running. Economically, this loop successfully bypasses the friction of human consumers.

But the initial “seed” of all this activity is still a human goal, and that is the tethered link. Because the A2A ouroboros is anchored to human purposes, it does not operate in a frictionless void. To deliver something like an Alzheimer’s cure, the relevant systems will often still need to interface with the human world: biological reality, legal and institutional processes, property and infrastructure constraints, and human judgments about acceptable risk. Some of these interfaces may become faster and more automated, but institutional adaptation is itself often contested and uneven (which is often a feature, not a bug).

So at some point, the bottleneck is no longer how fast humans can buy/consume things, but how fast AIs can deploy, verify, and physically build things in our highly frictional, human-regulated world. Reality bites: this ouroboros-shaped economy cannot spontaneously generate in a vacuum; it must navigate legacy infrastructure, power grids, API limits, and regulatory realities (yes, they will exist then too, for good reason). As long as AIs are instruction-following, there are no runaway scenarios. So whilst orders of magnitude more efficient than industry today, we shouldn’t confuse a future automated supply chain with a frictionless hard-takeoff type singularity.

What to make of this

It’s worth noting that the very forces that push toward better model development and faster experimentation — the general purpose nature of the improvements that AI provides — also apply to safety, to control, monitoring, verification, robustness, and all sorts of other desirable things. It is in the interest of companies and whoever adopts and uses these agents for them not to be reward-hacking, or for their agents not to do weird things no one asked, or for them to be vulnerable to serious attacks that threaten their consumer base.

So automating ML R&D should also accelerate many of the safety-relevant properties we care about, such as interpretability or getting more deterministic systems with better controls. This only looks implausible if you think of capabilities and alignment as almost entirely separate domains. I do not think that separation really holds. Many safety properties are deeply entangled with broader advances in model quality and engineering, even if that does not mean every failure mode is solved automatically. AI systems are engineered machines, and I expect some of the same forces accelerating capabilities to be brought to bear on alignment, control, and oversight as well. The case for using more intelligence to accelerate alignment work is at least as strong as the case for buying time to do that work manually.

And to be clear, as usual, that’s not to say everything will go perfectly well or that society is perfectly calibrated to handle new technologies optimally. Naturally, I expect all sorts of negative developments and externalities, though I expect many of these will get addressed if they become problematic enough; for example I do expect more cyber incidents in the short run but better adaptation over time (just as we did with spam or DDoS attacks). In general, it’s clear that you want a lot of resources devoted to safety and governance, which I think we do today (and will continue doing). And of course, in a world where you get incrementally faster deployments and societal developments taking shape, you also want governance to be benefiting from the technology. Think of the early days of the internet: you definitely want courts, regulators, and civil society to use the internet too, otherwise they wouldn’t do their job effectively at all. I think the same applies here, and improving governance and institutions remains one of the most important things to focus on in the next few years.

To conclude, the term ‘recursive self-improvement’ often conjures a science-fiction image of a blurry abstraction magically improving itself overnight and leading to some sort of hard take-off. The reality will be both more grounded and more profound. Because we are essentially ‘inventing the inventors’, we may well be heading toward a period of very high economic growth. Even so, I remain sceptical that this translates into a super-exponential takeoff in the wider economy within the current decade, even if model capabilities continue improving rapidly.

But rejecting an instantaneous ‘hard takeoff’ today doesn’t mean using AI to improve AI is no big deal. When this super-exponential flywheel eventually spins up, it won’t do so in a frictionless vacuum and will be tethered to the physical world, constrained by energy limits, robotic manufacturing speeds, and the messy reality of integrating software and robots across human institutions and societies. Unless you believe more intelligence magically bypasses all of this, or that it necessarily means power-seeking and deception, then the future is less about an overnight singularity and more about navigating a massively accelerated, but ultimately jagged and physical, industrial revolution. Self-improvement itself will be uneven: a jagged frontier where breakthroughs in some domains coexist with stubborn stasis in others. We have a window of time to upgrade our institutions for what’s coming, and I think one of the most effective ways to do so is by deploying AI across governance and institutions themselves.

With thanks to Nathaniel Bechhofer, Rohin Shah, Samuel Albanie, Jamie Rumbelow, Ben Clifford, Tim Hwang, Harry Law, and Gustavs Zilgalvis for discussions and feedback.