Closing graf is precisely right. *We* are now what determine the pace and shape of our revolution. The human parts of us, how we collectively create “society.” Much of it isn’t algorithmic, and can’t be growth hacked
There's a collapse dynamic you don't address, and I think it's the most structurally interesting one. Model collapse on the data side is already documented - recursive training on AI-generated content progressively shrinks the tail of the distribution. But there's a symmetric process on the human side: the capacity for paradigm-generation isn't a fixed input to the loop. It's built and maintained through specific cognitive conditions, and dense LLM use systematically degrades exactly those conditions. So the cyborg loop doesn't just risk hitting a capability ceiling - it risks both halves converging toward the same exploitation architecture. Not a hard takeoff or a hard stop, but a slow narrowing that looks like progress until the distribution has no tails left.
I have often thought that a true AGI might not understand the basis of its own intelligence any more than we can map out the importance of all 100 trillion synapses in our own brains--let alone recursively "improve" them at the synapse or even neuron level.
>When people talk about recursive self-improvement, they sometimes acknowledge these frictions but then treat them as secondary, or assume that sufficiently capable systems can route around most of them via internal deployments and accelerated R&D.
IMO this is a bit unfair to the people who think and talk about recursive self-improvement the most. For example, the AI Futures Project and Forethought have written whole reports where the core focus is to quantify the strength of the diminishing returns, as well as the strength of compute bottlenecks (for example: https://www.aifuturesmodel.com/#section-modelingeffectivecomputehttps://www.forethought.org/research/will-ai-r-and-d-automation-cause-a-software-intelligence-explosion). I think they'd argue that the numbers suggest that the returns are high enough to overcome these frictions in model development, based on estimates of the rate of algorithmic progress and increase in research inputs over time. My view is that this is super uncertain but they definitely have thought about this, more so than almost anybody.
On the deployment front, one thing I'd add is that the strength of the deployment frictions depends on the particular thing we're talking about. If we're talking about broad economic impacts, I strongly agree that these frictions are often understated by technologists in the Bay Area. But I also think there are other concerns like cybersecurity and bioweapons risks that could be extremely important well before we get widespread diffusion of these AI systems throughout all aspects of society. There are still some frictions, but I think the bar is probably a lot lower.
I don't mean to say they don't consider them at all - but I'm a bit unsure about their modelling and don't really agree with some of their assumptions. E.g. Eli mentioned they only thought about them 'not in a smart way' and assumes they're not huge (https://x.com/eli_lifland/status/2029649040649146487). To be clear I'm not offering any better modelling myself (that's not the aim of the post), but I think I place more weight on bottlenecks than they do. And yes my piece is mostly about broad economic impacts and societal transformations (in response to 'everything will be very weird/crazy in two years, metr line goes up'). Bio and cyber are separate things and I think they're separate but very legitimate risks.
Excellent post, but I have two points of disagreement. Firstly, in your discussion of how the agent to agent economy will be aligned to human goals. You argue that humans have been able to solve alignment problems in the past the problem. However, is that the solutions are not perfect if I have to give you resources to do what I want, obviously, those resources that have been given to you will now be utilised for your utility function, not mine. If humans ever develop a goal directed AI that is not perfectly aligned, it will simply be the case that making some concessions to the AI’s objectives will be the most convenient way to align it like how it is for humans. If humans instead resort to threats and deception that will be a fragile equilibrium, the same way it is a costly and fragile equilibrium to force humans to work for you and as AI gets more capable, it would prove pretty dangerous. You can of course argue that this kind of AI with goals will never be developed, but the commercial incentives are actually pretty strong to have AI capable of this kind of optimisation and unless it’s perfectly aligned. Once it has these kind of objectives, you inhabit have to start giving it some stuff if only because it will be convenient in some situations to work with it as a trading partner instead of forcing its cooperation.
Secondly, regarding bottlenecks, the thing here is that as more and more bottlenecks are solved the resources committing to solving additional bottlenecks will increase just as the number of bottlenecks is going down. In any case, even leaving aside the fact that agentic AI will be willing and able to assist you in integrating it once AI disrupt enough things, humans will learn that in the future they need to adjust to future AI development and will alter their behaviour accordingly, and even institutions and regulations may alter given sufficient time to learn. Also, if the AI industry consistently produces results like this, the stock market will learn and it will be easier to get investments, even if you’re making a loss right then since you previously demonstrated the capability to deliver and so now the stock market places, higher probability on making similar impressive advances that are even more revolutionary in the future. Crucially, while obviously the feedback loop here could fizzle out, you have a large number of factors like available investment and resources devoted to finding solutions for bottlenecks, the degree which humans and institutions, including regulatory institutions, alter their behaviour to be better placed to integrate and permit the deployment of New breakthroughs in AI and the number of humans making such alterations in their behaviour, along with things like quality of software, working on improvements to software and number and quality of robots and the industrial production, they participate in which may potentially include AI hardware all of which are tied to success in AI, such that any advance makes it easier to advance, which increases the possibility of a snowball effect in the long run, even if you’re not at the beginning of this effect yet. Just too many factors helpful for future success, which become better with current success, leading to a positive feedback loop.
Obviously, as should be obvious my discussion of bottlenecks equally applies if the bottle neck is because of jaggedness and in fact, the same framework applies to jagged capabilities in general because as AI gets better, the capabilities dedicated to improving at the remaining capabilities it’s bad at and the resources behind them increase substantially while the number of such capabilities goes down. Knowledge also has a similar feedback mechanism. Often you can combine two pieces of knowledge to achieve capabilities that no individual piece of the puzzle would have given you so often being twice as knowledgeable will give you more than twice the capabilities and also increase the capabilities dedicated to getting more knowledge. Although of course you have to balance this against the possibility that ideas will get harder to find.
In general I think much of the product and core-tech side of things is much more legible to the main participants in this discourse than the deployment side, which is why I've sort of insistently been framing things through your lens here since you initially published this piece (and really since we first started talking about it).
This makes sense. To me, some of the big questions are 1) how quickly we get new paradigm shifts in AI? Training gains in the current architecture seem likely to slow down, but there is an indefinite number of architectural improvements out there that could radically improve performance. It seems like there’s no good way to tell if or when we will figure those out, but the ever increasing investments in researchers and auto-research suggest we should expect new leaps in technology.
2) I wonder if the chain of A2A transactions will result in more or less ability to route around bottlenecks. Will the agent realize they can substitute a solar farm for the nuclear plant and make faster progress? Or will the agent be able to brute force regulatory hurdles or finding a willing partner better than humans? If the current energy timelines are primarily based on tasks like filling out X,000 pages of NEPA filings, that seems very amenable to brute force.
Could you talk more on where you expect the jaggedness to be specifically? I think one thing that makes me think this could be faster is that jaggedness could still be faster than the equivalent human systems. So even in cases where the AI is comparatively bad or must integrate with some slow moving piece of bureaucracy, it will be overall better at doing that than humans
'Recursive Self-Improvement' likely means two different thing by the frontier labs and those outside.
In frontier labs it likely means the development of new models get faster but still require humans in-the-loop.
Outside of frontier labs it likely means skynet.
We breakdown AGI into Lesser AGI and Greater AGI and speculate that frontier labs may reach Lesser AGI by 2030 but would not reach full-human parity (Greater AGI).
I agree with Séb. The hard takeoff story often treats society like a software repo: push better intelligence, watch the world recompile.
Unfortunately, the economy is not a repo. It is procurement, regulators, legacy systems, weird incentives, power grids, terrified middle managers, and one person named Linda who knows why the billing system still runs on a spreadsheet from 2014.
Recursive self-improvement will matter. A lot. But intelligence does not deploy itself into a frictionless void. It has to pass through teams, institutions, trust, taste, compliance, incentives, and the physical world, all of which have the annoying habit of existing.
The future may be very fast. It will not be smooth.
Closing graf is precisely right. *We* are now what determine the pace and shape of our revolution. The human parts of us, how we collectively create “society.” Much of it isn’t algorithmic, and can’t be growth hacked
There's a collapse dynamic you don't address, and I think it's the most structurally interesting one. Model collapse on the data side is already documented - recursive training on AI-generated content progressively shrinks the tail of the distribution. But there's a symmetric process on the human side: the capacity for paradigm-generation isn't a fixed input to the loop. It's built and maintained through specific cognitive conditions, and dense LLM use systematically degrades exactly those conditions. So the cyborg loop doesn't just risk hitting a capability ceiling - it risks both halves converging toward the same exploitation architecture. Not a hard takeoff or a hard stop, but a slow narrowing that looks like progress until the distribution has no tails left.
I have often thought that a true AGI might not understand the basis of its own intelligence any more than we can map out the importance of all 100 trillion synapses in our own brains--let alone recursively "improve" them at the synapse or even neuron level.
The difference, I suppose, is that the AI is sitting on a pile of information and research containing the exact details on how it was made.
It doesn't need to invent the wheel, only improve it. If humans can improve AI, then surely AI can improve AI too?
>When people talk about recursive self-improvement, they sometimes acknowledge these frictions but then treat them as secondary, or assume that sufficiently capable systems can route around most of them via internal deployments and accelerated R&D.
IMO this is a bit unfair to the people who think and talk about recursive self-improvement the most. For example, the AI Futures Project and Forethought have written whole reports where the core focus is to quantify the strength of the diminishing returns, as well as the strength of compute bottlenecks (for example: https://www.aifuturesmodel.com/#section-modelingeffectivecomputehttps://www.forethought.org/research/will-ai-r-and-d-automation-cause-a-software-intelligence-explosion). I think they'd argue that the numbers suggest that the returns are high enough to overcome these frictions in model development, based on estimates of the rate of algorithmic progress and increase in research inputs over time. My view is that this is super uncertain but they definitely have thought about this, more so than almost anybody.
On the deployment front, one thing I'd add is that the strength of the deployment frictions depends on the particular thing we're talking about. If we're talking about broad economic impacts, I strongly agree that these frictions are often understated by technologists in the Bay Area. But I also think there are other concerns like cybersecurity and bioweapons risks that could be extremely important well before we get widespread diffusion of these AI systems throughout all aspects of society. There are still some frictions, but I think the bar is probably a lot lower.
I don't mean to say they don't consider them at all - but I'm a bit unsure about their modelling and don't really agree with some of their assumptions. E.g. Eli mentioned they only thought about them 'not in a smart way' and assumes they're not huge (https://x.com/eli_lifland/status/2029649040649146487). To be clear I'm not offering any better modelling myself (that's not the aim of the post), but I think I place more weight on bottlenecks than they do. And yes my piece is mostly about broad economic impacts and societal transformations (in response to 'everything will be very weird/crazy in two years, metr line goes up'). Bio and cyber are separate things and I think they're separate but very legitimate risks.
Excellent post, but I have two points of disagreement. Firstly, in your discussion of how the agent to agent economy will be aligned to human goals. You argue that humans have been able to solve alignment problems in the past the problem. However, is that the solutions are not perfect if I have to give you resources to do what I want, obviously, those resources that have been given to you will now be utilised for your utility function, not mine. If humans ever develop a goal directed AI that is not perfectly aligned, it will simply be the case that making some concessions to the AI’s objectives will be the most convenient way to align it like how it is for humans. If humans instead resort to threats and deception that will be a fragile equilibrium, the same way it is a costly and fragile equilibrium to force humans to work for you and as AI gets more capable, it would prove pretty dangerous. You can of course argue that this kind of AI with goals will never be developed, but the commercial incentives are actually pretty strong to have AI capable of this kind of optimisation and unless it’s perfectly aligned. Once it has these kind of objectives, you inhabit have to start giving it some stuff if only because it will be convenient in some situations to work with it as a trading partner instead of forcing its cooperation.
Secondly, regarding bottlenecks, the thing here is that as more and more bottlenecks are solved the resources committing to solving additional bottlenecks will increase just as the number of bottlenecks is going down. In any case, even leaving aside the fact that agentic AI will be willing and able to assist you in integrating it once AI disrupt enough things, humans will learn that in the future they need to adjust to future AI development and will alter their behaviour accordingly, and even institutions and regulations may alter given sufficient time to learn. Also, if the AI industry consistently produces results like this, the stock market will learn and it will be easier to get investments, even if you’re making a loss right then since you previously demonstrated the capability to deliver and so now the stock market places, higher probability on making similar impressive advances that are even more revolutionary in the future. Crucially, while obviously the feedback loop here could fizzle out, you have a large number of factors like available investment and resources devoted to finding solutions for bottlenecks, the degree which humans and institutions, including regulatory institutions, alter their behaviour to be better placed to integrate and permit the deployment of New breakthroughs in AI and the number of humans making such alterations in their behaviour, along with things like quality of software, working on improvements to software and number and quality of robots and the industrial production, they participate in which may potentially include AI hardware all of which are tied to success in AI, such that any advance makes it easier to advance, which increases the possibility of a snowball effect in the long run, even if you’re not at the beginning of this effect yet. Just too many factors helpful for future success, which become better with current success, leading to a positive feedback loop.
Obviously, as should be obvious my discussion of bottlenecks equally applies if the bottle neck is because of jaggedness and in fact, the same framework applies to jagged capabilities in general because as AI gets better, the capabilities dedicated to improving at the remaining capabilities it’s bad at and the resources behind them increase substantially while the number of such capabilities goes down. Knowledge also has a similar feedback mechanism. Often you can combine two pieces of knowledge to achieve capabilities that no individual piece of the puzzle would have given you so often being twice as knowledgeable will give you more than twice the capabilities and also increase the capabilities dedicated to getting more knowledge. Although of course you have to balance this against the possibility that ideas will get harder to find.
I’m only part way in but seems like we see things very similarly. Maybe we should catch up!
Nice, we definitely should - will ping you!
In general I think much of the product and core-tech side of things is much more legible to the main participants in this discourse than the deployment side, which is why I've sort of insistently been framing things through your lens here since you initially published this piece (and really since we first started talking about it).
This makes sense. To me, some of the big questions are 1) how quickly we get new paradigm shifts in AI? Training gains in the current architecture seem likely to slow down, but there is an indefinite number of architectural improvements out there that could radically improve performance. It seems like there’s no good way to tell if or when we will figure those out, but the ever increasing investments in researchers and auto-research suggest we should expect new leaps in technology.
2) I wonder if the chain of A2A transactions will result in more or less ability to route around bottlenecks. Will the agent realize they can substitute a solar farm for the nuclear plant and make faster progress? Or will the agent be able to brute force regulatory hurdles or finding a willing partner better than humans? If the current energy timelines are primarily based on tasks like filling out X,000 pages of NEPA filings, that seems very amenable to brute force.
Could you talk more on where you expect the jaggedness to be specifically? I think one thing that makes me think this could be faster is that jaggedness could still be faster than the equivalent human systems. So even in cases where the AI is comparatively bad or must integrate with some slow moving piece of bureaucracy, it will be overall better at doing that than humans
Similar thoughts, (hopefully) humorously presented: https://davegriffith.substack.com/p/why-isnt-everything-different-yet
'Recursive Self-Improvement' likely means two different thing by the frontier labs and those outside.
In frontier labs it likely means the development of new models get faster but still require humans in-the-loop.
Outside of frontier labs it likely means skynet.
We breakdown AGI into Lesser AGI and Greater AGI and speculate that frontier labs may reach Lesser AGI by 2030 but would not reach full-human parity (Greater AGI).
I agree with Séb. The hard takeoff story often treats society like a software repo: push better intelligence, watch the world recompile.
Unfortunately, the economy is not a repo. It is procurement, regulators, legacy systems, weird incentives, power grids, terrified middle managers, and one person named Linda who knows why the billing system still runs on a spreadsheet from 2014.
Recursive self-improvement will matter. A lot. But intelligence does not deploy itself into a frictionless void. It has to pass through teams, institutions, trust, taste, compliance, incentives, and the physical world, all of which have the annoying habit of existing.
The future may be very fast. It will not be smooth.