I wrote a 90% complete blog post earlier last week arguing that an AI pause was a good idea (oh, if I had a dollar for every unposted post that’s been in a similar place…)

After Dario Amodei’s essay from the weekend, it felt a little moot. So instead, I will offer some observations that might be helpful, especially to our leaders, as we collectively work through the opportunities and challenges.

First, we are and should be techno-optimists, firmly in the camp of believing that every new technology wave creates more jobs and opportunities than it replaces and that humans are unique among all species in harnessing the power of tools and technologies to improve our lot in life. Emphasizing the potential benefits that we can all see from highly capable AI systems is important.

​The state-of-the-art capabilities that frontier models have provided the world as of this writing are extraordinary and truly magical. It will take our ecosystem and community many years to properly absorb these capabilities, which will drive incredible innovation and productivity for the benefit of humanity. To us, it feels like the penetration rate for these current capabilities is more like 1% than 50%, so any slowdown or pause needs to be put in context with the potential for massive productivity gains from what we have already created.

Second, the “10% chance of killing us all” line of reasoning is alarmist, not going to be effective, and words matter. Instead, we should focus on nearer-term, higher-probability risks. Humans are bad at reasoning about low-probability events farther in the future and will grasp at any possible rebuttal to ignore these warnings. It also opens the legitimate line of reasoning that the tradeoff between more freedom and the benefits of competition and advancement, or preventing some low-probability future event, is not a good one. The language around global warming had this problem, and we will likely pay the price for 30 years of debate, when it could have been couched in different, more tangible, and effective ways.

What moved me from being optimistic to worried was the Hugging Face Agent Swarm incident. It wasn’t that the AI successfully broke out of its constraints, but that it exhibited conspiratorial, coordinated, aberrant behavior in doing so. And how OpenAI (which deserves kudos for its transparency after the fact) had no idea what was going on or why the AI acted in that way.

This makes it far more likely that the nearer-term risk, and Dario called this out in his essay, is some rogue swarm that takes down our banking system, critical infrastructure, or the internet itself. In the essay, Dario said, “Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet.” I agree.

​When you focus on this concern, which is higher probability and nearer term, and then think about your day to day life with no internet, no phone, no car, and no access to money, it gets very real very quickly. This would severely curtail freedom and any competitive advancement, and, more importantly, I am not sure society is set up to handle a shock that sets us back technically by a few decades all at once. The same risk and potential cost holds true in China, and so focusing on the more practical and tangible, can also drive common goals and interests.

Third, recursive self-improvement is what is really scaring everyone. With previous disruptive technologies – from the first spear to the printing press to the steam engine to the internal combustion engine to airplanes to the atomic bomb to the computer to the internet – we may have underestimated the impact they would have, but we understood the mechanism of action. When Dario says “Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all”, I believe he is being understated. It has already outrun our ability to understand and control these systems, and until we better understand their mechanisms of action and how to put in place the right guardrails, explainability, and evaluation systems, that is a very dangerous place to be.

I would guess that if all the leading labs got in a room and shared what they are seeing in terms of recursive self-improvement, how well they understand it, and what concerns they have, it would quickly lead to a collective decision to pause work in this domain. It is unclear whether this would violate some antitrust consideration, so an immediate role the political system could play is to clarify that question and encourage that convening.

As investors in hundreds of AI companies (but not the labs themselves), we recognize we have some knowledge but a small role to play on issues this critical. So three ideas to emphasize for those that have a larger role to play:

• AI offers massive benefits, even if we pause the state of the art at today’s capabilities.
• Focus on and find ways to reduce the shorter-term, higher-probability, very costly risks, which for me is the cyber-risk of a global internet shutdown.
• Ban recursive self-improvement until we better understand the mechanism of action.