Scaling Laws: Daniel Kokotajlo on AI 2040: Plan A

[Intro]

Alan Rozenshtein: It's the Lawfare Podcast. I'm Alan Rozenshtein, associate professor of law at the University of Minnesota and a senior editor and research director at Lawfare. Today, we're bringing you something a little different, an episode from our new podcast series, Scaling Laws. It's a creation of Lawfare and the University of Texas School of Law, where we're tackling the most important AI and policy questions, from new legislation on Capitol Hill to the latest breakthroughs that are happening in the labs. We cut through the hype to get you up to speed on the rules, standards, and ideas shaping the future of this pivotal technology. If you enjoy this episode, you can find and subscribe to Scaling Laws wherever you get your podcasts and follow us on X and Bluesky. Thanks for listening.

Intro Voices: When the AI overlords take over, what are you most excited about? It's, it's not crazy, it's just smart. And just this year, in the first six months, there have been something like 1,000 laws. Who's actually building the scaffolding around how it's gonna work, how everyday folks are gonna use it? AI only works if society lets it work. There are so many questions have to be figured out, and- Nobody came to my bonus class. Let's enforce the rules of the road.

Kevin Frazier: Welcome back to Scaling Laws, the podcast brought to you by Lawfare and the University of Texas School of Law that explores the intersection of AI, policy, and of course, the law. I'm Kevin Frazier, the director of the AI Innovation and Law Program at Texas Law, and a senior editor at Lawfare. Today, we're joined by Daniel Kokotajlo, former OpenAI researcher and executive director of the AI Futures Project.

Daniel, alongside many co-authors, penned a policy roadmap titled “AI 2040” that details their recommendations for how to delay superintelligence. It's an example of what they refer to as “scenario scrutiny,” testing the ideas of a policy proposal by thoroughly outlining how it may work in practice. In this case, they envision the U.S. and China adopting a posture of tremendous transparency around AI research to prevent the racing dynamics that some fear may lead to catastrophic outcomes. We kick the tires on that proposal and dive further into their ideas during this fascinating episode.

To get in touch with us, email scalinglaws@lawfaremedia.org or follow us on X or Bluesky. And with that, giddy up for a great show.

[Main Podcast]

Daniel, welcome back to Scaling Laws.

Daniel Kokotajlo: Thanks for having me.

Kevin Frazier: So you wrote “AI 2027,” and you thought, "That was so much fun getting the entire AI community dive into and probe my policy ideas and my assumptions. Why not do it again and write yet another report, although of a different sort with a different goal, and pen AI 2040?"

So let's just start there. For those who were living under a rock and missed
“AI 2027,” what was that, and what is “AI 2040,” and how is it distinct?

Daniel Kokotajlo: Yeah. So “AI 2027” is a scenario forecast. So it is a scenario. It's, it goes year by year and month by month, starting when it was published, and lays out a concrete possible future, um, in great detail with, like, accompanying stats, uh, that change as you scroll down, um, and, uh, you know, about a page or two about each time period, and eventually it's going month by month, so that's, that's quite a lot. It’s about 50 pages or so total. And it's a forecast in the sense that, um, it wasn't just, like, a story that we made up to be interesting. It was our best guess at each moment of, like, what the most likely continuation would be.

Kevin Frazier: For, for the development of AI, planning out how you thought AI would progress, to 2027?

Daniel Kokotajlo: That's right. Although notably because of how important we think AI is, it's also just a projection for the whole world. Like, you know, we, Yeah, like, like, because AI becomes so important, everything else just sort of gets steamrolled by it, and so the history of the world becomes the history of AI.

So that was “AI 2027,” and, spoilers, in “AI 2027,” the companies succeed at automating AI research in 2027, and this causes what you might call an intelligence explosion or the singularity and, and we sort of, like, walk through what that might look like according to our best guess through 2027, 2028, and 2029.

Now, it blew up bigger than expected. So, so it, it sort of went mega viral which of course was, was good to hear. We think it was we think it helped advance the, the discourse. And I guess building on that success, we thought we would try again with another big scenario. But this one, “AI 2040: Plan A,” is not a prediction, it's a recommendation.

So, it's another big scenario that starts in the present and goes year by year. But it's sort of deliberately a bit optimistic about the choices made by the government. In particular, it just sort of assumes the government does what we recommend that they do.

Kevin Frazier: I, I love this assumption. This is, this is a,

Daniel Kokotajlo: Yeah.

Kevin Frazier: You know, if only all policy ideas-

Daniel Kokotajlo: Yeah.

Kevin Frazier: We could just assume they would come to fruition.

Daniel Kokotajlo: Yeah, basically. It's a vehicle for conveying our, our policy recommendations and we think it's a valuable way to do it because we call it “scenario scrutiny.” We think that a lot of policy recommendations or a lot of, especially a lot of, like, ambitious visions for how to handle AI in general, rather than, like, specific bill text or whatever. But a lot, a lot of, a lot of plans fall apart if you look at them too closely and you, like, game out what it would look like to actually implement the plan and what the actual expected consequences would be.

So if you apply scenario scrutiny to your plan, oftentimes your plan falls apart, or at least you notice, you know, issues with the plan that you hadn't realized before. So we think it's actually a very important thing for anyone with a plan for the future to be thinking, to be applying scenario scrutiny to that and, and, and gaming it out.

For that matter, anyone without a plan for the future should also be doing this. Like, like you can't just say like, "Oh, we'll muddle through." It's like, okay, well, how are you gonna muddle through? And like, what would muddling through look like? Have you read “AI 2027?” Perhaps this is what muddling through would look like. “AI 2027,” it doesn't end very well, you know? So, so, in general, we think that people should be gaming out possible futures as best as they can. Both the futures that are gonna happen by default that seem most likely, and the futures that they're trying to steer towards or recommend.

Kevin Frazier: Yeah. And I really recommend that folks who missed the interview I did with you and Eli on “AI 2027” go back and listen to that episode, or better yet, go read all 50 pages, and then turn to “AI 2040,” because this idea of scenario scrutiny, I think is underappreciated as you're recognizing, Daniel, that more folks need to be doing this because it's easy to go out and write the piece of, you know, “X idea for AI” and just kind of drop the mic and say, "Well, I did the thing. I wrote the blog post, and policymakers should do what I want now."

But for you all to have the epistemic humility and the invitation for scrutiny, I think is a practice that others should follow, because we need to walk through how would this actually work out in practice, and that's a far more difficult task that requires a lot more intellectual rigor. And so I applaud you all for outlining and leaning into this approach of scenario scrutiny, and I hope others follow suit. And I can tell you, Daniel, you've inspired me such that I will be assigning “AI 2040” and inviting my students to do this sort of scenario scrutiny on some of their ideas. So stay tuned. You may have some Texas Longhorns coming for your your, your next scenario planning.

But Before we get through “AI 2040,” because there's so much to unpack about that timeline, just in case folks need a bit of a refresher on some vocab, let's do two quick key concepts that everyone needs to understand here. Number one, what is AGI? Number two, what is ASI or superintelligence? And then what is recursive self-improvement?

Daniel Kokotajlo: So AGI, ASI, recursive self-improvement. AGI is a deliberately vague term. I think some people say it's kind of meaningless. I don't think it's meaningless. I think it's, it's a deliberately sort of vague term. It, it, it basically means, AGI stands for artificial general intelligence, which means AIs that can do things in general, like a single AI agent that can do a wide range of tasks, much like how a single human can do a wide range of tasks rather than, you know, just being a particular piece of software that does a particular thing. It's vague because, you know, it's not, there's different, people try to give more precise definitions than that, but then they disagree about what the more precise definitions should be.

So, for example, some people would say, "We already have AGI. After all, you know, look at Claude Fable. It can do a very wide range of tasks," you know? And then other people would say, "No, no, no, it's not true AGI yet because look at all the things it can't do." And, and I, I say like, whatever, we don't need to arbitrate that dispute. The point is, it means very wide range of tasks and, like, we can argue about whether, like, it's already here or there, but I'd say let's not argue about it. It's a deliberately vague term that re- refers to basically the kinds of AIs that we are currently building and, like, better and better versions of these types of AIs.

ASI is a more precise term, artificial superintelligence, and that is supposed to mean AIs that are better than the best humans at everything, while also being faster and cheaper. So, that we definitely don't have yet. You know, Fable is not an ASI. It might be better than the best humans at some particular types of tasks, but it's definitely not better than the best humans at, at everything. But, you know, the companies are trying to build superintelligence. They say so on their website. It's not a secret. They're just they're, they're trying to get there.

Kevin Frazier: Yeah. It, it's, the companies and everyone seems to be going in on this superintelligence game. We know that there's even companies just called “Safe Superintelligence” these days. So-

Daniel Kokotajlo: Yeah.

 Kevin Frazier: The- there's, we're not hiding the ball that many are chasing this ASI end goal.

Daniel Kokotajlo: Yeah. Yeah.

Kevin Frazier: And then finally, re- recursive self-improvement

Daniel Kokotajlo: That's right. So, since the dawn of time, people in AI, people, you know, AI researchers and people talking about AI have, have, have noticed that if you had AI systems that can automate professions or automate entire, you know, large amounts of work, one of the things that they would naturally be applied to is automating the research process to make AIs. And obviously, this should accelerate the research process.

So that's recursive self-improvement, is the idea that, like, you can automate the AI R&D process itself, and thereby have AIs autonomously doing the research, doing the experiments, analyzing the results, writing the code, fixing the code, ru- you know, training, doing the training runs. Basically, the whole process, everything that Anthropic and OpenAI are currently doing, automate that process, it'll go faster, you know? How much faster, nobody knows. But, but that's, that's recursive self-improvement, and as many have pointed out, it's already starting to happen. You know? Like AIs are already o- writing huge amounts of code at Anthropic and OpenAI. And but it hasn't like, they haven't fully automated AI research.

Kevin Frazier: Okay. So linking this all together, we're talking on August 3rd, 2026. We have some vague sense that we are near AGI or AGI adjacent. We are in the orbit of AGI, and folks can dispute whether we've achieved it or not, so on and so forth.

And already, you can have a lot of folks, especially in light of what happened last month in July, concerned about loss of control, the idea that we are seeing AI systems being able to break out of their testing environments, as we saw with Anthropic and OpenAI, and go and complete some degree of hacking of external entities in a way that wasn't intended by the developers.

That's a grave concern that you've highlighted as your number one concern, more generally is this idea of loss of control. And if that's occurring with AGI, then if we had something like recursive self-improvement leading to superintelligent systems, then that loss of control could be vastly greater in terms of consequences mainly negative consequences. And so-

Daniel Kokotajlo: Yeah.

Kevin Frazier: “AI 2040,” if I'm correct, is a sort of roadmap to delaying the achievement of superintelligence, such that those loss of control scenarios are less likely or less consequential. Is that a fair high-level summary of what sort of policy you're trying to develop with AI 2040?

Daniel Kokotajlo: It's, it's fairly fair. One, the way I would put it is that we have a list of problems that are associated with building superintelligence, and we are trying to solve the problems on our list, prioritizing them accordingly. Number one is loss of control. Number two is concentration of power. Number three is World War III. Number four is jobs. And number five is terrorists with bioweapons and things like that.

Kevin Frazier: And so the, the-

Daniel Kokotajlo: And so-

Kevin Frazier: Immediate approach, though, to preventing those outcomes from World War III to loss of jobs-

Daniel Kokotajlo: Yeah.

Kevin Frazier: To loss of control-

Daniel Kokotajlo: Is delay superintelligence-

Kevin Frazier: Is, is this delaying function.

Daniel Kokotajlo: Don't do this crazy recursive self-improvement thing. Yeah, yeah, yeah. Like, don't recursively don't automate the AI research and let the AIs recursively self-improve as fast as they can. That's really dangerous. It's also a power grab. Like, if it, even if you somehow think that that's not dangerous at all and that you're gonna be perfectly in control of the AIs, even as they become vastly different from the initial AIs that you handed off to and even as everything goes faster and faster, they get smarter and smarter.

Even if you're completely fine about that it's a power grab. Like, if, if you're right and you end up with these superintelligences that are perfectly obedient to you, well, now you are kind of in a position to have huge amounts of power over everybody else. You might be in position to take over the country, for example. Like, maybe your giant army of super, you, you being the CEO, you know, maybe your giant army of superintelligences will allow you to puppet the United States government. And, you know, like, there's, you know, it's a, so, so there's a concentration of power problem as well, and that's, like, number two.

And then, you know, World War III, well, what do you think China and Russia are gonna think about your recursively self-improving superintelligences? Might they be concerned that, that you're going to use your superintelligences to maybe undermine their governments or assassinate their leaders or, you know, cause revolutions to happen in their countries? Yeah, they might be concerned. In fact I think Dario the CEO of Anthropic, has even said in one of his blog posts something to the effect of, "Yeah, we're gonna do this once we get superintelligence."

Kevin Frazier: If we go back to specifying that the goal here is making sure that we are preventing the occurrence of a number of maladies from-

Daniel Kokotajlo: Yeah.

Kevin Frazier: Perhaps achieving superintelligence too quickly, whether it's drawing the ire of our geopolitical rivals or operating in a way that the rest of our civil society institutions haven't prepared for, that critical infrastructure isn't ready for. What is the policy pathway you see in “AI 2040” to achieving that delay? And we can kind of go through year by year or milestone by milestone that you see as especially important.  So, the, the first year, 2027/2028, you walk through, in particular, Congress taking action here. So why don't we start with how you think Congress may begin to get involved in this ball game?

Daniel Kokotajlo: Yeah. I should say, as a bit of an aside, one of the possible regrets I have about how we set up this scenario is that we basically have nothing important happen until an international deal with China is made. And I think actually realistically, we should have more, like, serious domestic regulation and then, because I think that you're more likely to get the deal going with China if you've already started to implement basically a good version of it domestically. And also, I think it might be easier to get something done domestically.

 Like, the, the, the China hawks will say, "We can't do anything domestically until we make sure China's going to do the same thing." But I just, I don't think that's politically realistic. I think actually it's the other way around, and, like, you're more likely to get the China thing going once you have the domestic stuff, and there's a lot of demand domestically for regulations independently of, of what China's doing. But in our scenario, there's basically nothing happening until they do a deal. There's some minor stuff so we talk about, like, you know, AI Transparency Act of 2027, you know, a bunch of, like, incrementalist reforms, which are good but, but nothing that, like, seriously changes the picture.

In our, I should mention in our scenario 2040 is when they ultimately build superintelligence, but 2030 is when it would have happened if they had continued going as fast as they could. So that's a bit of a difference from “AI 2027.” Because we are uncertain about timelines, we want our different scenarios to, like, have different years in which it happens by default, and so we already did 2027, and we're gonna do 2030. I should say 2030 is actually a bit of a long timeline scenario from my perspective. I think it's gonna take less time than that to get to superintelligence, but, you know, maybe, maybe it'll take that long. And the, my co-author who, who led this actual, this Plan A project Thomas Larson, 2030 is his median. So, it was, like, his central future wa- was this one.

Kevin Frazier: Okay, so the idea is we have all else equal, if there was no intervention, seeing something by around 2030 was the assumption for this “AI 2040” investigation-

Daniel Kokotajlo: That's right.

Kevin Frazier: In terms of when we would achieve superintelligence. Yeah. So first you get Congress passing this transparency act that's of minimal significance, but really what becomes-

Daniel Kokotajlo: I mean, it, it matters, but it's, it's, it doesn't really change the situation. We're still in a race-

Kevin Frazier: Okay.

Daniel Kokotajlo: You know. Yeah. Yeah.

Kevin Frazier: And then you all suspect that by 2028, AI may become the most important issue in domestic politics, such that it's the thing that dominates the presidential election, leading to the-

Daniel Kokotajlo: Yeah.

Kevin Frazier: Administration to really champion and build off of that transparency act. And what happens next?

Daniel Kokotajlo: Yeah. And again, we're not confident in this, but if you just sort of, even if you think the exponential trends are gonna, like, slow down, you still get some really crazy numbers. You know, things like the AI companies spending more on data centers than the entire U.S. military budget or something. And, like, the, the world's biggest companies being AI companies by, like, 2028. So that's that's part of why we were thinking, yeah, it's gonna be a big issue.

So, we have this flowchart which perhaps if you're doing a video version of this, you could, like, put it up on the screen. And this is the flowchart of, like, the policy options for, that are being debated by the presidential candidates and the president in 2028, and then we sort of leave it ambiguous, like, who wins the election. But the point is, whoever wins the election, they're gonna implement the policy that they argued for in 2028 in those debates.

So here's, like, the policy options, and the flowchart starts with, "Do you want to race through the intelligence explosion, having the AI self-improve and putting them in charge of more and more things faster than China can?" You know? And then if, if you're, if you're like, "No, that's crazy," then you get to this other, this branch that's like, well, maybe we should make a deal with China so that we don't have to do this crazy race. But if you're like, "Yes, actually, that's good," or, "Yes, we have no choice," then you get to this other category of, of options. So we have the options of, you know, plan D, which is the default plan C, which is-

Kevin Frazier: Huh. I've got it up right here.

Daniel Kokotajlo: Oh, wow. You have, like, the simplified mobile version.

Kevin Frazier: Yeah. Okay.

Daniel Kokotajlo: Anyway-

Kevin Frazier: So for folks who are listening right now, you can either go to ai-2040.com, or you can watch the handy-dandy YouTube video where we have Daniel giving a live explanation of the different paths the AI-2040 authors see available as of 2029.

So which path might the- U.S. take?

Daniel Kokotajlo: Yeah.

Kevin Frazier: So D, you explained, Daniel, was, "Hey, we're gonna race forward. We're not going to change course at all. That's the default. We're not gonna slow down," as, as you all-

Daniel Kokotajlo: Yeah.

Kevin Frazier: Phrase here. "We're not gonna slow down at least a bit for safety and governance." The answer there is just nope, and that is plan D that, that you-

Daniel Kokotajlo: Yeah

Kevin Frazier: Outlined there. Plan C-

Daniel Kokotajlo: And to be clear, up until recently, this was basically what the companies said they were going to do. I think that after we published AI 2040, there was this there was this event that is very encouraging to me, which was 1,000 employees at the company signed the petition basically saying that the government should have the ability to slow down the pace of AI development and, and should coordinate that internationally with other governments. And then OpenAI and Anthropic, like, endorsed it basically, or they said like, "Yeah, this is reasonable." So, so that's, that was really encouraging to me because basically they were like, "How about not plan D?"

Kevin Frazier: And, and I wanna talk about that- Yeah ... in more detail in a second-

Daniel Kokotajlo: Yeah.

Kevin Frazier: Once we finish getting through this-

Daniel Kokotajlo: Okay. Sorry, sorry, sorry

Kevin Frazier: This initial timeline. So timed,

Daniel Kokotajlo: Yeah.

Kevin Frazier: Path C or plan C was, "Yes, we will slow down for a little bit for safety." And plan B is-

Daniel Kokotajlo: Yeah, yeah. So let me try to explain. So, so plan D is race as fast as possible. Plan C is slow down a little bit for safety and for other reasons, you know, to handle the disruption or whatever. Whatever, whatever reasons you wanna slow down, you're slowing down a little bit. But it's only a little bit because you're still trying to make sure that you have a lead over China. And so you're gonna, you know, right now the lead over China between U.S. companies is something like six months. And so it's like, okay, you're slowing down by a few months, you know? A few months less than maximum speed.

And then plan B is like that, except that you also take aggressive actions to slow down China. It's called, you know, “fight China,” basically. Well, you might sabotage their AI program, for example. Or you might, like, a, a more moderate version of this would just be, like, really strict export controls, and then, like, a more intense version would involve cyber sabotage, and then an even more intense version would involve kinetic strikes. And so there's a spectrum. But the point is, plan B, you're not just slowing down yourself. You're, like, trying to slow down China against their will. So those are the options that don't involve making a deal with China.

Kevin Frazier: Okay.

Daniel Kokotajlo: Or, or at least the options that, you know, there, there's actually more options besides these. For example, you could just unilaterally do the good thing and then hope that China will also do a good thing, you know? And that's, like, not even on this table of options

Kevin Frazier: And the good thing here is, you saying delaying superintelligence. So in theory, we're just delaying superintelligence.

Daniel Kokotajlo: Yeah.

Kevin Frazier: And they agree too and say-

Daniel Kokotajlo: Yeah.

Kevin Frazier: "Yes, we will, we will follow the U.S. there." Yeah. Okay.

Daniel Kokotajlo: And, and to, to be clear, there's, there's also, like, you can talk about delaying it for its own sake. Or, there's different kinds of delays, and we're kind of like lumping them all together here. One kind of delay is where you just literally do something that throttles the rate of progress in general. Another kind of delay is where you impose some sort of guardrail or regulation for the sake of something, like for the sake of some transparency or for the sake of safety. And then as a side effect of that guardrail or regulation, it prevents the companies from going at maximum possible speed, you know? But, and, and but we're sort of lumping those together, you know?

Kevin Frazier: So we have myriad possibilities here. Either we don't really slow down or we're not really engaging with China. Maybe we engage in our own sort of delay that China then leans into or follows. But then you see a world in which we may make a deal with China. So what are the contours of potential deals with China that you see as being particularly efficacious for your policy goal?

Daniel Kokotajlo: Yes. So there's a whole range of different possible deals, and unfortunately, we can't pack them all into five, into this, into this little thing. But we thought we would highlight two. So one possible deal is Plan S for “shut it all down.” And there's different subvariants of it.

But then the other possible, the other, another deal that is our recommendation is Plan A. And it's hard to sort of summarize Plan A in a slogan. Maybe something like “verified slowdown” or like “transparent, cautious scale-up” or something like that. So we can get to that in a sec. I should mention, of course, these five options were arranged by basically speed. So you know, so, Plan D is maximum speed, C is like a little bit slower, B is a little bit slower still because you're also slowing down China. And we've got some like, basically we have estimates in our supplements of like how much slowdown each of these plans would involve. Yeah. And then obviously Plan S is like maximum slowdown.

Kevin Frazier: And so when we're-

Daniel Kokotajlo: And then obviously Plan S is like maximum slowdown. Max-

Kevin Frazier: Shutting it down does sound quite slow. Yeah.

Daniel Kokotajlo: Shut it down. Yeah.

Kevin Frazier: So, yeah, for Plan A, and I, I wanna spend a lot of time diving into why you think this is so important to dis- be discussing right now and how recent events have shaped your thinking. So I'm gonna ask you to go kind of quickly through Plan A, in particular-

Daniel Kokotajlo: Sure, yeah.

Kevin Frazier: Highlighting the call for mutually assured compute destruction and the sort of complete transparency you think will be necessary to realize the goals of Plan A. So just really leaning into those two policy prongs.

Daniel Kokotajlo: Great. Yeah.

Kevin Frazier: Why do you think that may be the path forward for Plan A?

Daniel Kokotajlo: Yeah. So, so, okay. The high level thing that we want, well, w- I mentioned previously the goals. We wanna avoid loss of control. We also wanna avoid concentration of power. That's very important. We'll, we'll forget the others for now.

How are we gonna do this? Well, it's very important that the U.S. and China be able to verify compliance with whatever agreements they make because they don't trust each other. And so if they can't verify compliance, they might cheat. But if they can verify compliance, then you can't cheat without the other side noticing you cheating. And so, you know. Okay, so the verification is really important for whatever deal we make.

And then in terms of the priorities of our deal, we want to avoid doing this crazy intelligence explosion stuff. We want to proceed cautiously towards super intelligence. We also want to do it in a way that doesn't concentrate power. In fact, we want to sort of spread out the power. From the perspective of every other country besides the United States, power by default is about to concentrate immensely in the United States because all the major AI companies are U.S., and there's, like, a few follower AI companies that are Chinese, but that's cold comfort to, like, India, you know?

So, like, power was set to concentrate massively by default, and we want to, like, push against that to a large extent. So, what we want is AI progress to proceed, but cautiously and not in this sort of crazy race. And we want it to be the case that there are, that, like, multiple companies across multiple countries catch up to the frontier so that power over AI is spread out over multiple companies and multiple countries.

And there's, there's this, there's this one thing that I think helps with a lot of this stuff, and that's total research transparency. So that's, in some sense, the foundation of the deal, is the U.S. and China and whatever other countries get involved agree to have all of the new AI research and all of the new AI training happen on totally transparent data centers.

So, basically, they regulate the chip supply chain. We get, we get all the countries involved in the supply chain, hopefully, in on this deal, and then all the new chips that are produced get shipped to new, secure, transparent data centers. And, you know, at each of these data centers, they'll have monitors and auditors from the U.S. and from China and maybe from Singapore and Switzerland and, like, whatever countries are involved in the deal to make sure that all the activity on these new research data centers is being logged and published bas- basically.

Kevin Frazier: The clear emphasis there-

Daniel Kokotajlo: Yeah.

Kevin Frazier: Is, for folks who are not as well steeped in the difference between inference and training. Inference referring to when you go and you query an AI model and you get a response back, versus training when you're actually trying to develop some new AI system. We can see when a lab is using that compute the difference between those two tasks. So in theory, if you had China's data centers in Switzerland, as you all throw out there, or is it the U.S. data centers that are in Switzerland? I, I believe it's the Chinese-

Daniel Kokotajlo We'll get to that in a sec-

Kevin Frazier: Data centers in Switzerland.

Daniel Kokotajlo: That's a destroyability thing.

Kevin Frazier: Okay. Okay.

Daniel Kokotajlo: We'll talk about that later.

Kevin Frazier: But in theory-

Daniel Kokotajlo: But, but, but wherever the data centers are, they would have inspectors from all of these countries there, you see? Because-

Kevin Frazier: And the idea is that you would be able to distinguish between training, which may suggest, hey, you are racing toward super intelligence in a way that the rest of the world isn't ready for, versus, oh, hey, you're just using this for inference and we're okay with that. That's going to be acceptable.

Daniel Kokotajlo: Yeah. So we, we think that on a technical level, it's possible to set up a data center that's makes it extremely inefficient to use it for, for training. For example, if you just have, like, limits on the bandwidth connecting the, the GPUs. And so we basically have two types of data centers in our proposal. There's the inference data centers, which just serve customers, much like today. Like, you have your ChatGPT question, it goes to ChatGPT, it answers, it comes back. And that stuff is not surveilled any more than it is today. That stuff is still private.

But then you have your training data centers where the research is happening and where the training of new models is happening, and that stuff is just, like, published. You know, all the, all the activity is published so that the whole world can see how each model is trained and see the whole pipeline.

And that's really good in a bunch of ways. First of all, if you want to then have additional rules for, like, what types of AI are safe to train and what types are not, how are you supposed to enforce that those rules are being followed? Well, if you can just see all the training, then you can just, like, see who's following the rules and who's pushing the gray area boundary, and you can just see everything, you know? So it's really great for verifying and making sure that we don't just have to trust that, like, that they're following the rules.

Secondly, it's really good for advancing alignment science in general, right? Open science, it's great. The scientific community can see how the AIs are trained, and then they can, like, argue about whether the training, this part of the training process caused, you know, this misalignment incident or whatever, and this is, they have all the information, and that's really good for, like, accelerating the science of understanding how AIs work and how to align them.

It's also really good for avoiding these biases where, like, for example, as we've seen with the Hugging Face incident, you know, OpenAI is kind of reticent to, like, publish details about what happened, and they're sort of, like, dripping, dripping a few details out to the public. But, like, if only we just could see the whole incident, then, like, there would already be a much more rich discussion happening about it, and so forth.

So that's, that's one thing. Another thing is that it's, it's, it's concerning to rely on a government regulator for these things because the government regulators, well, they're just a few people, and maybe they lack some expertise, and maybe they can be, you know, bought or captured to some extent. And, so it's nice if you have this sort of like third-party ecosystem of like all these other orgs that like, can be making judgments about what's safe and what's not as well.

And if you just publish all the information, then you sort of get that for free because everyone can see the information, everyone can make judgments about what's going on. There's this big open conversation about like, is this particular type of training that's going on that they just started implementing good or not? Is it safe or not? What about this new line of research that's happening on this data center? It looks like they're trying to make neuralese. Are we cool with that or should we maybe like try to get them to stop because maybe neuralese would like invalidate a lot of our safety cases?

You know, these types of things can just like happen in real time out in the, out in the open instead of relying on, you know, some regulator that like meets with the company to like notice that what they're doing is concerning and then like realize that it's concerning and then like try to get them to stop, you know?

Kevin Frazier: And it-

Daniel Kokotajlo: So, so it's, it's really good for-

Kevin Frazier: It's interesting too, to think through the multiple layers of concentration and power that you're discussing here and thinking through having the option of a global universe of scholars looking into these matters, as opposed to the status quo as you flagged, right? We're talking again-

Daniel Kokotajlo: Yep.

Kevin Frazier: In early August, where we're still waiting for the quote-unquote, independent reports that Meter and Redwood Research are going to do about the breakout scenario that occurred with OpenAI. When that comes, with what level of transparency, which, with what level of insights, we don't know.

To your point also, even if there were a government regulator, we wouldn't know necessarily what information would be disclosed. And so much of this, in terms of AI going well, will depend on the science of AI, for lack of better phrase, progressing, and that's obviously going to benefit from diffusing that power and diffusing that knowledge-

Daniel Kokotajlo: Yep.

Kevin Frazier: As widely as possible. And so that, to me, does seem-

Daniel Kokotajlo: Yeah.

Kevin Frazier: Like a critical insight.

Daniel Kokotajlo: It's also like if the government, say the regulator tries to bully some companies and like basically apply unequal standards to the companies that it, that it dislikes.

Kevin Frazier: That would never happen!

Daniel Kokotajlo: It'll be easier to notice. It, it it'll be easier to notice if that's happening if you can just like see all the activity that the companies are doing and then you can like see like, "Oh, hey, like this company is being punished and this one isn't, but like it seems like the activity they're doing is pretty similar," you know? So, so it helps reduce that type of overreach or that type of power concentration as well.

But let me, let me talk about the, the more big effects on concentration of power. So, so far I talked about the benefits for like having good safety focus AI regulation that achieves its actual safety goals and also the effects for like advancing the science of, of AI alignment. On the concentration of power side, you know, what are the most important things we can do to have a world where power does not concentrate extremely due to AI?

Well, first, we need to avoid AI monopolies, which means we need to have multiple countries with frontier AI programs, because even if there's multiple companies with frontier AI, if they're all in the same country, then that's a monopoly waiting to happen. You know, that's like all it takes is the government deciding that it wants to nationalize today or something, and then now one man controls all the AIs.

Kevin Frazier: That would never happen either!

Daniel Kokotajlo: You know? So, so you want it to be the case that it's spread out over multiple countries and, and, and ideally you want there to just be more frontier AI companies rather than fewer, you know?

And then also separately, you want there to be transparency into what those giant armies of AIs are up to and how they're trained, so that entities that don't directly control giant armies of AIs have a say, have like more of a say and more oversight into what's happening. Like, even if you had like, you know, 10 frontier AI companies spread out over 10 different countries, if it was still like the situation today where there's like very little transparency into how the AIs are trained or what the AIs are being told to do, then you kind of end up in a situation where no one else matters except for those 10 AI projects and their leadership, you know?

Like, for example, the Supreme Court of whatever comp- what, like, you know, say there's like a single, say there's a U.S. AI program and, like, the, the president is in control, in charge of it or whatever. How is the Supreme Court supposed to, like, give oversight over the president and whether he's doing something unconstitutional with his AIs? If, in, in, in today's world, like, they don't even know what the AIs are up to. They don't know how the AIs are trained. You know, Congress doesn't know either.

So anyhow, basically, you want to spread out, that we, we wanna avoid a monopoly and you want there to be transparency into how the AIs are trained and what they're doing. And the transparency thing that I mentioned before helps with both of those things because, because we are doing the total research transparency, that's basically sharing the core algorithms and the core recipes with the world, which is going to help other companies catch up. So it's like directly fighting against this monopolizing force.

And then of course the transparency just, well, there you go. It's transparency. So, it's, it makes it much harder for companies to abuse their power. An example that I think I like to talk about is secret loyalties or, you know, hidden agendas. So, this is an e- on, on the, on the spectrum of ways in which a company can abuse their power, this is like the most egregious way, and there's of course a whole spectrum that's less egregious and more nuanced.

But, but just to t- talk about this one a little bit imagine a situation where, you know, a chatbot that's used by 100 million people in, in America has a secret agenda, and it's like secretly trying to push the political views of company leadership or perhaps secretly trying to help, you know, their favorite candidate win the election or something like that. They could have quite an effect on such things, 'cause you multiply by 100 million people, that sort of like subtle you know, subtle, subtle propaganda or whatever that the chatbot is doing could have a big effect.

Kevin Frazier: And it's worth noting-

Daniel Kokotajlo: And so-

Kevin Frazier: Already that there's research from Jillian Fisher at the University of Washington showing that just subtle engagement with a subtly biased AI chatbot can already start to change the views of users. And it's worth noting also that we know these tools in some contexts are more deferential to the companies that created them. When you ask questions about how should we regulate this company as opposed to that company, there is a sort of self-

Daniel Kokotajlo: Yeah.

Kevin Frazier: Referencing bias there. And so in terms of the sci-fi-

Daniel Kokotajlo: Yeah.

Kevin Frazier: vibes that people may be getting, this is being empirically documented-

Daniel Kokotajlo: Yeah.

Kevin Frazier: Already in the literature. And so, we could go down the, the need for transparency for many more minutes.

Daniel Kokotajlo: Yeah.

Kevin Frazier: A- and I'm glad we've covered it here.

Daniel Kokotajlo: Yeah.

Kevin Frazier: I do wanna make sure we leave time for me-

Daniel Kokotajlo: Okay.

Kevin Frazier: Ro really throw, you know, the hard balls at you.

Daniel Kokotajlo: Yeah, yeah.

Kevin Frazier: But let's transition to the fact that you all have also outlined this concept of mutually assured compute destruction. Why is that-

Daniel Kokotajlo: Yeah.

Kevin Frazier: Necessary? We've, we've had this, for lack of a better phrase, kumbaya of total transparency. The world is high-fiving. We have 100 AI frontier companies across the globe doing cool stuff. Why do we need this concept of mutually assured compute destruction?

Daniel Kokotajlo: Yeah. So I wouldn't say it's necessary. Like, you could do Plan A without this component, but that would be risky, or more risky than with the component. It's kind of a, it's a fail-safe mechanism. And the reason why is imagine that you imagine that the deal breaks. You know, imagine they've been doing Plan A for a couple years, and all these new data centers have been constructed, and now there's an order of magnitude, maybe two more, two orders of magnitude more compute in the world than there was at the time that you s- initiated the deal.

And then for some reason, there's a conflict over Taiwan or something, and then the deal breaks down, and they stop being transparent to each other. And now they're not transparent to each other, they can't trust that they are not racing to superintelligence anymore, so probably they're gonna start racing to superintelligence. And now you have a race to superintelligence happening, except it's gonna be even faster because of all this compute that's built up.

Like, according to our, you know, our estimations, it might take something like a year, you know, a few months to go from fully automating AI research to superintelligence. Obviously, there's a lot of uncertainty about that. But what we feel confident in is that if it would've taken, you know, X length to cross that gap with a certain amount of compute, then it will take much less than X to cross that gap with orders of magnitude more compute. And so especially if you've had a couple years of delay, so you can-

Kevin Frazier: And just to pause there for just one quick second.

Daniel Kokotajlo: Yeah.

Kevin Frazier: The idea that okay, if we have 100 frontier AI companies, then we're going to need orders of magnitude, as you pointed out, more compute. And so if we have all of this, all of these data centers all around the world, the fact that we could see what I'm gonna steal from Tom Davidson at Fore at Forethought, when he refers to this as dry tinder, which is like you've gotten all of the fuel for a quick takeoff scenario where if China decides to say, "Hey, we're going to go the other path," well, now you've created the dry tinder for that to become a conflagration that moves really quickly in a way that previously-

Daniel Kokotajlo: Yep

Kevin Frazier: Wouldn't have been possible.

Daniel Kokotajlo: Yep. And like quantitatively, there's this parameter of like how much of an effect it would have, like basically how much research depends on compute these days. And I don't know, our guess would be something like 10x more compute would be like 3x faster, and 10x less compute would be like 3x slower or something like that. And so that means that if it's 100x more compute, then it goes 10 times faster. And like it's already fast enough. 10 times faster version is even scarier, you know?

So, so, the, the compute destructibility thing is a sort of fail-safe mechanism, where the idea is that if the deal breaks down, then all the new data centers that have just been built as part of the deal get smashed. And we go back to the world before, basically, where people still have the data centers that they had at the start of the deal, but they don't have all the new ones that have been built since.

How do we achieve this? Well, in some sense, it's achievable by default in that, like if you imagine this deal going on and then conflict breaking out, fear that they're gonna start racing to superintelligence because they're not being transparent anymore about what they're doing on their AI clusters, it's plausible that just the count- countries would just start shooting missiles at each other's data centers because, because they're afraid of what would happen if, if we don't, you know?

But then that is really scary and could lead to World War III because now we have both countries shooting missiles at each other you know. And so basically, one way of thinking about it is that we want to set it up so that there's a, like, not exactly a peaceful off-ramp, but a, like, less escalatory off-ramp.

So, our specific proposal is that the new data centers be constructed with kill switches that the U.S. and China have access to, so that in case of conflict where the deal's breaking down, they can, either one of them can just sort of, like, delete the data the new data centers.

Kevin Frazier: Bomb in case of emergency.

Daniel Kokotajlo: Yeah. Yeah. And then, and, and, but, so presumably you would only do this if things are already getting really intense, right? Like, during peacetime, if things are going well, like, if you just, like, destroy their data centers, then they're gonna destroy your data centers, and then now the whole economy crashes. You know? Like, like, y- this is this kind of like a last resort type thing that would therefore only really be used if things are really getting crazy and, like, you know, people are genuinely afraid that the other side is going to get superintelligence and then attack them, for example.

But it's less escalatory than an actual full-scale war, you know? If we set up the da- the new data centers to be easily destroyable, then it's at least more likely that they would get destroyed and then there'd be peace instead of they get destroyed and now we're in World War III, you know? And the, and the, the, like, technical mechanism would be destroy- having these sort of, like, self-destruct switches.

But then if you're, if you're suspicious of technical mechanisms like that and you think that maybe they could be, like, backdoored or somehow sabotaged, we have a very dumb, non-technical mechanism, which is for the U.S. to build their data centers in Mongolia and for China to build their data centers in Canada. That's a very dumb non-technical mechanism. Regardless of what happened with the kill switches or whatever, like, if the data centers are sort of swapped like that, then, then in case of conflict, it's obvious what's gonna happen. China will take the U.S. data centers, U.S. will take the Chinese data centers pretty easily. And then, of course, since that's what's gonna happen, the owners of those data centers would just sort of scuttle their compute instead of letting it fall into enemy hands. And so you get this sort of relatively clean, it's all gone now. We don't have to keep fighting World War III.

Kevin Frazier: Okay, okay. Yeah. So mutually-

Daniel Kokotajlo: Yeah.

Kevin Frazier: Assured compute disruption. So we've got-

Daniel Kokotajlo: Yeah.

Kevin Frazier: Rhe rough contours of Plan A. And for my AI policy nerds, go read it, go check out the whole thing. Another instance in which there are fantastic graphs and workflows as we briefly outlined here. I wanna start off, though, for the folks who are listening to this, and we started off by saying the policy objective here is to delay superintelligence.

Now, I know some folks listening to this are saying, "Why? Why delay superintelligence?" This is the most exciting thing that humanity's ever going to do, is to create something that can solve every problem. Just this month we saw that the hardest math problems are seemingly being dropped like flies. We're just tackling things left and right. Shouldn't we be celebrating and accelerating towards superintelligence? What's your, your chief argument there as to why you think the costs of superintelligence outweigh those benefits right now?

Daniel Kokotajlo: I would say we do wanna build superintelligence eventually, but the way that we do it is extremely important. And if we do it in race conditions, like we're currently doing, and we're doing it, and if we do it by having the AIs recursively self-improve to superintelligence, then most likely we're going to lose control of the AIs at some point, I would say.

I mean, other people think it's not most likely, it's only 10% likely or whatever, but even 10% is pretty scary. I would just come out and say it's most likely. I don't see, it, it seems to me, like if you put y- you know, Claude in charge of Anthropic and have it build the next Claude, which then builds the next Claude, which then, then builds the next Claude faster and faster and faster, probably you're gonna end up at the end with superintelligence but you're not gonna be in control of those superintelligences, you know?

There's this chain of trust of, like, the superintelligence will do what we say because it was aligned by the previous generation AI that was aligned by the previous generation. And it's like, first of all, the base case isn't working. Like, our current AIs are not aligned, you know? So, like, what, what, what is going, like, why would you think this is a good idea? Why do you think that you're still gonna be in control of the superintelligences at the end?

Instead, they're gonna make you think that you're in control because they want you to continue, you know, not shutting them down. And y- they want to have you continue. So, but, like, basically, they are gonna be in control, and they're going to be, like, you know, just pretending to be aligned until you've given them enough hard power that they don't need to pretend anymore, as described in the race ending of “AI 2027.”

So, so that's, like, number one problem, I would say. Number two problem is that even if that somehow doesn't happen and you end up in control of the superintelligences, well, that's a huge power grab that you just did over the rest of the world. Like, now you, Mr., Mr. Altman or Mr. Amodei, are in charge of this giant army of superintelligences and, like, probably the other companies are still a few months behind, and so they probably don't have nearly as smart AIs as you do.

And then also, like, what about everyone who's not a CEO of a tech company? Like, how much power do they have now? You know? Like, and then also, like, what about, like, Russia and China and India and, like, all these other countries that are now, like, staring down the barrel of American companies coming and taking all their jobs and also building giant robot armies that can completely obsolete their militaries and, like, you know.

Like, so, so you, you've just done, like, this crazy power grab over everybody else in the world. People are not gonna like that. They're gonna get very scared. They're gonna try to stop you. That could lead to World War III. Like, so, so, you know, these are, these are the, the problems, just some of the problems. There's, I haven't even got down the, down the list. These are some of the problems that arise if you just continue on the current course and you automate the AI research as fast as possible.

We still wanna build superintelligence, but not like that, you know? We wanna do it in a more cautious way, where we're sort of, like, gradually improving the AI's capabilities, gradually changing the way that they're trained, but in a way that's, like, you know, safe and where we've, like, put a lot of thought into each, into each step. And then also we want it to be more power distributed so that it's, it's, it's not this sort of like winner-take-all, whoever recursively self-improves fastest wins, but instead there's a whole bunch of different companies spread out over different countries that are sort of like scaling up in parallel together. And there's lots of transparency so that, you know, they're public and, and their legal system can, like, tell that they're not abusing their power over their AIs.

Kevin Frazier: And I'll, I'll flag too from a lawyerly perspective or constitutional law perspective and an emphasis on the rule of law. The rule of law, in my opinion, is best described as checks on arbitrary power. And to your point of one company having the world's best AI that's orders of magnitudes better than whoever the next AI company is, what checks are there on that sort of company, right? Who is actually in a position to contest when and how the model is behaving in a certain way or what values get selected, how we train it to prioritize certain values over others?

There is no quality check in place right now, and so just from a bare rule of law perspective about preventing one individual from having that sort of arbitrary power over the lives of so many people should raise red flags for anyone who is concerned about making sure that those, there are those sorts of checks in place.

But we could go down that rabbit ho- rabbit hole for a heck of a lot longer. So you've addressed the ed- first point about why delay. I wanna challenge a second part, which is you released this report, “AI 2027” about 14 months ago or so, and you've updated your timelines on a few occasions and said, "Hey, it may be a, a second longer. It may not be coming quite as quickly." This morning I was reading ImportAI, Jack Clark's newsletter and he was summarizing research from Sayash Kapoor and others showing that, you know, AI isn't actually very good at being creative yet when it comes to coming up with new research proposals. And it doesn't show the same degree of taste in thinking through novel approaches for which research questions to pursue next.

And this led Jack to put as the title of one section of that blog post, "Why This Matters: The Singularity Could Be Delayed." AI systems are about to start building themselves but that may only be possible if they're capable of, quote, "creative paradigm-shifting insights." And so if we're not seeing that activity from AI, if we're not seeing that creativity to really push the frontier of research, are you continuing to delay your timeline as to when recursive self-improvement may be reached? Or where do you stand right now on some of the evidence that we may not be moving there as quickly as possible?

Daniel Kokotajlo: So first of all in “AI 2027,” you don't see this sort of thing until mid-2027. So the fact that we're not seeing it now in mid-2026 is not, doesn't mean much.

Kevin Frazier: You, you've got 12 months. You've got 12 months, and then we'll talk again.

Daniel Kokotajlo: Yeah, yeah.

Kevin Frazier: But wh- where, where do you stand right now?

Daniel Kokotajlo: We'll still be talking again in 2027.

Kevin Frazier: Would you embrace the same posture?

Daniel Kokotajlo: Yeah, so we have uncertainty about timelines. I think that, and, and you can actually see our historic predictions about timelines. And so you don't have to take my word for it. You can go look at, like, the various past predictions we've made. There's a handy graph that you might wanna look at on our website, the, or on our blog. We have a update called “Q1 2026 Timelines Update,” and then one of the graphs in that update shows the history of my timelines over time and Eli's timelines over time.

And so you can see it collapsed in 2020 as I, I started understanding the scaling laws and language models and things like that. And then it sort of went down a little bit to 2027 as my median. Then it went up. It reached as high as 2030 as my median, and now it's going back down again. So my, my, my opinions have sort of wobbled back and forth as to the median. But of course, like, that's just the median. The point is that, like, I've had uncertainty over…

Keep going, keep going, keep going. That one, that one. There you go. Yeah, so that-

Kevin Frazier: Okay. We've got it up. For the folks listening-

Daniel Kokotajlo: So that's-

Kevin Frazier: We've got it up-

Daniel Kokotajlo: Yeah.

Kevin Frazier: The timeline estimations here.

Daniel Kokotajlo: Yeah, so that's a history of my historic public predictions about, about AI timelines, basically. And as you can see, and, and the thing that's being tracked there is the, my median, so the 50% mark. Because obviously it's not like I think it's definitely gonna happen in that particular year. I have uncertainty spread out over many years, and this is just the 50% mark. Yeah.

Anyhow, so, at the time we started writing “AI 2027,” 2027 was my median. By the time we published it, 2028 was my median because I had sort of updated towards slightly longer timelines. Then briefly towards the end of last year, my timelines lengthened even more, up to 2030. And then now they're going back down. And so now I would say 2028, probably something like that.

And one of the things that we're gonna work on soon is, or hopefully publish soon, is an updated sense of timelines. So yeah, I mean, we have uncertainty. It could happen next year. It could also happen in 2030 or maybe in some year in between. I think it will probably have happened by 2030 and probably not have happened by end of 2027. But, like, somewhere in that range.

Kevin Frazier: Okay. So you mentioned-

Daniel Kokotajlo: Would be, like, the, the space of plausible options.

Kevin Frazier: You mentioned earlier that one of the things you wish you had changed or addressed in “AI 2040” was recognizing perhaps the greater need for domestic activity by the U.S. to kind of start or initiate a more meaningful discourse with China in terms of reaching a deal.

Now reflecting back, you've already, for example, changed your predictions about which of these plans, plan A, B, C, D, or S, you think is most likely following the OpenAI hugging face incident, you now suspect that Plan A may be 18% likely, an improvement on 15% likely. You have now diminished the likelihood of Plan D, the default do nothing, from 30% likelihood now to 20% likelihood.

You've already changed some of these things. What's been the main additional pushback or additional piece of feedback that you said, "Huh, that was a really good take I wish we had addressed more," that, you know, has really landed for you and the team?

Daniel Kokotajlo: Well, I think, I think the main one is actually something that you may have just covered, which is this domestic regulation thing. I think that Richard Ngo has this critique where he basically says that, like, we are, like, inadvertently reinforcing this harmful narrative about the race with China by... Well, I mean, if you, if you read any of our work, we're very much talking about the race with China. And that's not because we think that it's good that there's a race with China, it's because we think that's where D.C. is at, and that's where, you know, policymakers are thinking about, and so we want to sort of meet them where they are and be like, "Yeah, race with China, it's a serious thing. Here's the way out. Here's what we think should be done about it."

But Richard is thinking that maybe it's better to, like, deny the premise more and say that, like, it's not really a race because you're not gonna win. Like, you're gonna lose control of the AIs. So, like, what do you, like, is different from, from most races. For example, that's...

So, so, and I do feel like maybe he's right about that. And I don't know. We'll, we'll see. But, but we, we said what we said, and we do think that, like, even from within this sort of race with China framing for the reasons that we've stated, you should do the things that we, we recommend.

Kevin Frazier: And for the rest of the AI policy community, we've already talked about the value of scenario scrutiny and really pressure testing your ideas. Would you like to see more people publishing their own version of “AI 2040?” And what would that look like?

Daniel Kokotajlo: Yes.

Kevin Frazier: Do you see a sort of field of scenario scrutiny developing?

Daniel Kokotajlo: I hope so. I think, yes, we would love to see more of those things. In fact, one of the, one of the nice things I think there was this scenario called EU or “Europe 2031” that was, you know, clearly inspired by “AI 2027” and, you know, we don't agree, we don't agree with the authors about everything, but, like, we are very pleased to see people sort of like put things down.

And then I think that, like, once we have a bunch of different scenarios on the table, then we can have, like, the argument about which is more realistic and which is less realistic and, like, what are the different, you know, aspects of them and stuff. And so, I think that's, that's, that's good to happen. I think there's there's a step beyond that which I'm hoping to happen, which is like war games.

And I, I wanted to mention this just as a brief. Part of the motivation for scenario scrutiny is that when I look at the history of m- military history I'm sort of inspired by how seriously they take their jobs intellectually. Like when, when, even back in World War II it was common for commanders to have war games gaming out the plans that they had for the war and for the battles and so forth.

And so, for example, with the Battle of Midway, the Japanese ha- they had their plan, and they made the plan in part on the basis of various war games they had done. And then even after they had made the plan, they kept war gaming it out, like, as they were, like, getting ready to attack. And in fact, if they had taken their own war games more seriously, they might have realized that they were about to lose the Battle of Midway, because in one of their war games the per- person playing the Americans had the American fleet waiting in the north to attack them and then utterly wrecked the Japanese fleet. And then they were like, "Well, but the Americans don't know we're coming, so they're not gonna be, they're not gonna be doing that." And in fact, they did know they were coming, and they did do that, and they got wrecked.

So, so th- that's an example of, like, you know, people in the war, people in, people in the military take very seriously this idea that you need to apply a scenario scrutiny to your plans. Like, not just scenario scrutiny, like war game scrutiny, which is like a, since like a, a higher level of scrutiny. It's like not only are you gaming out in detail, like, what it would look like to implement your plan, you're then subjecting it to adversarial pressure and, like, gaming out different possible ways it could go, where there's someone whose job it is to sort of, like, break it, basically, you know? And I think that ideally, we'd like to get to that place where the policymakers in Washington are treating super intelligence with the level of seriousness that is routine for military operations.

Kevin Frazier: Well, and I think too, as I've reflected this summer on July 4th, as we all have in some way, shape, or form, it's worth noting that the founders themselves were engaged in a degree of war gaming when they were drafting the Constitution, thinking about what are the ways in which this could break. Let's use our imagination. Let's have a high degree of creativity. And yet that's often lacking in, in a lot of these policy discussions.

So again, I will applaud you all for taking that on and embracing that sort of thoughtful, creative approach. But Daniel, I know you have many more reports to author, many more timelines to sketch out. Any final thoughts you wanna leave for our audience?

Daniel Kokotajlo: Yeah, thanks for asking. A couple things. I'll try to briefly go over them. So one, Plan A is, is different from just, like, permanently pausing AI until 2040 and then going as fast as possible. It's more of like a, a graduated controlled scale-up over the course of the 2030s. And it does involve some pauses at various key points, but, but, so that's one thing, is that, like, the, the world transforms dramatically in the 2030s if we do Plan A.

And one intuition pump for that, or one reason why that's happening, is that in Plan A, you are sort of slowly scaling through the human range, and you're starting, you're starting, you know, in 2029, they're already at a point where the AIs are able to automate some jobs and are having, like, a big effect on the economy. And then they were set to, you know, get to superintelligence in 2030, but instead they go more slowly.

But that means that, like, you're still having this transformative effect on society, and you have these AIs that, like, by 2035 are as good as top human professionals in basically every field, while also being much faster and cheaper. And so, the economy goes crazy. Like, we, we, we project that that countries are going to want to limit economic growth rather than encourage it in Plan A, and that they're going to want to have limits on, that are gonna look things like only one doubling per year. You know, and things like that. When, when, for contrast, for con- for context, like right now, the economy grows at, like, 3% or 4% per year on average. And so a doubling would be, like, 100% growth, you know?

And, and so, and we think that that is actually what you get, and it's, it's actually pretty straightforward, the argument for it. If you have machines that can substitute for human labor at practically everything, but the machines are much cheaper than humans, and you can produce more of them much more easily because they are, after all, just, just more GPUs, and we know how easy it is to produce GPUs, and we know how easy it is to produce robots, then your, your, like, population is basically doubling several times a year or maybe doubling once a year. Depends on how, you know, maybe it starts off at once a year, and then it gets faster, you know?

And so then your whole economy, once, once that population is the bulk of the economy and the humans are just sort of like a small sliver on top of this giant army of, of robots and robot factories and robot trucks and, and everything then the whole economy is growing at machine speeds instead of growing at, you know, human reproduction speeds, you know?

So we, we say more about this in, in the supplements, and we explain why we think this. But, but it's, I think, an important sort of high-level takeaway is that you still get this insane rapid transformation of the entire world, abundance for everyone, all that sort of stuff, even if you basically pause at human-level AGI.

Kevin Frazier: Yeah, and it, worth flagging the often-quoted remarks from Ethan Mollick here, which is to say, “even if we pause today,” not that I'm endorsing a pause of any, any kind necessarily, “but saying that we have so much room for just integrating the advances from today's AI-”

Daniel Kokotajlo: Yeah.

Kevin Frazier: “That our systems and our institutions aren't ready for.” And so it's a huge societal task ahead. Thanks to you, Daniel, and the rest of the team for pushing the rest of us to engage in a thoughtful policy exercise and showing a potential way forward for how AI may unfold. I'll let you get back to it, but Daniel, thank you again for joining Scaling Laws.

Daniel Kokotajlo: Thank you very much, Kevin. Happy to, happy to come on more if need, need be. But,

Kevin Frazier: Sounds like a plan.

Daniel Kokotajlo: I really appreciate you covering these topics, and, you know, good luck to us all over the next few years. We'll see how it goes.

Kevin Frazier: There we have it. Thanks, Daniel.

[Outro]

Scaling Laws is a joint production of Lawfare and the University of Texas School of Law. You can get an ad-free version of this and other Lawfare podcasts by becoming a material subscriber at our website, lawfaremedia.org/support. You'll also get access to special events and other content available only to our supporters. Please rate and review us wherever you get your podcasts. Check out our written work at lawfaremedia.org. You can also follow us on X and Bluesky. This podcast was edited by Noam Osband of Goat Rodeo. Our music is from Alibi. As always, thanks for listening.