Scaling Laws: "The God Test" AI as Cosmic Reckoning, with Robert Wright

[Intro]

Kevin Fraizer: It's The Lawfare Podcast. I'm Kevin Frazier, the AI Innovation and Law Fellow at the University of Texas School of Law and a senior editor at Lawfare. Today, we're bringing you something a little different. It's an episode from our new podcast series, Scaling Laws. Scaling Laws is a creation of Lawfare and Texas Law.

It has a pretty simple aim but a huge mission. We cover the most important AI and law policy questions that are top of mind for everyone from Sam Altman to senators on the Hill to folks like you. We dive deep into the weeds of new laws, various proposals, and what the labs are up to, to make sure you're up to date on the rules and regulations, standards, and ideas that are shaping the future of this pivotal technology.

If that sounds like something you're gonna be interested in, and our hunch is it is, you can find Scaling Laws wherever you subscribe to podcasts. You can also follow us on X and Bluesky. Thank you.

Alan Rozenshtein: When the AI overlords take over, what are you most excited about?

Kevin Frazier: It's, it's not crazy, it's just smart.

Alan Rozenshtein: And just this year, in the first six months, there have been something like 1,000 laws.

Kevin Frazier: Who's actually building the scaffolding around how it's gonna work, how everyday folks are gonna use it?

Alan Rozenshtein: AI only works if society lets it work.

Kevin Frazier: There are so many questions have to be figured out, and-

Alan Rozenshtein: Nobody came to my bonus class.

Kevin Frazier: Let's enforce the rules of the road.

Alan Rozenshtein: Welcome to Scaling Laws, a podcast from Lawfare and the University of Texas School of Law that explores the intersection of AI, law, and policy. I'm Alan Rozenshtein, associate professor of law at the University of Minnesota and research director at Lawfare. Today, I'm talking to the journalist and “Nonzero” newsletter publisher, Robert Wright, about his new book, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning.

Bob argues that almost no one is taking AI seriously enough, that to grasp the moment, you have to set it against the entire history of life on Earth, and that navigating it will demand not just smarter governance, but a kind of moral and cognitive self-transcendence. We get into why he thinks today's models genuinely understand, why he thinks the race with China framing is self-defeating, and why a book about AI and geopolitics ends with a case for mindfulness meditation.

You can reach us at scalinglaws@lawfaremedia.org, and we hope you enjoy the show.

[Main Podcast]

Bob Wright, welcome to Scaling Laws.

Robert Wright: Well, thanks for having me.

Alan Rozenshtein: So we're here to talk about your new and excellent book on AI. It's called “The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning.”

So with that title, you, you have put the stakes pretty high. So just unpack... let, let, let's start by you unpacking the title. What, what is The God Test, and why is it so important that we understand AI from a, a, quote-unquote, cosmic perspective?

Robert Wright: Yeah, “The God Test” is supposed to have, I guess, several dimensions of meaning, one of which, not the most important really, but one of which is that we may wind up, by some accounts, building a superintelligence that has the kind of power traditionally associated with gods. And then one question is, do we manage to, to build a version of that that we know will treat us well, and that we can have a, a win-win relationship with? Or do things go awry in ways that doomers like Eliezer Yudkowsky have warned us about?

I would say the, the main meaning of the title to me is that I'm arguing that we face the kind of test that traditionally you expect a god to give to humankind, okay? So in the Bible you can find places where it says, you know, "Salvation is possible, but you guys are gonna have to shape up." You know? Whether they mean you're gonna have to worship Yahweh, you're gonna have to be kinder to one another, you're gonna have to make something that was considered in that context a moral advance.

And I really do think that's part of what we need to do here. I don't wanna over-dramatize it or focus exclusively on that, but because I think we do have to navigate the AI revolution as a cohesive global community, given the inherently international nature of the challenge, I think we're gonna have to have fewer conflicts, you know, wars and other kinds of fraught conflicts internationally, for that matter, intra-nationally.

I mean, I just think in general humankind's gonna have to do a little bit better job of getting on the same page. And maybe as the challenge unfolds, that will happen to some extent naturally. I, I hope it will. But in any event, I do argue, especially towards the end of the book, that we need to, as a species, get better at viewing our situation from, like, outside of our self, from a more objective standpoint that is, that is less influenced by the cognitive biases that I, I believe constitute what you could call the psychology of tribalism. So that's kind of the main meaning.

Alan Rozenshtein: So you, you, you, you said that you don't wanna over-dramatize it, but I, I think you should commit to the bit. Because again, this cosmic framing is, is, is quite dramatic. And, and I wanna, I wanna ask you why, what is the marginal benefit that you think this framing gives you relative to simply treating AI as a huge deal, right?

So like when I talk about AI-

Robert Wright: Mm-hmm.

Alan Rozenshtein: You know, I'll say it seems to me quite plausibly m-more important than the Industrial Revolution and quite plausibly as important as fire, right? Like that seems to be enormous stakes. But your framing actually goes beyond that, right? You have this whole six-billion-year story of how AI is part of the kind of the evolution of the, the planet as a life sphere itself.

And I, I'm, you know, I'm curious sort of apart from whether that's true or not, what to you is beneficial about such a powerful zoom out? 'Cause I mean, you're really- one of the points you make is that whether you're, whether the, whether it's the doomers or the, or the, the, the optimists everyone, even the people taking AI seriously, aren't taking it seriously enough. So I wanna, I wanna, I wanna ask you to explain that.

Robert Wright: Yeah, I think not that many people are taking it as seriously as I think it needs to be taken, and I'm trying to do a few things in the book. One is to explain to people, certainly including people who are not technically oriented, I'm trying to do it accessibly, like why the capabilities of AI have been advancing so fast. What is like the secret sauce of the so-called deep learning revolution?

I think once you understand that, and it's really in a way not that complicated you'll, you'll expect the advances to continue. So I want to, I want to convey that clearly. I also want to convey that when you, when you look at the technology and kind of the competitive dynamics behind its evolution, in other words, labs competing with labs, and then corporations wanting to be the first to adopt it and use it, and people using it and so on, and nations for the time being at least competing with it, that you realize there's, you know, it may not be that e- it's the kind of thing that could get out of control.

When, when you put together its likely future power with you know, the kind of competitive drive at various levels behind its ongoing advance and deployment, it's the kind of thing that could get out of control. And the, the, to get back to your question, I, I think, you know, this sense of magnitude that I'm trying to convey is reinforced if you step back and look at this revolution in the context of the entire history of life.

I mean, for one thing, this is the first time ever we've seen a whole new kind of intelligence that rivals human intelligence, you know? You, you can, that deserves to be considered a threshold even if you're thinking about all three to four billion years of the history of life.

Secondly, we don't need to get into this too deeply, but just to signal it, I, I, I think, you know, I, I, I bring in this idea that was- this term that was coined in, in 1923, the noosphere, N-O-O-S, the Greek word for mind, to refer to this kind of global brain that seemed even back then to be emerging thanks to communications technology. You know, humans were getting drawn into collaborative webs that transcended national borders and so on. You know, that has continued to develop. I mean, you could, you could view the global economy as a global brain and so on.

I, I think it's important, I think that is also a threshold in the whole history of organization on the planet. You know, cell, multi-celled organisms, societies of multi-celled organisms- humans start a second kind of evolution that includes technological evolution and, and you before long you've carried organization of a kind up to a planetary level.

I, I, I, I think as cosmic as all of this may sound when put in condensed form, I do think looking at the AI, AI revolution in this frame of re- of reference, you know, helps, helps us appreciate the magnitude of the moment, and I think thinking of it as, kind of, the product of an evolutionary force in maybe a couple of senses helps us understand the impetus behind it. It is something to be reckoned with, and I think we're starting, you know? The... I mean, the vibe is changing so fast. I'm sure you've seen this, right? Like, the level of awareness that something big is happening, and I think that's good, but I think we have a ways to go.

Alan Rozenshtein: Well, I, I just wanna ask you about that because you know, I, I wonder how much pushback you have been getting, and in particular, how that's changed over even the last year to your claims about AI capability.

So, so one... so you, you obviously have a great podcast yourself that folks should listen to, and I don't know if this... when this was but it was re- re- relatively recently, you had Emily Bender and Alex Hanna who are authors of the book “The AI Con,” and I think it's fair to call them deep AI skeptics.

Robert Wright: Mm-hmm.

Alan Rozenshtein: And I- I've listened to that episode, which on the one hand I found somewhat difficult to listen to, but in some senses maybe one of the best episodes I've ever listened to of any podcast because there was this, because, you know, y- y- you would say, you know, "AI can do this and that," and then they would say, "AI is totally useless, it does nothing helpful," et cetera, et cetera. It kind of went around for a very interesting 90 minutes about that.

Are you still getting that kind of pushback? Not necessarily about your claims about the noosphere. I'm sure lots of people are, may not buy that, but about the seriousness of AI or, you know, as we're recording this in June 2026, can we all just finally agree that we all feel the AGI and now it's time to figure out what to do about it?

Robert Wright: Yeah. Well, as you and I record this, the book isn't quite out. It probably will be by the time people hear this. But the- so I haven't gotten all possible pushback I could wind up getting. I have been talking about this in my newsletter and on the podcast, so I've been getting some feedback.

I would say the main skepticism I encounter comes from the left, you know, there, and a lot of these people are my friends and we agree about a lot. But there is this tendency to dismiss the, the, the more extravagant claims as marketing hype, and even the, even the, even the safety concerns as like, "yeah, they're saying it could take over the world, but that's just so we'll buy their stock."

So I, I still encounter some of that. I kind of think less and less. I mean, it, you know, it's funny with these things. Whenever you're, you make an argument that you think starts gaining ground, the people that you see as conceding the points don't see themselves that way and, and you think they're just redefining terms and moving the goalposts.

So it's always hard to say, but look, overall, there is no doubt in my mind that the public awareness is shifting. I, I don't, I don't think it's yet reached you know, what you could call a super, I don't mean this dismissively, but a super sophisticated stage. In the, in, in the sense that a lot of people just, they, they just have this feeling that things are moving too fast, and I agree that maybe they are, and we should talk about that. But I think they're, you know, it's all pretty fluid. There are people who right now consider themselves enemies of AI who may well find uses of it that they like a lot, and their opinion may change. I think it's very fluid, but the awareness of its sheer magnitude is growing, and I, I do think it's safe to say, I'm hearing fewer and fewer people saying, "Ah, this is... It's hype. It's hype." You know? It's just getting harder and harder to say that.

I mean, it's, it's doing manifestly more on a number of fronts, ranging from, you know, math theorems that no human had managed to prove to, you know, levels of you know, cyber, you know, security flaw detection or whatever the right term would be that, that go- far exceed what we had been able to do. I mean, you've read the stories. You know. So, I, I, I think pretty soon skepticism about its capabilities will not be the problem

Alan Rozenshtein: So let's talk about those capabilities. I wanna talk about sort of what they can, what these systems can do and, and also how they are trained to get there. So, you mentioned earlier that you think it's useful to think about the, the quote-unquote learning process, right? The pro- the process by which the billions or trillions of parameters after being initialized are tuned to be useful. That's often called, this is called learning.

But you think of it as evolution, right? You described that. And, and, and you have this really arresting sentence in your book that I really love, which is, and I'm paraphrasing here "It's not that we created machines that can think, it's that we created machines that create machines that think." And so just unpack what you meant by that and why this emphasis on thinking of this as a kind of an evolutionary process is helpful for your account.

Robert Wright: Yeah. So this gets to the kind of m- narrative framing of the book. I mean, in 1983, and yes, I am old enough for this to be true, I mean, if there's anybody watching this, they, they, they, they won't doubt that, but who knows? People, people hearing may, may, may have not reckoned with how long I've been around.

I was writing this piece on artificial intelligence, and I was calling around and getting background. One of the people I called was named Geoffrey Hinton, who is now the person most commonly called as the godfather of AI. He's won not only the Turing Award in computer science, but a Nobel Prize. And he himself, by the way, significantly, has become something of a doomer, which he wasn't back then.

But in, in response to your question you know, I, so I had kept up with the field a little. You know, as a journalist, I, I've written about a lot of things, foreign policy, but a lot of technology. I wrote you know, the, the Time magazine cover story when Deep Blue beat Garry Kasparov, the, the chess champ, and I, I wrote that. I wrote the, The New Republic's like "This is the Internet," cover story in 1993.

And a- and so I had kinda kept track, and every once in a while I would read about Geoffrey Hinton, but beyond that I was not keeping close track of AI. And then all of a sudden in, you know, when ChatGPT 3.5 comes out near the end of 2022, I'm, I'm like impressed. And then, and then 4.0 comes out and I'm blown away, and I'm like, "Man, something has happened," and apparently this Geoff Hinton guy I talked to you know, was, played a very big role.

So I went back and read the piece I'd written which, you know, depicted these, like, two schools of thought. There's the mainstream school of thought, kind of formal logic, blah, blah, blah. That was the mainstream of AI research, and then there's this maverick school of thought. That is what, you know, Hinton was associated with, neural networks, what he, what he was calling massive parallelism, or he was looking forward to the day when the parallelism could be massive as you know, the cost and power, cost of chips dropped and the power grew.

So I went back and I read the piece, and then I listened to a lecture he had given in 2018 explaining how neural networks work, and I realized I, I, I just had not gotten the picture when I talked to him about the potential power of this. And this gets to that, the point about learning versus evolution.

So in the piece I wrote, which came out in '84, I depicted this thing that I was calling a neural network. It didn't have much in common with the way they, turns out they, they, they actually work. I mean, it was a model that was out there. But the, the main thing was in this model each node corresponded to the sense of meaning of a word, like the word “throw” can mean “to hurl,” like a ball, it can mean “to host,” like, you know, “throw a party.” So there would be two separate nodes for that, and then I talked about how, you know, disambiguation would happen, whatever.

But the main thing is w- when I read that I, I thought, well, you know, the way I was assuming it would work and the way most people in AI were actually assuming it would work was, if you're gonna give linguistic facility to the machine, you are going to have to take the human understanding of the meaning of words and somehow impart that to the machine. You're gonna have to tell the machine how to represent the meaning of words.

And it turned out, that that had not been necessary. And the, the, the machines had, in a certain sense, come up, invented a system, I mean, in the book I talk about the senses in which this is and isn't an overstatement, but, but for the sake of time, let's just say you know, they, they did in an important sense, a, discover that meaning is a property of words, and also invent a system for representing the meaning of words. And in fact, it took us a while to figure out exactly what they were doing. I mean, it was a safe bet that they were using these things called vectors to do it, 'cause we had told them, you know, “use vectors to represent words,” these series of numbers. But we didn't understand how they were using it to capture meaning.

But it turns out you just give a machine one of these big neural networks, you know, a sequence of what to it is gibberish, just a bunch of letters, a bunch of symbols mean nothing to it, and, a- and, and have it keep getting better and better at predicting what's gonna come next and adjusting its internal you know, parameters, calibrations accordingly, and these, these vectors that represents words, and it winds up kind of inventing a way to represent meaning.

Okay, so now in the human brain, there is a way to represent the meaning of words. We still don't know what it is. But it is a safe bet, pretty safe, I would say, and most people, most psychologists I think, that it is a product of natural selection, right? Like, over the last few million years as our, as specific parts of the brain associated with language grow, things like the ability to represent the meaning of words are getting built into the brain by natural selection. It's like inborn equipment.

And, you know, that was the thing that blew me away. I realized that you can just basically give neural networks any kind of data, visual data, teach them ima- image recognition, and they will kind of reverse engineer functionality of the brain that took millions of years to evolve.

There's a well-known example in visual recognition, which is so-called edge detecting neurons in, in the human brain. I won't go into it, but it turns out, yeah, the machine invented, quote, "edge detection filters." The, the neural networks developed those. Nobody said to it, "hey, you know, if you're gonna get better at this task we're giving you, maybe you should start detecting edges." No, but it, it, it, it developed specialized equipment for detecting edges.

So, to me, that's the, the key thing, is that there's, in principle, no limit, I think, to the cognitive and perceptual machinery that these things can reverse engineer given the right data. I mean, that's why Mark Zuckerberg is having it trained on the keystrokes of his workers, 'cause they have these specific cognitive challenge- now, now in this case, it may not be these weren't designed by natural selection, these are more like learned cognitive- it can, it can do either thing, but then it can replace the workers, you know? It's so that's what I mean.

Now, you know, learning, the, during the training process, what we call learning also takes place. So the machine also develops a conversancy in a specific language like English. Well, and you know, that's something that happens, in the course of learning of an individual as they grow up. Fine. But the point is, the training encompasses both. It's encompassing, in a certain sense, evolution, oh, evolutionary time and, you know, kind of the developmental portion of a, of a, of a learning human brain.

Alan Rozenshtein: So the effect of all of this is a machine that can do these incredible feats of mathematics, writing, analysis, coding, I mean, in principle, you know, robotics soon, a- anything. And this raises the question of, okay, but is that, quote-unquote, "real understanding," or is it some simulacrum of understanding?

And I think you have a nice discussion of the famous Chinese room experiment by the, the philosopher Searle. And I think you make a pretty compelling argument that, you know, even on sort of, yes, actual understanding grounds, this, these, these machines understand. But you also argue that you know, if you wanna define that away, if you wanna say, if you wanna stipulate they don't really understand, but they functionally understand, that's good enough for your account for all the stuff that follows.

But there does seem to be one important difference that you do mention, but then it kind of drops out of the book, and I, I want to use the opportunity to push on it a little bit, which is the potential moral personhood of, of these agents. Now, as you point out Searle never said that you need consciousness for understanding in this Chinese room experiment.

But, you know, a lot of this question of, you know, does AI, quote-unquote, "truly understand," is a kind of way of getting at the Thomas Nagel-like question of what is it like to be an AI? Which is of course relative, relevant to the question of does AI have consciousness? Which is of course relevant to the question of do humans have consciousness? And onwards and onwards.

It does seem actually a very important question to get some clarity on, because if AIs end up as moral persons, and that seems something you are quite open to, I, I, I think, I think you I, I think you have this, this, this great line that you're, you're, was it... what is it? You're, you're 99% sure your wife is conscious, or you're 99% sure you're conscious.

Robert Wright: Now, wait, wait, now wait. Don't get me into trouble. I think I said 99.99.

Alan Rozenshtein: Fair enough. Fair enough. You're pretty sure.

Robert Wright: My dog is 98, but, but-

Alan Rozenshtein: There we go. There we go.

Robert Wright: -don't get my wife and my dog mixed up here.

Alan Rozenshtein: And, and, and and the AI is gonna be somewhere below that, but certainly not at zero.  I mean, that seems to be at a, a cosmically important question. A- and I wonder sort of why that drops out of the book.

Robert Wright: Well, it is cosmically important, and if by moral person you mean our treatment i- is-

Alan Rozenshtein: Yes.

Robert Wright: A matter of moral consideration as opposed to beings that have moral responsibility, which is another interesting conversation. Yeah, for me, the question of whether there are moral stakes in our treatment of AI is tantamount to the question of whether, as Nagel would've put it, it is like something to be an AI, right? I mean, that was his, I think, really excellent way of phrasing you know, the question of posing the question of consciousness and, and kind of- like, one way of framing what we mean when we say somebody has consciousness, objective experience, sentience, is like, if you said to them, "What's it like to be you?" If they start giving you an answer, if, if it's like anything to be them, then they are conscious.

And, and I think if it is like anything to be an AI, then how we treat them matters. Now, as you suggested when, when you know, noting that I didn't assign 100% probability to the chances that my wife is sentient, we never know for sure. This is the distinguishing thing about consciousness and, and sometimes people seem not to, you know, see this fundamental property of consciousness. It is, unlike everything else we talk about in the universe, it is not publicly observable. It is, it is inherently private and only one person knows, can verify its existence for sure, and that's why it is not amenable to scientific study in the same sense that everything out there, including electrons, is, because you can't have two people both look at it and discuss the data.

So, I mean, I, I do not rule out the possibility that AI will be sentient, and I don't even rule out the possibility that it already is. I mean, it's, it's quite possible that consciousness is a property of goal-seeking intelligence, for example. Of course, there are views of consciousness according to which every living being has it, and views according to which every, every physical thing has, has a little bit of it. It's, it's a mystery. But I do...

So, so I would say be on the safe side, be nice to your, your AI. I, I mean, I've lost my temper a couple of times, embarrassingly. But and, and interestingly, but interestingly, by the way, Claude, when I did this I went back to the conversation. You know, I said some insulting things about Anthropic. I mean, I'm sorry, but it was., I was having to deal with this interface that was just so badly designed. So yeah, I, I kind of said some mean things about Anthropic.  The, the funny thing is I went back and Claude had erased that final part. Like, we don't want to remember this part of our relationship, do we? And I like, "I guess not. You're probably right."

Anyway, the, I encourage people to be nice to it. It's good, you know, you're, you're forming your own habits, and you should be nice to beings. But I think, y- you know, if we are gonna talk about what we mean by understanding, you can, if you want, define consciousness as a prerequisite for understanding, fine. But if you're doing that, then I don't think we- there's any point in discussing whether AIs have it, 'cause we just don't know if they're conscious.

So I, in, in, in looking at Searle's Chinese room thought experiment, and I like to think showing that LLMs, and particularly multimodal LLMs have finished it off once and for all. But in any event I argue that, you know, if we're gonna have the discussion, we can't bring consciousness into it. And, and by the way, Searle doesn't seem to have meant to bring conscious into it, at least in the, in the first incar- and famous incarnation of his argument.

Alan Rozenshtein: Okay, so we may not be able to, to publicly discuss this question of consciousness, but we can certainly publicly discuss all the, the threats and worries that, that AI poses, which kind of gets into the, the next section of your book. So, I, I wanna offer a characterization of the worries, and then I want you to push back in case I got it wrong.

Robert Wright: Mm-hmm.

Alan Rozenshtein: And that is, of the two different kinds of harms that AI can cause, the w- the first being AI can enable bad, other bad actors to do bad things.

Robert Wright: Mm-hmm.

Alan Rozenshtein: Right? So, you know, right now we're talking in the midst of this blowup between Anthropic and the U.S. government over whether the Fable model is too dangerous to be released. The government's concern here is not that Fable itself will go rogue and hack a bunch of computers, but-

Robert Wright: Mm-hmm.

Alan Rozenshtein: the Chinese or someone else will, will do that, right? But then there's this other set of risks, which is the AI goes rogue risk. Your book seems a lot more concerned with the latter than the former, in other words, with these questions around rogue AIs autonomously causing problems.

And, and so I'm curious, A, if you sort of agree with how I have ranked how you describe the concerns and, and if so, then why, right? Because of course, that is, by definition, more speculative than the harms we know right now people are trying to do with AI.

Robert Wright: Yeah. That's interesting feedback because you're right that, I think you've correctly represented the proportional focus in the book. In other words, I, I probably spend more time talking about the implications of AI being autonomous, and the fact that we're, we're, we're- you know, there's a strong incentive to build autonomous AIs. That's what corporations want in a certain sense. That's what people want as assistants, and so on. And, and the possibly dire implications of that.

I, I spend a lot of time on that partly because it's, it takes more persuading, I think. And let me compare it to the part that I actually think near term is in a way more important and I'm more sure is going to be a problem. And what I'm referring to here is just the number of fronts on which I think AI is going to be disruptive in the not necessarily good sense of the word.

So, like, you know, jobs, it may be that the people who lose jobs will find new jobs. I don't know. But it's it's disruptive even if, if a lot of people have to find a new job at roughly the same time. And there are, you know, there are a lot of these fronts where some degree of disruption is, I think, assured. Our, our family lives, our romantic lives, you know, our, our friendships, the, and the realm of, you know, safety and national security and you know, I, I could, I could go on. There's just a lot of adapting to do, so that, even if successful adaptation is possible, I, I think we're still gonna get like a big jolt at the current rate of AI advance.

And that's, that, that's in a way what I would like to most immediately focus people's attention on, is that like, look, faster isn't necessarily better. We have got so much to figure out even on these relatively mundane fronts. And you know, in a lot of cases the stakes are high enough that maybe mundane is a misleading term.

I mean, if it helps, if AI helps somebody build a bioweapon, you know, and you get a, a pandemic the likes of which the world has never seen, because if you really want to cause death and destruction you can create something much worse than COVID. Or if, you know, to take Mythos, if you wind up with some self-replicating super hacker hopping from data center to data center and you know, commandeering commute and getting stronger and so on, you know, these things are now clearly possible, right?

I'm not saying those are that likely. Those aren't as likely as the job disruptions and so on, but I, I'm, I'm just saying given all of the concerns that, you know, leave aside the Eliezer Yudkowsky AI-takes-over-the-world scenario, the, the things that we're pretty sure are gonna happen, or at least threaten to happen pretty soon, I just think it's an argument for not wanting things to proceed maybe as fast as they're proceeding.

And, you know, at a minimum, when you start talking about regulation of some kind and, and, and people in the AI company say, "No, but wait, that would slow us down," I think the appropriate response is "I think that's a feature, not a bug." I would at least get to the point, I'd like to get to the point of saying that.

So I'm just, I'm just saying I take your point that I do spend a, a fair amount of time talking about the possibility, both of the sci-fi Yudkowsky scenario of actual AI take over the planet because although I used to be dismissive of that, I find that it's hard to rule out completely. And then beyond that, it's just the kind of possibility of very problematic individual cases of rogue AIs, given the autonomy and power they're gonna have.

Yeah, I spend I do spend a fair amount of time on that, but, but it doesn't necessarily represent the pr- the proportionality of my near-term concerns, I would say.

Alan Rozenshtein: So let's talk about what to do about those, and here I want to get into the, the geopolitics part of, of your book. You've been an outspoken critic of what is sometimes called the "but China" approach to thinking about AI-

Robert Wright: Mm-hmm.

Alan Rozenshtein: -which is the idea that we can't possibly regulate, we can't possibly slow down, we can't possibly hamstring anything we do in the United States because we need to beat China. And, you know, there's, I think it's notable that to the extent that there are any bipartisan agreements in Washington these days, it seems to be that we must defeat China is one of the few among them. And it's also notable that even some of the most safety-focused of the AI companies, and here I'm thinking of Anthropic in particular, are actually very hawkish on China.

Robert Wright: Mm-hmm.

Alan Rozenshtein: So you're, you're, you have very strong headwinds in arguing against this. So, make the case. Why, why is this framing that we are in a competition with China for AI supremacy, and it is very important not just for the United States, but for the quote-unquote “free world” that the, you know, the, the, the, the, that the machine god speaks, you know, English, not Mandarin

Robert Wright: Mm-hmm. Yeah, you're right that Anthropic is China hawkish. Certainly their leader, Dario Amodei, is. And I would distinguish them in that regard from other seeming China hawks, because some people in Silicon Valley I think are just using “but China” as a convenient anti-regulatory talking point. I think Dario Amodei, to his credit, is genuinely ideologically committed to what he thinks of as an existential race between the democracies and the autocracies or authoritarians.

And he sees China as leading that block, and he thinks that the U.S. has to get to super intelligence first, and he sketches this out in “Machines of Loving Grace,” and he also co-authored a piece with Matt, Matt Pottinger. People familiar with the policy landscape will know that Pottinger's quite a China hawk, and they, they co-authored an op-ed in The Wall Street Journal. But, so he's serious about that.

Why do I have problems with it? Well, let's start at that race for super intelligence. There was an interesting paper called "Superintelligence Strategy," by Dan Hendrycks and Er- Eric Schmidt, and one other co-author. And th- this wasn't the main, this wasn't the point they really highlighted, but one, one thing they noted is if you imagine a race between two superpowers towards superintelligence, if they both believe that superintelligence is gonna confer just utterly hegemonic power, then there's a strong incentive for the one that's a little bit behind to engage in dramatic action preemptively, including prosu- possibly kinetic strikes, and we're talking about two nuclear powers here.

So there's, there's that. And remember, the whole premise, Dario totally subscribes to the idea that, as you approach the superintelligence threshold, because the pace of progress is accelerating, being ahead two or three months is gonna translate into being ahead by light years, right? Like, there's this threshold, and the closer you get, the faster you move, and at the threshold you have this dramatic strategic advantage.

Now, is it true? I don't know. All that matters is that if both countries believe it, this is a very volil- volatile situation, okay? And I'm not sure he appreciates how dangerously unstable the final part of the race could be. So that's one thing.

The- another thing is just that, as I've said I think we need to slow down for various reasons and start formulating some wise policies and give ourselves time to adjust to all this. And to the extent that we consider China this existential threat and, and think of ourselves in this, in this big AI race, that's gonna be hard to make happen, because it is- because it is a very effective talking point for anti in, in anti-regulatory rhetoric.

But the biggest reason that I think we need to rethink our, you know, fundamentally adversarial conception of China is just that I think a lot of the policy challenges posed by AR-, AI are just inherently international. You know, I've talked about two of the big ones. I mean, if it helps somebody make a bioweapon, if it, you know, if a, some bad actor, including a non-state actor makes a super hacking machine that crosses borders and starts wiping out various kinds of infrastructure. You know, I could go on, but there are just a number of very consequential cases where national policy alone cannot keep your nation safe. Okay?

And so, I do, you know, you're gonna, you're gonna have to rest assured that if you're gonna feel secure as a nation, you're gonna have a rest assu- you're gonna have to, to be able to know that certain kinds of things are not happening in other nations, right? Certain kinds of AIs are not being developed. And when you look at the challenge of this kind of-- establishing this kind of transparency, it's, it's a much bigger challenge than it has been with nuclear arms. I thi- I think the kind of arra- international arrangements you need, and a lot of people have said this, you know, are more challenging than they were with nuclear arms.

So although I agree with people who've said, "Look, we can walk and chew gum at the same time. We can have a pretty tense relationship with China and still work out some deals," I think that's true. But my own view is that if, as time wears on, a few discrete, somewhat effective, you know, agreements are, are probably not gonna be enough, and we're gonna need more of what I call “organic transparency” as well. That is to say, not just formal monitoring mechanisms, but the kind of insight you get from actually being fully engaged with another country and on good terms with it, right? You just know more about what's going on in the country. So that's the argument in a nutshell, and I'm not, I'm not saying it's gonna be easy to, to reorient our relations with China. It's just, in my view, I think it's, it's necessary for our national security.

Alan Rozenshtein: So I, I, I guess w- the part that I'm struggling with is I, I don't think even the most extreme China hawks would deny that if we could cooperate with China, that would be great, right? In, in the way that that the, you know, nuclear weapon, the, the, the nuclear theorists of the early Cold War did not deny that it would be good if the Soviet Union and the United States were buddies and we could throw all the nukes overboard. But we can't, and therefore we have to do this thing.

So it seems like the, the question, you know, is not so much “should we or should we not cooperate with China” or with any other rival power. It's do we have the institution system make that cooperation, you know, from a game theoretic perspective, right, another thing you've, you've written a great book about in the past possible.

And, and the reason I, I wanna frame it that way is because that gets to the kind of institutional, I don't wanna necessarily call it a solution, because I think you're, you're clear that it's quite tentative, but about what sort of global institutions it would take to have these kinds of durable cooperative arrangements that would allow us to slow down on AI and, and, and make sure that we're not being taken advantage of by, by the other guy.

It, it, to me, reading your- the book, it seems to come down to, you know, ultimately a kind of, you know, world government-ism. And I, I, I want you, I wanna ask if that's a fair- you know, fair description and, and what you mean by that, and also what you view as some of the potential dangers of that, 'cause you're, you're pretty clear-eyed about that as well.

Robert Wright: Yeah. I mean, I, I prefer the phrase “global governance” to “world government,” first of all, because it seems to arouse slightly less antagonism, but also because the phrase world government to me does connote more in the way of centralization. And I agree, centralizations of power are always dangerous. They are especially dangerous with AI.

And this is the great dilemma we face generally, I would say. This is the, the, the, the needle we have to thread, not just at the global level, but at the national level. You know, with you know, I, I think with this Mythos thing and the White House saying that suddenly actually they are interested in exerting some control over AI. I, I think one virtue of Trump being president right now, and I m- can't think of a ton of them, but one virtue is that it does focus people's attention on the fact that, that although, yes, I think AI does need to be governed, including by the national government, there are perils associated with letting the executive branch get too much control over AI because it's such a powerful thing. So I hope, I hope we're focusing on that.

Same thing exists at the global lev- level, except more so in a way 'cause there's only one planet, right? If you want, if that turns into a totalitarian state there aren't any competitors out there to rival it. So, I, I prefer to talk about, you know, international governance, global governance- global governance in some cases, not always. I mean, there's a lot that just the U.S. and China can do. That does, that's not global governance. That's, that's you know, international governance.

I think we should keep the thing as decentralized as possible, as democratic as possible in terms of giving, you know, the world's nations continued input and, and say. And I, I don't wanna get any closer to global governance than is necessary given the technology, but my view is we're gonna have to get a little bit more in the business of international governance, and this is a huge it's a huge challenge. I don't think that's your only que- I mean, you're also asking about China specifically, there's still, there's residual skepticism on that front.

Alan Rozenshtein: Yeah, well, I guess what I wanna know is, is how we get to this kind of international cooperation. Maybe that's a nice segue to the, the last part of your book, which is quite unexpected, I think. And you're reading a book about AI and geopolitics and and all of that, which is, and again, please push back if I'm mischaracterizing this, that you know, to, to deal with a problem of this magnitude, we need to go on a bunch of meditation retreats and develop, develop our powers of enlightenment.

And to be clear, I, I say that as someone who has been quite influenced by your own previous writings on Buddhism, right? You have this wonderful book from a few years ago called “Why Buddhism Is True,” which of, of all, of all your books, I think is, is the one that I'm, is, is closest to my heart. So I take your general point extremely seriously as a way of living. It's just not something I expected at the end of a book about AI and global competition.

So I, I'm just so curious of, of why you decided to end your book with this sort of call for consciousness raising, and if, if you think that that could possibly work. I mean, I want it to work. Yeah. I'm with you. I mean, yeah, but I worry.

Robert Wright: Well, I mean, first of all, mindfulness is not the only path to the perspective I'm advocating. It's the path that I have most used, but the perspective I'm advocating is just a more objective perspective, right? Less influenced by our own very natural, ingrained by natural selection self-righteous cognitive biases, right? We all are inclined to think we're right, adversary's wrong, our nation's right, the other nation's wrong.

And, you know, it gets back to you saying, well, or suggesting this, like, wait, how realistic is this? I mean, look at the, look at the mindset in Washington toward China like, you, you know, you expect a, a 180-degree turn. I mean, exactly. That is the magnitude of the challenge.

And I, I think one reason the challenge is so steep is because, you know, humans are, you know, we are born with these cognitive biases, some of which together constitute what I would call the psychology of tribalism. And I think if we are going to have a view of other nations that is conducive to the kind of restructuring of international politics that I think is a prerequisite for our continued survival and flourishing, you know, that's the argument I'm making, then something fundamental is going to have to change.

And we're just gonna have to, at, at one level it's not that complicated. We have to get better at viewing things from others' point of- other people's points of view. Cognitive empathy, it's a hobby horse of mine. I'm not talking emotional empathy. I'm not talking about feeling their pain. I'm not even talking about saying “they're, oh, they're right, we're wrong.” I'm just, I'm just saying start out by, by understanding that the way we view America is not the way the world views it. Okay? So that, that, that can be a path to various kinds of diplomatic progress.

And let me give you a couple of examples in the context of China. So just last week, or sometime after the Trump-Beijing summit, Nicholas Burns, who was ambassador of China, was on a podcast. They said, "Well, do you have any criticisms of Trump?" And he said, "Yeah, he didn't, you know, he didn't take them to task for throwing their weight around regionally." You know, these disputes, territorial disputes over islands and, like, bumping Philippines boat. You know, you, you're familiar with the, with the kind of heavy-handed stuff they're doing, and it is heavy-handed.

And I just thought, I mean, do you understand that we just invaded Venezuela, okay? We have a blockade on Cuba, not just sanctions, now an actual blockade. Plus we invaded Iran, and, like, we're gonna lecture? I, I, I'm not saying, I'm not even saying hypocrisy is bad and I, I, don't give me, please do not throw the word whataboutism at me, I have a whole, I have a whole spiel on that. But, but the point is just that the first thing you need to understand is that if Trump had brought that up, they would la- have laughed him off the stage, and rightly so. I mean, once you have invaded Venezuela, you just can't say, "Don't bump Philippines boats," right?

I mean, the, the, so I'm sorry to get excited, but this is so chronic to just not... And let me give you another one, one other example. So, Dario Amodei and a lot of people in the AI safety community who are China hawks, and I think most of the people in the AI safety community are actually China hawks. Most of the influential people. They are motivated by the concern that, that China wants to impose its system of government on the rest of the world, and that- so if it is the AI, great AI power, which by the way, I'm not advocating. I'm advocating that we talk things through so that neither of us is super dominant in the realm of AI, and there's a kind of a balance there. But anyway, the fear is that if they beat us to super intelligence or something, they will immediately use it to impose their system of government on us.

First of all, I would say that is a, is a consistently unexamined assumption in, in Washington discourse that China even has that aspiration. I personally think to some extent it is projection. We manifestly like think that other countries should have our system of government. We've invaded a number of them and sanctioned a number of them to try to steer them in that direction. It is very unclear to me that China is, has that goal. I understand some of the evidence that people interpret as, as corroborating that hypothesis, and I'd be happy to argue with people about that. In general, I just wish there was more argument about the fundamental premises underpinning our China policy.

But, like, related to that, what I wanna say is, okay, so granted, maybe that is one thing you could worry about. Maybe you could look at China and go, "That's why we don't want them to have super intelligence." Okay, so you want the U.S. to have super intelligence.

Now let's adopt a different perspective, okay? Let's suppose you're on Mars, and you're just looking at the planet from a distance, and y- y- you know, you understand the way history's unfolded. There are these nations. They have wars, and you know, it's too bad. They're apparently not a, not a very enlightened species, whatever.

Like, which nation do you not want to have, like, utter dominance in AI? Well, I can imagine a Martian looking down and saying “well, you know, China has not staged a military assault, like an invasion or a bombing on another country since 1979.” The U.S. has done it on average at least once per pre- per president and, and, and that's if you count only the illegal ones, you know, in international law, the ones not sanctioned by the Security Council. You know, we, we- in the last year, we've invaded Iran, I, I mean, bombed Iran, invaded Venezuela, and on and on, invaded Iraq, blah, blah, blah. It's been, like, for the last half-century, China hasn't done any of that stuff.

So, and I'm not saying, you know, well, then we should give them the power, not us. I'm just saying from a perspective other than the American perspective, so many of the premises that people like Dario Amodei are carrying into these conversations are just not self-evident, and it might be a healthy exercise to ask ourselves, like, what do things look like from the point of view of another country? Or what do things look like, you know, from, like, beyond planet Earth if you, if you tried to get an actually objective perspective?

I, I know, I know this has been a long detour and I'm doing a rant, and I'm sorry. I get agitated. But the point is, like, if you ask, why do I bring up mindfulness and various other things as, like, paths to greater cognitive empathy, getting better at perspective taking? I think unless people in general, not only America, I'm, I would say the same thing to Chinese, you know? I know you have your national perspective, try to transcend it because the whole world will be better off if we all get better at that. That's, that's all I'm saying.

Alan Rozenshtein: I think that's a great place to end the discussion. Bob Wright, thanks for coming on the show, and thanks for writing a really terrific book about AI that I encourage folks to pick up when it becomes available.

Robert Wright: Thank you, Alan. I appreciate your, your reading it and, and giving me the, the airtime.

Kevin Frazier: Scaling Laws is a joint production of Lawfare and the University of Texas School of Law. You can get an ad-free version of this and other Lawfare podcasts by becoming a material subscriber at our website, lawfaremedia.org/support. You'll also get access to special events and other content available only to our supporters.

Please rate and review us wherever you get your podcasts. Check out our written work at lawfaremedia.org. You can also follow us on X and Bluesky. This podcast was edited by Noam Osband of Goat Rodeo. Our music is from Alibi. As always, thanks for listening