Lawfare Daily: Cullen O’Keefe on "Chips for Peace”—AI Supply Chain Governance

[Intro]

Cullen O'Keefe: If we're going to create a network that will simultaneously allow this technology to be deployed when it is safe to do so, and also to prevent unsafe deployments or unsafe usages of the technology that we will be able to do that most effectively if we have broad cooperation from a large number of countries.

Kevin Frazier: It's the Lawfare Podcast. I'm Kevin Frazier, assistant professor at St. Thomas University College of Law, and a Tarbell Fellow at Lawfare with Cullen O'Keefe, research director at the Institute for Law and AI.

Cullen O'Keefe: And so you have a case where the company that makes the best AI that is good at automating AI research is able to much more quickly expand the capabilities of its AI system. And it's kind of unclear what the limits or bottleneck factors to that are.

Kevin Frazier: Today we're talking about Cullen’s recent proposal to govern AI through regulating its supply chain.

[Main Podcast]

Full disclosure Cullen, I've been reading your research for quite some time, at least since I served as a summer fellow at Law AI. It's fair to say that you were looking into how to regulate AI way before it was cool. Your most recent research keyed in on this novel idea, Chips for Peace, involving regulation of the AI supply chain. For listeners who haven't yet read your most recent Lawfare piece, can you give us a zippy rundown of your proposal?

Cullen O'Keefe: Thanks, Kevin. And yeah, I should say that the piece is not entirely novel in the sense that I think it largely pieces together a collection of ideas that have been floating around in this space for a while. So I don't want to take full credit. I'm mostly just trying to articulate them in a way that most people will find compelling.

That said, the proposal is, as the name suggests, an adaptation of the atoms for peace style deal that the U.S. led last century for the regulation of atomic energy, another famously dual use technology. The idea is that Chips for Peace, much like atoms for peace in the IAEA, would have kind of three main pillars. The first is safety regulation, the second is benefit sharing, and the third is non-proliferation. And yeah, happy to go into each of those.

Kevin Frazier: We will dig into the weeds soon. I guess I wanna start at a higher level first, which is to say AI governance is typically framed as having kind of three legs of a stool: data, algorithms, and chips. Why do you think chips are so worth focusing on from a regulatory standpoint, perhaps above the other two legs? So now we have a wobbly stool, but we will deal with that issue later.

Cullen O'Keefe: Yeah, I mean, I, I, I don't know that I would endorsing that it's worth focusing above the other pieces. They all are important to regulate in various different ways.

That said, when it comes to the most compute intensive models, so-called frontier AI, and I'm happy to go into that term and why we use it, those, those are models that require large amounts of computing power to create. And as we go into, into this report that we released earlier this year, compute of those three legs of the triad are, is by far the most regulatable of the components.

The other two are essentially information goods. They're, you know, zero marginal cost to reproduce. A lot of the data that is used to train the frontier AI models is available on the open internet. The algorithms are often published, although increasingly less so. But you know, there's fluidity of people who know about the different algorithms that are used, and chips by contrast, have a famously brittle supply chain concentrated fortunately for global democracies in a handful of democratic countries in East Asia, Europe, and the U.S.

And it's that group of countries that if they wield, it could form a backbone to a global governance mechanism that could set ground rules for how the downstream uses of that compute should be governed and, you know, I think that that is a quite a extraordinary power that they have and this is a proposal for how they could use it. I think that would be legitimate and hopefully beneficial both for those countries and anyone who wishes to join them.

Kevin Frazier: So when we're thinking about this concentration, just how concentrated is the supply chain of compute in these nations that you're highlighting, the U.S. and its allies? Is it close to parity with China and let's say perhaps a more adversarial nations or just how concentrated do we see, see that supply chain?

Cullen O'Keefe: Yeah, the supply chain is really extraordinarily concentrated. So for the most advanced chips you need a machine called an extreme ultraviolet lithography machine, EUV. Famously, only one company is able to make those machines, ASML, which is based in the Netherlands. There's a multi-year backlog of orders for those machines. They're famously complicated. I think one article describing how accurate the kind of light that these machines use to etch features onto chips is equivalent to shining a laser from the earth and hitting a target on the moon the size of a thumb.

So that's the, that's how advanced they are and with that level of precision, they're able to carve many more features into chips, which, you know, makes them the chips faster than any kind of competing chips. I should say, that China has kind of unexpectedly been able to make breakthroughs using older generations of lithography techniques and has been able to make at some scale seven nanometer chips, which was not expected by a lot of observers.

So, I, I don't want to claim that this is a extremely stable lead, but I think most people expect these countries to have a lead in that technology for quite a long time. As a, as another piece of evidence for this, Japanese engineers and firms were very involved in the development of EUV lithography machine. They were Japanese firms were quite good at making the older generation of lithography machines and tried to make EUV machines in the same way that ASML did. And were just unable to, despite having a, a long experience in the kind of precursor technologies and indeed active involvement in making EUV machines.

So in that stage of the chip supply chain, there's 100% concentration in a close U.S. ally, the Netherlands. At other stages you find similar levels of concentration. So TSMC fabricates, 90% of advanced chips and their only major competitors are Samsung, which is based in South Korea and Intel in the U.S. and overall research from CSET shows that the U.S. and democratic allies in Europe and Asia capture about 90% of the value in this value chain.

And it's not for a lack of trying of by Chinese firms in particular, they've subsidized their domestic chip making industry, very heavily. And you know, I think it's very much an important question, an open question when and whether they'll be able to catch up. But suffice it to say for now, the U.S. and its allies have a substantial lead. And this piece is really building an idea of what we can do with that lead.

Kevin Frazier: And again, to keep things at a high level for just a second longer. This is such an important finding, just how concentrated the supply chain is, because to your point, it's not even close, right? This is 90% or upwards across various points of this supply chain.

Do you think that this has been a underappreciated area to regulate AI? And if so, would you attribute it to just a lack of understanding or what's going on here? Why isn't this front page news of the New York Times of brilliant straightforward solution to regulating AI presented by Cullen O'Keeffe. Let's do it.

Cullen O'Keefe: I'm not the first person to realize this. There's been a lot of compute governance, as we call it, going on. The Biden administration has worked closely with these allies to impose export controls on both the chips themselves, but also the underlying hardware to make those export controls on chips themselves sustainable.

This idea of compute governance has gotten a fair amount of traction both within the U.S., the EU, the U.K., other nations and jurisdictions at the edge of AI and also kind of people thinking about how to regulate the most compute intensive AI systems. So it's definitely not a unique realization and building off a lot of work, especially CSET in particular has done a ton of great work, really raising attention to this, this amazing fact of how much democracies lead in compute that's had.

If, and as I kind of predict the most compute intensive AI systems continue to bring with them a set of risks and opportunities that are qualitatively different from the challenges that lower compute systems face. The implications of our ability to regulate large amounts of compute will be increasingly salient and the if that prediction turns out to be correct chips for piece is a kind of idea of what we can do about those high compute systems.

Kevin Frazier: So in the same way that you're not the first to realize that compute might be a great way to govern ai, you're also not the first to use the analogy of nuclear regulation as an analogy for regulating ai. We've also seen some people turn on similar analogies. For example, calling for an IPCC for AI emulating climate change regulation, and others have pointed out the merits of something like a CERN for AI.

Still, others have looked to global aviation standards as a potential guide, and there's an argument to be made though that this sort of regulation by analogy might be a poor strategy given the distinct aspects of AI. What would be your response to this idea that maybe we shouldn't rely so heavily on outdated or simply too distinct technologies to look to for analogies when we're regulating AI?

Cullen O'Keefe: Yeah, I think there's a lot of merit to that. I think it's easy to make mistakes in either direction. I think you don't want to be totally untethered from the lessons of the past. You know, if we're thinking about major dual use technologies that I believe could have huge destructive and transformative potential, but could also carry a ton of promise, there aren't that many of those that humanity has confronted, and so we'll take historical lessons where we can find them.

It is super important to be attendant to the analogies and dis-analogies of which there are many for nuclear. The supply chain concentration is kind of the main analogy that we draw on, and in this compute governance paper that Girish Sastry, my colleague while I was at OpenAI, was the lead author on we go into what the analogies and dis analogies are in one of the appendices. So I encourage people who want to think more about this, look that up.

But yeah, some important analogies are, yeah, the supply chain concentration, the dual use nature, important dis analogies aree also many. So currently uranium is really just has two uses. It's either used in weapons or used for civilian energy. And you know, civilian energy usage is, it's not a major area of economic development for better or worse, you know, and compute in that sense seems like a lot more general purpose people use various types of chips in their phones, in video game consoles, et cetera. So depending on, you know, at what level of abstraction you look at, it might be a much more, much more broadly used technology.

Secondly, like it is just dangerous to carry around uranium in a way that it is not dangerous to carry around chips. And so I think that like fact is actually relevant to thinking about what it would mean to constrain and put, put limits on people's ability to get compute because it would, in fact be a much bigger imposition than carrying around uranium.

I still think there are cases in which it's worth it, but it's worth taking seriously that people have an interest in accessing compute, including on their person in a way that they don't in, in accessing uranium.

Kevin Frazier: Yeah, I typically don't bring uranium with me through the TSA line, but, you know, perhaps this could be a, a future use case to study to see how that goes.

So thinking about the difference too, between nuclear and AI, this is a little far afield from your paper. But what is your general sense right now of popular attention to the risks posed by ai. We've been talking about AI risks since ChatGPT. A lot of people have been talking about it for a lot longer before then, but that's kind of when we saw popular attention turn to AI risk.

Do you think we've hit a level of popular awareness that we need to take this risk seriously akin to nuclear energy? I mean, there's still no ads on TV that say, hey kids, this is how you duck and cover in your fourth grade. You know, science room when that nuclear bomb goes off, do you think we're just not seeing enough popular attention to AI as something akin to nuclear when it comes to preparing for and responding to risks?

Cullen O'Keefe: I'm pretty happy with where the conversation is right now. I think lawmakers and policy makers have been attentive listeners. They understand the basic theoretical case for there being extreme risks that this technology could carry. They also understand that the risks right now are primarily based on theories, and I think that being theoretical is not the same thing as being unsubstantiated.

But you know, right now I think we need to focus on building epistemic institutions to help us understand the technology better, understand the rate of progress of the technology, improve our ability to measure the capabilities of systems, and happy to get into some of the challenges with doing that, which I think is one of the big things that I'm worried about at the moment.

So overall I'm pretty happy with where the conversation is. I think it's appropriate for people to not be dismissive, to keep an open mind to the possibility that the upper limits of what this technology will be capable of are quite high and therefore carry quite significant risks. But also to note that, you know, we are not at that point right now for the frontier foundation models.

And, you know, we also need to understand the risks that other models carry in a lot of different contexts. And I hope that neither one of those conversations completely eclipses the other. And yeah that's kind of where I hope the conversation is. And I, I think we're, we're pretty close to that right now. So overall, I'm pretty happy with it.

Kevin Frazier: So you've spent some time in D.C. and you've seen congressional action and congressional inaction. The first part of your proposal turns on domestic regulation by the U.S. and its allies. What's giving you a sense of optimism or confidence that this regulation may be the one that breaks through in the U.S. and in some of these other legislatures around the world?

Cullen O'Keefe: Yeah, I mean, I, I wouldn't say that I'm necessarily an optimistic person on our institution's ability to rise to this challenge. I would really love to be proven wrong about that. That said, I think again, as the risk becomes more concrete and backed up by, for example, empirical demonstrations of what the technology is capable of, perhaps more rigorous threat models of the types of capabilities that would really change the amount of risk that the public faces from these models.

You know, I, I would hope that that would lead to significant urgency and consensus on the part of lawmakers. You know, I think I have been very happy to see that lawmakers from both sides of the aisle have taken this quite seriously. And it really, when it comes to these large scale threats to public safety and global security, it should not be a partisan issue.

You know, these are things that could cause mass casualty events if, if I'm right about what the capabilities of these systems will be in the future. And so when it comes to preventing that you would hope that there's going to be bipartisan consensus and urgency and also international consensus.

Kevin Frazier: So assuming we get that international consensus and we get the domestic regulation you're looking for, another big part of your proposal is benefit sharing. So listeners know I'm a bit of a law of the sea nerd and the UN Convention on the Law of the Sea or UNCLOS has an article related to the distribution of revenue from deep sea mining in the quote, end quote. A spot in international waters. No such mining has occurred in this area, and I think few anticipate that any sort of distribution will take place when it does.

You nevertheless propose to share the benefits of frontier AI among both the developers of the technology. And countries with more humble AI efforts. So let's break this down a little bit. What does benefit sharing actually look like? Does it look like revenue sharing? Does it look like technology sharing? What does it mean when we say we're going to share the benefits of frontier AI?

Cullen O'Keefe: Yeah. This may be something of an unsatisfying answer, but I'm really not sure I, my main goal with this piece is to start a conversation on what that could look like. I think the thing I want to draw attention to is that if we're going to create a network that will simultaneously allow this technology to be deployed when it is safe to do so, and also to prevent unsafe deployments or unsafe usages of the technology, that we will be able to do that most effectively if we have broad cooperation from a large number of countries.

As I said, the supply chain for this technology really, our allies, the U.S. as allies primarily dominate that. And so I think the idea that the U.S. is just going to be able to impose its will on people and get, be able to get people to play along with this without kind of providing an incentive to do so, strikes me as potentially both unrealistic and, you know, immoral in some sense.

And so, but I, I, I think it's worth taking seriously the question and yeah. Also like engaging with states whose cooperation we would want to need for this, what they would find a compelling, you know, deal or structure to do this benefit sharing rather than me deciding that. But I'm happy to just throw out some ideas for people's consideration.

So, you know, I think one, one kind of dynamic that people who are new to. The issues of compute might be interested in is that the types of data centers that you need to develop these very compute intensive models are different than the types of data centers that you need to deploy them. The latter can be much smaller in terms of the number of chips and therefore energy footprint.

They're called inference data centers. There are reasons that they can be smaller but they can, and building those locally in the recipient countries of these benefit sharing promises might be one example. And then providing technical assistance to make sure that the models are as useful for the context in which those models are being deployed would be a nice layer on top of that to make sure that, you know, every economy can benefit. I think it's unclear to what extent you need different models for different places, but I think it's very reasonable to think that that could very well be the case.

So like right now, the models are best in English. If we expect that to continue to be the case, then making sure that we are able to, for example, have models that work well in the native language of whichever country would be an important example.

Similarly, like understanding local context, you know, there's tons of great work talking about the way that these biases and broad internet of text and other modalities like really influence the model behaviors in way that make them, ways that make them less useful in different contexts and getting rid of that bias or working to create versions of models that are useful in different contexts is like an example of, of how we can do benefit sharing in a way that makes sense.

I think also, you know, this is a zero marginal cost to replicate technology. So I think as compared to the sea mining context where you're, you, you have a single lump of metal and there's just some choice about who gets how much of that lump of metal the cost of creating a new instance of GPT, whatever in whatever country is much lower.

Obviously there's infrastructure costs that you would have to that, that are substantial that you would have to do, and I don't wanna abstract away from that, but yeah, I am, I, I, I'm pretty convinced that you would need some sort of benefit sharing component of this to make it worthwhile for everyone to, you know, partner with the U.S. on this.

And the other thing I, I bring up in the piece also is that, you know, the U.S. has really been on its back foot in making inroads in a lot of parts of the global south as compared with the kind of aid that China has been offering without strings attached. I've heard the saying that like every time China comes, we get a hospital. Every time the U.S. comes, we get a lecture and we don't want to be seen as not having anything to offer.

You know, I think we want to build a global community that is community invested and stands to benefit from tackling these shared challenges by everyone bearing their part of the regulatory burden and therefore everyone bearing. Everyone receiving their fair share of the benefits.

Kevin Frazier: Well, as a professor, I do have to say that receiving a lecture sounds like a pretty good deal, but your point is well taken. And I think too, it gets to an emphasis on looking to, again, use analogies here, looking to social media for instance, where we saw the U.S. proliferation of social media around the world was arguably done in a haphazard way that meant we were distributing this technology without building the requisite infrastructure or assisting in the building of the requisite infrastructure around the world.

And maybe now's the chance where we can be more deliberate about benefit sharing. So I'm looking forward to the next two or three articles you write on that proposal fleshing that out. I know you've got nothing else in your regulatory hopper.

Cullen O'Keefe: Yeah, I, I, I think this is a really exciting area to think about and yeah, like I said, my main goal is to start a global conversation on this and would love to see people do creative thinking about what types of benefit sharing would be simultaneously just not destabilizing and attractive to states.

Kevin Frazier: So I wonder, one of the things that I'm theorizing is that the folks pushing for AI regulation have suffered a little bit from paralysis by analysis. There are just so many wonderful regulatory proposals out there that it's hard to just say, okay, let's rally us all around this one proposal. We're gonna push for it.

We're gonna cross that off, and then we'll go to the next thing. So how would you rank this proposal in terms of. Other proposals to STEM risks from AI. I am thinking, for example, of the right to warn that Larry Lessig has touted, you worked at OpenAI, he's representing some of your colleagues now, and he's been quite adamant about this, right, to warn and creating a culture of safety in labs. If you had to pick one or the other, which would you push for first: a right to warn or chips for peace?

Cullen O'Keefe: Without commenting on the right to warn proposals specifically, I think the main idea is that Chips for Peace would be a way to harmonize safety regulations, domestic safety regulations of whatever sort across this group of states.

Right? And so it's possible that whatever is a good component of domestic regulation with this technology should therefore be kind of harmonized at the international level. And that's the main idea. And I think we will need a lot of experimentation at the various national levels, at the state levels to figure out what we would want to be a component of international regulation and worth harmonizing.

It's not supposed to be a policy that addresses every risk. There's a lot of risks that arise in other contexts from other systems, non-compete intensive systems risks that are best handled at the local level or the state level, the, the nation state level, and where international harmonization is not as important.

And yeah, I, I, I, I guess I hesitate to try to trade them off against each other, even though there are obviously trade-offs eventually. It's more just to say that for this particular set of issues, the stakes are so high, and the downside of international heterogeneity that enables regulatory arbitrage around high safety standards set by any one jurisdiction would really just completely undermine some of the goals that such regulation would impose.

Kevin Frazier: So you anticipate as a sort of third leg of chips for peace that this would aid with non-proliferation, theoretically limiting the number of countries with high risk frontier AI systems. How would you take on the critique that this is merely denying? Other countries, the chance to lead in AI is this not just, you know, technology Neocolonialism 2.0 or 3.0.

We're saying, don't worry, the rest of the world, the U.S. and its allies will harness AI for good. Don't worry about it. We'll share the benefits down the road. It's all gonna work out. I think some might be, obviously I'm being facetious with that summation, but what is your. Response to that idea of, oh my gosh, here we go.

It's just the U.S. again, telling us that we can trust them to harness this technology for good.

Cullen O'Keefe: Yeah, I mean, I take that critique very seriously. I, it's why I put benefit sharing ahead of non-proliferation and why I emphasize it as a, an integral part of the three-legged stool. Not a, a mirror nice covering on top of it.

I come at this from the viewpoint that there are serious risks attending to this technology, and that importantly, those risks are global risks. So I take that perspective very seriously. But I think there's another perspective in which this is a technology that by default under the current trajectory, the Global North is leading in and is on track to internalize a lot of the benefits and externalize a lot of the largest harms on.

Global catastrophic risks from AI are a real thing that I, as I think they are likely to be in the sense of risks that are worth taking seriously and preparing to guard against those are global risks and. By default they would be externalized onto a, onto the whole world and therefore the kind of default trajectory without this is really quite unjust as well.

And it seems to me that the most just way from a global perspective out of this is to both have regulation that guards against global catastrophic risks. And I would want the kind of substance of the regulation to really hopefully be. Carefully circumscribed to limit itself to like truly global risks.

But then, yeah, you run into the risk that that is used as an excuse for economic protectionism or worse. And I think the way out of that is benefit sharing and that without that, it really does run a, a huge risk of being unjust. So I think that's a very reasonable reaction for people to have. I think if you think that the risks are real, that the way to reconcile that tension is through generous benefit sharing rather than a more laissez-faire approach to regulation that has its own risks for the global south, where state capacity to guard against the downsides of this technology might be lower.

Kevin Frazier: So we talk a lot on the podcast, or at least I do with my fun AI guests about the risks posed by AI. And just to take things on a different track for a second, what are some of the most positive use cases you've either seen from AI or anticipate we will soon see from?

Cullen O'Keefe: Yeah, I mean, I can talk about some that I see in my personal life.

So my wife is an English as a second language speaker. So, just for her, in her professional context, it helps a lot to make her English, which is already very, very good sound a bit more natural, and therefore to, you know, be more competitive with people like me who are native English speakers. And I think that's a.

Significant benefit. You know, the area in which these foundation models have, I think provided the most economic value is in software development. Yeah. I think most coders probably use these systems significantly throughout their. Professional careers and workflows. That said, I think the, the real question for how beneficial these technologies are going to be is how good they're going to be at aspects of scientific research and engineering that they currently have.

Not that great of capabilities at, but I think there's reason to think that they could. So I think it's an open question really, like how, how beneficial kind of, in the grand scheme of things these technologies are going to be, you know, economically people are apparently willing to pay significant amounts of money to, to use this technology.

So I think that that's decent evidence. And I think the, you know, they're a general purpose technology, so the purposes for which people use them can be quite varied. Many of them, I don't know about, many of them I can't speak to. But I, I think the biggest benefits and biggest risks lie in the future.

Kevin Frazier: That was, that was enough. Positivity. Let's go back to the darkness. So the internet is a buzz about the gains and gambles of AI agents, which you mentioned as one of the potential sources of accidental harms caused by ai. And I, I have a feeling that for a lot of listeners, this term, AI agents is kind of like.

You know, a supernova, something we talk about a lot, but we don't really understand. So what are AI agents and what is it about them that you think could lead to tangible harm in the world?

Cullen O'Keefe: Yeah. So again, it's worth thinking about general purpose AI agents as a hypothesized feature direction that this technology could take rather than a concrete thing that really, yeah.

Exists in any significant sense right now. The concept, I think, really gained popularity when people started putting various, what are called wrappers around GPD 3.5 and four like lang chain and attempted to make those things more agentic. So yeah. What do we mean by age agentic In a paper on which I was a co-author earlier this year, again, we, we talk about age agent as a kind of spectrum of properties that.

AI systems can possess to various degrees. It's about the degree to which an AI system can accomplish complex goals in complex environments with limited direct human supervision. So to. Kind of make that a bit more concrete, as people know, you know, we often talk about large language models as generative ai, and that's because their core competency is being asked to produce some piece of media, whether that's text or pictures and output that piece of media.

But, you know, that's obviously a very. Limited set of activities compared to what humans do. You and I are researchers. We have to, you know, go out and find the perfect articles to inform our views on things. This is not something that AI systems can very reliably do right now. We would like it to be the case that AI systems can do things, and I, I use this example in the paper of like being told that, Hey, I want to take a vacation with my family.

Can you help me do that? And so examples of what a more agentic AI system could do as compared to the systems that we have today would be things like reason through what it would mean that the system would have to do to set up a good holiday. And it might lead it to conclude that things like making sure that everyone is free finding a week where every member of the family is free.

You know, maybe emailing bosses to request time off, trying to find a dog sitter, booking flights, taking care of visa applications, things like that. So really receiving goals in a high level sense, coming up with a plan that. The agent thinks will be useful to accomplish that goal, being able to overcome roadblocks.

So if Turks and Caicos is fully booked, thinking through what another comparable holiday vacation might be, and being able to do that without the human holding its hand every step of the way. Right now people are mostly interested in software based agents, so think about any kind of action that could be done through a computer interface, which is obviously how most of us conduct a lot of our business today, I shouldn't say most of us, but a lot of people conduct a lot of their employment these days.

Rather than, you know, physical, physical interactions. But you know, over time, that could also lead to a greater capability through robotics to take actions in the physical world, or at least oversee actions in the physical world through cameras and things like that.

Kevin Frazier: So when we think about the. Specific harms that AI agents may cause.

What are some that stand out to you? I mean, besides not getting that beautiful vacation in Turks and Caicos because the agent was a little too slow, what is it that should make us fearful of this proliferation of AI agents?

Cullen O'Keefe: Yeah, I think there's a lot of things that are, a lot of risks that are worth taking seriously in this, I think.

The risks to like economic distribution of power are quite significant and we're thinking through, I think there's a large number of other risks many of which I'm kind of currently researching. But just to go to the kind of poor catastrophic risk that people are worried about. And give people some background on that.

You know, we talk about agents as goal directed entities. They, in the. Foregoing description that I gave, they have some goal that they're trying to execute on. And so the fundamental challenge called alignment is to make sure that the agent is following trying to achieve that goal only in ways. That we would want them to we being both the individual principle that is giving that goal, but also society as a whole.

So, you know, if I tell my agent to, you know, book us a nice my travel agent to book us a nice holiday vacation, you know, there are implied side constraints on what the agent is allowed to do. And the fundamental goal with AI agents is going to be to make sure that they are subject to those same kind of side constraints.

That they're not allowed to, for example, steal money so that they can afford a first class, first class ticket instead of having to fly economy. That that's like the, what's called the alignment problem. And I think you can see from the way that chatbots and technologies like generative AI are currently trained that we.

Actually do not know how to perfectly prevent these AI systems from violating the types of rules that we give them. So there are things called jailbreaks that I'm, I'm guessing most people who have followed LLMs are aware of where the organizations that develop and deploy these systems, you know, want them to be.

Cordial and respectful systems. But then because they are trained on broad swaths of the internet, and the internet can be an ugly place, they have also somewhere deep in their neural network, a disability to act quite mean. And people have found ways to circumvent the techniques that. The developers have put on the system to try to prevent them from acting means so already, our techniques for doing this sort of alignment when the stakes are pretty low are, are imperfect, but they also rely on our ability to kind of assess, is this the type of thing that we want our chat bot to be doing or not?

And both when the stakes are higher, those failures obviously get more consequential, but also. As agents take more complex actions, the ability to really understand whether what we're, what they're doing is good or bad becomes harder. So if instead of outputting, you know, five lines of code, it's outputting thousands of lines of code building a whole application, how do you ensure that the agent is not putting some backdoor into that system?

Because the. Russian government has asked it to or something like that. These are the types of kind of alignment problems and kind of fundamental AI safety problems that people are worried about as AI agents become more capable and are given more consequential kind of scope of action in the economy.

Kevin Frazier: And I think this is really important to put a pin on, is to say that we do need these sorts of tangible examples. And I think that was a, a really great one. But it just goes to show that there are real risks posed by these models. They may not become real tomorrow or the next day, but the more we can think about them and plan around them and regulate around them.

Then the better we can realize this sort of responsible AI development. And I guess with that in mind, another issue you flag is this winner take all dynamic of ai. What does that mean and why is that concerning to you?

Cullen O'Keefe: Yeah, so this again goes to the importance of benefit sharing. So AI technologies have a lot of features of a winner take all industry.

So. Like a lot of utilities, for example, they have a very high upfront cost. Which is the cost of training the model, which is tends, you know, trending to be hundreds of millions of dollars, possibly going to continue growing and, but very low marginal costs. So, you know, you can get a subscription to chat GBTI think it's $20 a month and you know, run it as much as you want.

And you know, we see that industries like this tend to be quite concentrated because yeah, there are high barriers to entry. Posed by the, the cost needed to train these models. And then the rewards are, you know, spread out over a long time after. And as the market becomes more competitive, you know, it's harder for additional entrants to earn back the training costs.

On top of that, you know, we talked about how software development is one of the things that these technologies are really good at, and so. One of the things that you might worry about is that AI systems will be quite good at developing new AI systems. I think, you know, again, because they're so good at software development and AI development is fundamentally a software development task not only is it the case that, you know, they have these classic winner take all dynamics like kind of a utility.

Business model does, but that they, they really like entrench the winners in the first movers in a way that, yeah, I'm not sure how, how well unprecedented this dynamic is. And so you have a case where the company that makes the best AI that is good at automating AI research. Able to much more quickly expand the capabilities of its AI system.

And it's kind of unclear what the limits or bottleneck factors to that are. But that's one thing to, to worry about. And Yeah. I, I think given that the first movers and leaders in this space as it currently stands, are concentrated in the global north in the us that, yeah, I, I, I worry for the kind of distributive impacts of, of what this will mean, especially as more and more jobs become susceptible to displacement from the capabilities of these technologies.

If that is indeed what happens, you know, we've seen. Plenty of past cases where people were worried about automation that turned out not to on net destroy jobs. So I think it's worth taking seriously that we don't know whether or when that will happen.

Kevin Frazier: You know, if there's one hobby horse that I've been riding pretty hard, it's that this displacement by AI needs more research and needs more popular awareness.

So consider this a, A call for podcasts. For all those working on displacement, reach out. We'd love to have you on the pod. Colin, one final question. You've been in the belly of the beast. You worked for open AI and now you are wearing a different hat at law ai. How are you enjoying the freedom that comes with being a researcher not within a specific lab?

What's, what's it been like? Having a bit of a broader purview perhaps?

Cullen O'Keefe: Yeah. It's been, it's been really good. You know, I, my main interest is in shaping public policy, and I think it's reasonable for people to be skeptical of voices from within industry and what they want. I think there's tons of people at OpenAI who share my interest in.

Creating a future in which AI technology is safe and beneficial. But you know, I, I think it's also important that there is a lot of talent spread out through independent parts of the ecosystem. And so trying to add my voice as an independent commentator on these issues yeah, has been really rewarding.

Kevin Frazier: Well, we'll go ahead and leave it there. Thanks Cullen.

Cullen O'Keefe: Thank you so much, Kevin.

Kevin Frazier: The Lawfare Podcast is produced in cooperation with the Brookings Institution. You can get ad free versions of this and other Lawfare podcasts by becoming a law fair material supporter through our website, lawfaremedia.org/support. You'll also get access to special events and other content available only to our supporters.

Please rate and review us wherever you get your podcasts. Look out for our other podcasts, including Rational Security, Chatter, Allies, and The Aftermath, our latest Lawfare Presents podcast series on the government's response to January 6th. Check out our written work at lawfaremedia.org. The podcast is edited by Jen Patja and your audio engineer this episode was Noam Osband of Goat Rodeo. Our theme song is from ALIBI Music. As always, thank you for listening.