Lawfare Daily: AI Targeting Systems Are Coming—But Not as Fast as You Think

[Intro]

Steve Feldstein: Yes, there are growing capabilities linked to AI that affect the battlefield and war, and yes, we should be concerned about that, but we also shouldn't jump to conclusions without understanding in a more granular detail what it takes to make these systems work in the way that we saw in the Iran war.

Kate Klonick: It's the Lawfare Podcast. I'm Kate Klonick, senior editor at Lawfare, with Steve Feldstein, senior fellow at the Carnegie Endowment for International Peace.

Steve Feldstein: So I think we'll see a lot of push when it comes to AI integration on the edge, when it's on devices that are being used directly in battlefield. But for a full-fledged comprehensive system like Maven, I think that's still gonna remain in place for a small group of, of high-capacity militaries for the near future.

Kate Klonick: Today we're talking about his recent article in the “Bulletin of the Atomic Scientists” on AI targeting systems.

[Main Podcast]

So I thought that your take in this piece was super kind of contrarian and a little bit, like, unexpected, and therefore, as I'm kind of a little bit of a contrarian myself, valuable and kind of, like, put up, like, sent out my spidey sense, as it were.

So there's kind of an ambient panic in the room for people that are not familiar with this space, which is that military AI is extremely real, it's consequential, and it's gen- but it's like, it's also genuinely hard to build. It's hard to get the components, it's hard to get the, like, the, the factories and the s- and the s- the setups to make the components turn into usable products.And then, so, like, let's just lay out the whole thesis before we, you know, you start pulling the threads that you do so well and so quickly and clearly in this piece, which we'll link to in the show notes.

So the conventional wisdom right now is post Iran war, post Ukraine, that, like, your title is doing a lot of work: “AI targeting systems are coming, but not as fast as many assume.” So what is the story that everyone else is kind of, like, hung up on and grasping to and is, like, very much has been in the ether for almost as long as I can remember? What is that story getting wrong?

Steve Feldstein: Yeah. So, you know, I think that with the Iran war, there's been so much attention paid to how the targeting and the kind of pace of strikes have occurred, and I think the question on everyone's mind is, well, is this now the future of war? So if you look to other conflicts future conflicts, battles around, you know, is this how war will now be fought? Will we see something happen where every 70 seconds another target is struck, whether it's, you know, in theaters like India, Pakistan, or China, you know, in, in another theater or something, another conflict in the Middle East? Is this the new way that things are gonna look?

And so I wanted to kind of delve beneath that a little bit and sort of ask some, some questions. You know, in particular, how, like, how do these, all these different pieces work toge- together. To what extent is this replicable? What do you need to actually make an AI targeting system work at scale as effectively as what the U.S. showed in the Iran war? And so that's where I got some interesting insights as I kind of delved beneath the, the surface there and kind of looked at all the component pieces necessary to make the system operate in a fast, in a fast-paced and destructive manner.

Kate Klonick: Yeah, and so you kind of open with this staggering data point that during the Iran war, which was 38 days I think by the time, when, at the time that this piece published, that the Pentagon's use of Maven surged to something like, what, 20 billion tokens a day? And it was a over 4,000% increase and 13,000 targets struck.

And if the diffusion story is over-hyped, how do you square that with numbers like these, though? I mean, that's a huge amount. That's a huge amount of kind of the reason that people, I think, see numbers like that and then say, "Oh my gosh," like, "this is an absolutely incredible amount of warfare." I, I don't know, is that, should we contextualize that? Is the argument that the U.S. is exceptional or that the numbers themselves overstate what AI is actually doing in the supply chain?

Steve Feldstein: Yeah. Well, so I, I mean, I think the question for me really is, the U.S. clearly has been able to strike a lot of things. Now, you can make an argument about whether they struck the right things or not, and I think those are fair. There, there's still enough of a fog of war that we don't really know. But granted, they have struck many different military assets, period. Right? And the question for me is, can others do the same thing? Is this replicable? Are we gonna see this happen with other militaries? And the answer for me is not really, that this was unique.

Over time, you know, as the technologies embedded in these systems become more ubiquitous, as they get lower in price. As we've seen when it comes to other types of technologies elsewhere, sure, they will spread. But they're not gonna spread, it's not gonna be like an issue of turning the switch, where one day the U.S. has it and then in, in six months' time, Saudi Arabia, UAE, and everyone else has it right away. It takes a lot of investment, both in terms of the data infrastructure needed, training the algorithms appropriately, tying that together in a targeting system that is linked to a cohesive command and control structure.

So in other words, you can't just simply say, "Oh, we're gonna, we're gonna take this off the shelf, give it to X country, and now they have have the same capability that the U.S. has." So we can walk through kind of all the different levels beneath that. But the top line argument is essentially that, yes, there are growing capabilities linked to AI that affect the battlefield and war, and yes, we should be concerned about that. But we also shouldn't jump to conclusions without understanding in a more granular detail what it takes to make these systems work in the way that we saw in the Iran war.

Kate Klonick: Can you quickly give me a description of just like a kind of a, like, a two or three-sentence description of Palantir's like Maven just for people to like differentiate it between models like kind of Claude or things like that, like what Maven is and like how they can think of it?

Steve Feldstein: Yeah. So Palantir's Maven is essentially a user interface software packaging that helps to bring together different streams of data to facilitate decision-making. So it's something, you can think of it in a business perspective when it comes to allowing a person to move along decisions through clicks so that you can go from identifying a target to assembling a targeting passage towards a final decision about whether to actually authorize a strike to destroy that particular asset

Kate Klonick: Perfect. What I thought was like so, so useful in this piece is really kind of talking about what it takes to make an AI targeting system, which is something that I think most people don't understand, and also kind of frames up like the anatomy, as it were, of like a kill chain.

So if you could kind of walk us through what it actually takes to build these AI targeting systems, because I think most listeners imagine that you kind of, like, if they, if they're familiar with this problem, most listeners are imagining that you buy an AI model and you point it or train it on a satellite imagery, right? Or like you describe something basically closer to a many decades-long national infrastructure project, and I thought that that was like, it was just a really great way of really kind of building out the complexity and scale. So if you could just talk about that a little bit.

Steve Feldstein: Yeah, sure. No, I mean it, and the first thing, this, and a large part of this actually relates to my earlier work, research I've done when it comes to surveillance systems because there's a lot of similarities there, and that's where I kind of got the idea to kind of think about how that wor- relates from one to the other.

But essentially what you need is you need to have an intelligence surveillance and reconnaissance capability, ISR as, as it, as it is, in order to actually have data that's usable. So, what, what do we mean by that? So it means especially when you're looking at dynamic targeting, which is the idea that as you're in a war, new targets pop up, new information comes up that allows you then to position assets and munitions to strike those targets.

So, for example, let's say you're looking at a command and control cell or several high, high, you know, senior generals and you're looking to strike that, that group. Well, they're moving constantly, and so as new information comes in, that then gives you a better understanding of where they are, where, let's say, a missile battery is. A- and so it's not something that you can just sort of assume will be in the same place that it was maybe three months prior. So that's what you need, what you mean, what I mean by dy- dynamic targeting.

So how do you, how do you get that information? Well, that requires inputting tons of data, right? So that means not only looking at satellite imagery kind of in real time, it also means examining signals intelligence, intercepting phone calls. Oftentimes social media. Telegram has proven to be a great trove and resource, especially in the Ukraine war when it comes to identifying where different Russian military assets currently are located.

So you put that all, all in. In order to get that information, that takes a pretty high technological capability. It's not like you can just sort of make it up overnight. I mean, you really need to have a pretty deep-rooted surveillance apparatus, and the more deep-rooted, the more accurate it is in terms of feeding into the algorithm these different data points.

Kate Klonick: And, and before you put all that in, right? Like, you need to vet that information. Like, this is, like, coming, I mean, is that, is that part of that, or is it not part of that? Like, I kind of feel like there's a question of like, you know, it's not, obviously it's not just satellite imagery. You just kind of spelled it out beautifully. But, like, does that information, is that including the vetting time that you have to do with all of this information before you feed it into the system? Or do you feed it all in and, like, you're trusting the AI system to effectively sort it out and vet it for you?

Steve Feldstein: You know, I honestly don't think there's that much vetting that occurs because, and partially because we're looking at so many different points of data. I mean, essentially what we're talking about is raw intelligence coming in, right? So if you're, you know, intercepting phone calls, I mean, to what extent are you able to vet and say, "Is that exactly the person we think?" I mean, you have a reasonable certainty that, that, you know, these interceptions are correct, and you put that all in.  And what you actually hope for is essentially you're put- inputting a bunch of noise, and then you're hoping the system will be able to kind of pierce through the noise and derive insights.

But I mean, the, the, the reason you have a system in the first place is that there's so much data coming in, it's impossible for humans to sort through that. I mean, that was the original premise for what, you know, Palantir essentially came up with in 2003 when it was started up. They basically said you had all this data when it came to potential terrorist activity related to 9/11. You were unable to sift through that because of, of just human capacity limitations. Let us provide software that can do that for you.

Now, does that software work accurately? Are there errors associated with it? I mean, that's a whole other question. But the, the general idea is that there's way too much data and way too little analytic capacity, especially at the front end, to do something with it, which is why then you sort of feed it into the algorithm and then you try to then sift through it throughout that process to figure out what's actually legitimate and what isn't.

Kate Klonick: Great. So I'm sorry I cut you off, but I just kind of want to loop back. Is there, so you were kind of going through the supply chain.

Steve Feldstein: Yeah. Right. Okay, so you, so you have this like, you know, you want, you're building up this surveillance apparatus over time, and, and frankly, the kind of more in-depth and, and the better you're able to make it to have it constructed, the better ultimately the, the target generations will be at the end. That's the theory of the case.

So you, so you, there is an incentive to make sure the information coming in is good. You don't want just bad information, but you also recognize that, that you're balancing with the fact that more information is generally a good thing. So the more data that comes in, hopefully the right information will sift through. So that's one piece of it.

The second sort of related piece to that is then actually having a well-trained algorithm that can identify, particularly if you're, you're trying to identify military objects or individuals something that's able to kind of better filter through what's real or, you know, color combinations or so forth when it comes to computer vision and what's not.

So an example is this idea that when the U.S. military was developing Maven, they found that models that had seventy percent success rates in theaters like Afghanistan that are dusty, desert-like, and so forth plummeted to less than thirty percent when they applied it to theaters like the Philippines, which are, you know, dense rainforest foliage, right? And, and, and so you need to, context matters, and you need to have an algorithm that is properly trained.

And so to do that takes a lot of time. It's, again, it's not impossible, but each one of these steps requires dedicated personnel feeding in labeled images, making sure that you have an algorithm that is properly attuned to the conflict and terrain in which it's operating.

Kate Klonick: Yeah, I actually was gonna bring up that incredible Maven example because it reminded me very much of kind of the flaw of like, well, I don't know if you remem- you remember this from, like, your civics and American history, but, like, the flaw of the Revolutionary War, which was, like, the British marching in with, like, their red coats and, like, becoming, like, literal targets versus, like, the, like, the guerrilla revolutionaries obviously, and rebels.

And it just like, this was like, it just was like, wow, so much has changed, and then still so much has not changed that we have, like, there is just something so simple as, like, what are people wearing and, and in what context? And then it completely changes the game of war.

Steve Feldstein: Yeah, no, absolutely. And, and look, en- you know, enemies have an incentive to try to hide and camouflage as much as they can. I mean, at, at this point, this has become kind of a key part of the battlefield in Ukraine, where you have different ways that soldiers and who are in, like, small little encampments try to blend in so that they detect, you know, identification from drones above and so forth. So, you know, even a well-trained algorithm will struggle at times to find accurate imagery, let alone one that isn't right for the context.

So that, you know, that, that's, that's another input in the kind of targeting value chain that requires investment. But then, you know, the sort of, kind of infrastructure piece that I think is also really interesting is that the more data that you collect the more phone calls that you're storing and then trying to sift through, the more you actually need the right physical infrastructure, the compute infrastructure in which to then analyze this.

And this is, you know, an interesting example really came up in terms of Israel's campaign particularly in its, you know, war in Gaza, in Lebanon, in other sorts of places where it bumped into this very problem. It had developed a surveillance capacity that allowed it to, you know, ingest like a million phone calls a day or so forth, intercepted from, from Palestinians residing in West Bank and Gaza.

But it soon ran out of space on its servers internally, right? So it didn't have the data infrastructure, and we're talking about one of the most sophisticated militaries and countries in the world. So what they had to do was turn to Microsoft as, and, and some other companies, but Microsoft in particular and basically say, "Look, you know, let's come up with a contract so that Unit 8200, you know, the kind of Israelis equivalent to the NSA, would actually be able to store this data and then process it on Microsoft Azure servers based in the Netherlands and Ireland," right?

So this shows you that, I mean, it's not, again, something where you can just, like, sort of come up with an off-the-shelf system, have a bun- you know, intercept a bunch of phone calls and be, you have to store it somewhere. You have to do something with it. You need a, the compute capacity, and that costs a ton of money.

I think Project Nimbus, like this, Cloud computing contract that was signed between Google, AWS, and the IDF, the Israeli Defense Forces, in 2021 is a $1.2 billion contract. It provides a suite of machine learning tools other sorts of storage and infrastructure capacity, but we're talking about really large expenditures that a lot of countries will struggle to match from a resource perspective.

Kate Klonick: Yeah. So this is also something I love, which is unfortunately, like, f- unfortunately or fortunately, it's always about logistics. You know, one of the things that I think is really incredible is, like, just people just because they're just dealing with these systems and these models or these chatbots in front of their own computers, it just seems it's like running on your computer, and they're, like, missing from this equation is just, like, the apps, like, the petabytes of just huge, huge swaths of, like, of resources and data and energy infrastructures that are, like, invisibly falling into place, but, like, not so invisibly because we're seeing all of this controversy around data centers and everything else.

But, like, I, I mean, just it's, it's really true. It's like these are not just the, like, to, to create this artificial intelligence, to create this capacity to crunch this kind of data in the way that human beings would crunch it is, like, costing us billions and billions of dollars. Like, we're creating artificial human intelligence, sure, and it's v- turns out it's very expensive. Like, it is-

Steve Feldstein: That's right. That's right.

Kate Klonick Very-

Steve Feldstein: Yeah.

Kate Klonick: Very expensive to build an artificial human brain that is, like, as good as Steve Feldstein's. Like-

Steve Feldstein: Or, or Kate Klonick's-

Kate Klonick: Right.

Steve Feldstein: To be honest.

Kate Klonick: Right.

Steve Feldstein: Yeah. But no, but I was exactly, I was just gonna make that exact analogy to the data centers, 'cause I think there's just cognitive dissonance where on the one hand, we don't want data centers. We're angry about them. Like, justifiably, we are worried about the environmental damage. We're worried about the rise in electricity costs.

On the other hand, like, don't take away my chatbot, right? Let me use that as much as I can, but get, keep those data centers out of my backyard. I mean, you know, you, the sort of one goes with the other, but we forget that when it comes to these, these complex systems.

Kate Klonick: This is a, a story that, like, is very familiar to me. For, I've studied speech platforms and private governance my whole life, and I'll kind of get to that. That was kind of where my question was gonna go.

But I was just gonna also say that it's a very familiar question of people want the good parts and want to take away the bad parts of technology, and they do not understand how difficult it is to extricate, either through policy or even in practicality, if you can write the right policy to, like, incentivize the correct practicality, these systems to preserve the things that we like about these technologies without having these negative externalities, if it's even possible at all.

Steve Feldstein: Yeah.

Kate Klonick: Most of life has negative externalities to, to certain types of convenience and certain types of benefits that we create. And so, like, I, you know, trying to hit that balance, I mean, like, it, you know, took us, like, 100 years with cars to make them safe and things like that. And so I just, you know, I kind of wonder about that all the time. But anyways, that's a different, we can talk about the history of cars in a different podcast, Steve.

Steve Feldstein: Right.

Kate Klonick: Re- you know, reading your piece, it really struck me, particularly the private part of this, right? And so I talk about, like, kind of, I've talked about the private governors of speech, but here you kind of have a lot of the same players, but not in their speech capacities, just in their private company capacities. Google, Amazon, Microsoft, OpenAI, Palantir, Anthropic have become something more like kind of private governors of this kill chain, essentially, right?

And Israel's targeting capacity, as you're describing it in the piece and you just kind of described it now, runs substantially on American commercial cloud and AI infrastructure. So, like, who's gonna be accountable in this arrangement? Who is the best target for accountability when you have something like this? I mean, is this what we're going to think of, like, the defense kind of supply chain in the future? Is it going to be not, like, who has access to, like, Patriot, like, kind of systems or whatever to build actual, like armature, but like, kind of like who has access to Microsoft Azure?

Steve Feldstein: Yeah. No, I mean, it's a great question, and there's not an easy answer to it because I think the way the system works, it essentially diffuses accountability to lots of different players. And so, and I, and I, you know, again, I've looked at this from, like, different angles, from like a surveillance angle. You know, when you look at big systems that are being used in, in countries, like, do you blame the data servers? Do you blame the police forces who are weaponizing them? Do you blame the agents on the ground? Like, who is responsible?

And the answer is, like, there's a little bit of responsibility all around. So like, let's say we look at this targeting system. On the one hand, you know, it's not fair to sort of say, "Microsoft, your infrastructure is responsible for, you know, largely or, or wholly responsible for strikes that are being carried out by drones that are manufactured by domestic Israeli companies that are then used in, in Gaza targeting."

Yet, I mean, there is a crucial aspect to the supply chain that Microsoft has provided, and when called out upon it, reputationally, by a Guardian investigation, they have pulled out from the contract. So clearly they feel that it's not, you know, they, they have some amount of culpability, even if they are not sort of directly responsible for the end result of that.

So that becomes complicated. I mean, I think sort of as you get further down to the kill chain and as you get to the actual weapons that are being used.  So, you know, the, the thing is, you have all the state of infrastructure in place, you have all the surveillance apparatus that kind of comes in.

At the end of the day, you still need two things, right? You need to have a software interface or some way to make decisions so that what information that's presented and the targeting packages that come together are then actually used for something, to destroy something. That's where Palantir's Maven comes in. So you can make an argument that that interface that puts it all together, that compresses the kill chain, that gives you four clicks towards destroying a target, that's probably closer in line, at least from a software perspective, to the end product than, you know, the data infrastructure behind it. And then, from there, you still need the handover to a drone or a ballistic missile or you know, a manned aircraft, or something else that actually will deliver the munition that destroys the target, whatever that tar- that target is.

And so, I, I, I mean, I kind of work backwards that way in that, like, you know, you start with who actually fires the shot or directly authorizes the kind of final orders that kind of go into the end targeting. But then you kind of work your way back and say, well, there is the interface, and to what extent did that bias or lead towards decision-making, erroneous or not, that led to that target? And then behind that you have the data infrastructure and so forth. So there's all these sort of different pieces to it.

Kate Klonick: So this is, kind of, a great point. You give us this anatomy of the kill chain, and the thing that I really kind of also want to focus on specifically when we're talking about kind of the, all of this kind of array of private actors that take place in this and that you just described, is that I think it's contrary to the notion that when you talk about a chain, there's kind of this idea, well, if you break one link, the whole thing is like a, a, a bit of like a, it's a chain, and so it's, it, every point in it is like a, if you can disrupt one part of it, you can like have a, a like a choke point essentially.

And I think that one of the things that is really great about what you're describing and what this piece describes is that there really isn't a chain. It's kind of a web or it's kind of a stack, and the stack is diffuse and has many options on how it can go. But I kind of just wanted to see if you could add a little bit of color to that, and if that's something you agree with.

Steve Feldstein: Yeah, no, I think you've actually described it really well. I mean, I, I think th- this idea that all of a sudden you can stop a, a kill chain or disrupt it through taking out one thing doesn't really work. I mean, kill chains just describes a process of getting from identifying something to destroying that thing, and all the steps along the way in which to go from, from A to B.

And so, you know, kill chains existed before AI and this, the, you know, tech stacks and so forth. They were just more analog and they were more manual. And so, you know, you could certainly disrupt the kill chain in the sense that you could take out a certain capacity and make it harder to do something, but at the end of the day, you're still gonna go from A to B in either case, and it really is something that you can't sort of stop.

Also, I think it's important to note that there is an owner of the kill chain. That's the Pentagon. You know, these different private entities and, and service providers give software, they give options. Sometimes those options, there's a design element that's biased towards a certain outcome or another, but I don't think that means that you sort of take away, you know, legal or military accountability from the Pentagon itself.

They ultimately are the ones who have to make the clicks. They're the ones who can demand different pieces to how the process is set up, and they're the ones who have to sort of make the final decision, say, "This target should be destroyed. The evidence coming in points to X conclusion." Or to say, "You know what? We need to take more time. I don't feel comfortable with this. The pace is too fast. The scale is too vast. We ne- we need to, to slow this down because I'm not confident that, in the accuracy of what's being recommended to me right now."

Kate Klonick: Yeah. I'm increasingly a fan of, and I'm, I'm plenty a podcaster in this, about the idea of, like, deciding to opt out of certain types of technology and, and our human, like, autonomy and ability to say, like, "Let's slow this down." Of course, there's not incentives, as you say, in the fog of war, but even in war in general to always do that. But it is something that's certainly possible and, like, policy as a, has a role in shaping and which we'll get to in a second.

But I kind of want to talk, really quickly before we kind of flip to the policy discussion, of kind of like the diffusion, like, what's next, who's next, what actually spreads? And so, you kind of bring up the UAE, and, like, kind of the idea that this is, like, the forward-looking thing that we should kind of talk about. And I, you know, just to be clear, just define diffusion in this context for people. Sometimes I try to imagine my parents listening to this podcast and like, I'm like, there are so many things that I'd have to define. But just define what you mean by diffusion in this context.

Steve Feldstein: Yeah. Sorry, Mom and Dad. So by diffusion, what I mean is sort of the spread the emulation, the replication, and then a- adaptation and use of a particular technology. It's the idea that one country has an innovation and others will get it, and the questions behind diffusion is how quickly and who can replicate it, and what does it take to do that?

So people look at a classic example like nuclear weapons, right? That's you know, example of something that actually has diffused quite slowly. It's a closed system. It's very hard to replicate, takes a significant amount of capital investment in order to operate, and at the end of the day, the, the costs required are not commensurate with the benefit that most countries would get out of owning them. So it's something that doesn't diffuse quickly.

The converse of that is the spread of something like OpenAI's, you know, like chatbot, like ChatGPT. I mean, that, that shows you something that was only owned by a few software developers that was opened up to the public and now has taken off like wildfire in the last couple years. I mean, that's an example of rapid democratized diffusion across the globe

Kate Klonick: Yeah, and so if you take kind of like an internet policy or internet history background, internet use generally required a lot of infrastructure, but once it was kind of in place, there was a massive diffusion of software-based products, like a massive diffusion of access to certain types of products.

And kind of similarly, I would say here, kind of one of the things about diffusion with, you know, systems of war, systems of defense are, are, are that like it requires, again, like a huge infrastructure kind of buy-in, but once you have it, there's like maybe it diffuses faster?  

How is this or isn't this different in your mind than something like, I think actually your kind of your comparison between like a ChatGPT diffusion and nuclear, like, capability diffusion is, like, absolutely perfect because they, we kind of have both in this setting. Like you, like just having the intelligence, just having the access to certain things, like, does not make this a better or easier way to let, you know, doesn't actually change the game. It's the marriage of that with the, with the physical kind of kinetic capability of a drone or a strike system or something like that, which is a different thing.

Steve Feldstein: That's right.

Kate Klonick: Yeah, and so do, what are you, what are you seeing with, like, the diffusion kind of issues here, and why is the UAE the example that you use?

Steve Feldstein: Yeah, so I mean, I, I think y- you've explained it e- exactly right in the sense that, I mean, first of all, you need to have capacity, right? And so, so the first question I asked is who has, who doesn't have these systems yet but has the necessary resources, so they have the money to be able to acquire the necessary components for this, or who potentially has some elements already in place, like a surveillance apparatus that they've built up?

But then the, the other element, the other ingredient to this, in addition to capacity and resources, like the material, is political incentive. Who is, who is kind of worried about or living in a volatile region, who is subject to an unpredictable neighbor who already is facing bombardment from military adversaries?

And the UAE, you know, like, fits that pretty well. So, they're feeling politically and militarily vulnerable. They actually have invested over a period of, of many years, a, a pretty substantial surveillance infrastructure anyway. They are also you know, have, have signed lots of deals when it comes to AI data centers and so forth. So a lot of the data infrastructure elements are already in place for that. So it's very logical. And then what we also know, from the kind of weaponry standpoint, is they've really are investing a lot in their, this military conglomerate edge. So they are seeking to build homegrown drones, to build up their own arms industry to a, a greater extent. So all these other pieces are there and, you know, but most importantly is the political incentive. They, you know, see a very volatile, unpredictable world and they want to protect themselves. And so investing in a system like that is natural.

Now, there are other, a few other countries like that as well. I mean, I would sort of think of them as like these, like, sort of rising powers. You know, Saudi Arabia is in a similar situation. You can plausibly argue that India, maybe Brazil you know, are other countries that have significant capacity and that over time, you know, would be kind of next in line to want to take up and adopt these sort of systems. So that's kind of what I looked at as like a, a, a sort of next set of, of actors who potentially, you know, I would anticipate would take these systems up in the coming, coming years.

Kate Klonick: Yeah, I really wonder how much of that hinges on frankly just money. Like, how much of that hinges on the wealth of the nation. You have Brazil, India, Pakistan, Singapore, Turkey, Saudi Arabia that you kind of list in this, and list, and UAE is- to the, and the reason Steve is, to those who are, are listening and not watching, the reason Steve was laughing and kind of giving his answer was, like, he was, like, describing the UAE. Who has, like, who has a neighbor that is, like, very, like, and I was, like, stroking my chin, like this was a very hard question. Like, who's incentivized to kind of do this?

But I think that this is, like, you know, this is a really great list. Who is the most incentivized, and then who has the resources to actually build this? Because obviously, like, you know, I'm going to go out on a limb and say that Turkey is not quite as well-resourced as the UAE, and like, you know, and it doesn't have as imminent a set of threats. And so like, you know, we're going to see it, but maybe we won't see it first. So I think that the UAE is a well-selected kind of, you know, as the dominoes fall, who's going to have this diffusion?

But you draw a kind of a sharp contrast. I kind of want to use this example. You draw a sharp contrast with drones which kind of diffused everywhere. I mean, frankly, like, from militias to, like, people in the park, like, spying on you as you're, like, lying on your picnic blanket type of thing. Dro- drones are, and I know that we're not, I'm like, hopefully they're not armed, those drones whether you're in the, hanging out in Central Park. But just generally I kind of, like, the idea that, like, this, these, the, that something could, like, be so much in your personal space so quickly and so seamlessly is a huge, huge change with drones, and it diffused so fast, and it just diffused so fast.

When I was in, when 2014 killer robots and killer drones was at the very forefront of, like, like, when I was starting my PhD. And, you know, now it's just a given. Like, this is just how kind of, that, and at that point, people were like, "Oh, drones." Like, oh, here we are in, like, the future of sci-fi kind of.” But no, it's now just, like, how war is done.

And so is AI targeting con- like, categorically different or just earlier on the curve than drones right now? Like, what's the load-bearing bottleneck here? Is it data? Is it talent? Is it ISR compute? Like, w- what do you think actually holds, like, the, this back from diffusing?

Steve Feldstein: Yeah, no, it's, it's, it's really interesting to kind of think back to the drone analogy. I mean, like yourself, you know, I, I sort of watched the evolution of drones from being these very exquisite $30 million reaper drones that only the U.S. and a handful of other countries were using for, like, signature strikes against the Taliban or in Pakistan against al-Qaeda or, or against al-Shabaab in, in Somalia.

And then over time, but really accelerating, I think, between the Islamic State and investments by Turkey into kind of low-cost, medium-range drones, their TB2s, you've all of a sudden saw this, like, explosion. Then with 2022, with the Ukraine war and their innovations with FPVs, they were able to take something You know, they were able, able to take something that was essentially consumer grade, you know, and, and make it mass-produced and show that a very simple quadcopter that you could buy for $500 off Amazon, you could outfit it with a basic munition and create cr- tremendous damage. And if you did that a million times or two million times, like look at what you had, right? So, you know, like what an interesting insight like that showed.

So I guess the question is like, will we see the same thing with AI? I mean, I think we will, not necessarily for the kind of targeting system that I mentioned. I mean, I think that's gonna be a ways away from just becoming ubiquitous around the world. But what I think you will see are like lots of different experimentation with, like, autonomous drones, other autonomous weapons. So things like the last mile where you have a chip embedded in the device. We're already seeing this in Ukraine, and where you lose contact with the operator and the drone is able to use computer vision to kind of lock on to particular targets and destroy them. Now, maybe the targets are actually the right targets or maybe they're not. I mean, this is the problem when you sort of like leave your hands and, and sort of say, "Well, it's up to, it's up to the machine to make the, the final decision."

So I think we'll see a lot of push when it comes to AI integration on the edge when it's on devices that are being used directly in battlefield. But for a full-fledged comprehensive system like Maven, I think that's still gonna remain in place for a small group of, of high capacity militaries for the near future.

Kate Klonick: Yeah. And so this has been so, so clearly like kind of laid out and I think is a, is a per- very persuasive argument. And so like I kind of want to move to like the, the so what. So like what do you think that we should do if we decide that like this, we have a little bit more time than maybe people are forecasting or people are doomsday predicting?

Your prescription kind of has three parts as I kind of like put it together. There's the restrict cutting edge chips, right? Curve kind of access to the most advanced models and limit kind of military specific software like Maven. And so the chip piece we kind of know how to do, but there are parts of it that we're doing already, at least in theory. But model access, how does that work?

Like how are we going, I mean, and this is something that I think that we've been, we've talked about in, in a lot of different rooms, you and I, Steve, like that we've been in and, and we just kind of talked about in like the nature of what diffusion is, like how things diffuse. Software is the hardest thing to stop kind of from diffusion. I mean, the internet, like information wants to be free. Systems want to be free. How do you, like how do you basically do that? Weights leak, capabilities like get distilled into open models and then the API is a global product.

Like, you know, is export controls for a model even like a coherent way of thinking about this? I know that there, there have been a lot of people and experts in this field calling for export control on models, and I think that that's just not one of the most effective ways that we've controlled software in the past. So I'm just kind of interested in your thoughts on that.

Steve Feldstein: Yeah. Yeah. Look, I mean, I, I think you've laid out I mean, I think all the challenges right, and I, like I, I don't know that I can, like, legitimately come on here and say, "Look, I have a plan for stopping the export of software and AI models," because even though, like w- as we're going around, we can see that not happening everywhere. Like I, I think the, at least on software and modeling, I think the cat's out of the bag. I don't really think there's much we can do on that front. I don't think that's a re- a realistic choke point, and I'm not gonna pretend otherwise that, like, there's some plan we can do.

So I think that leaves other pressure points. So, you know, w- certainly one of the pressure points is, I mean, you can, you can, you can make an argument, maybe, that the very most advanced models, the Mythos of the world or Fables or whatnot, maybe that has l- slightly extra capabilities, and so as new frontier models come out, we should vet and, and look for, for ways to safeguard and think carefully about how that could be used. But I think that, that to me doesn't really solve the problem.

I mean, a second part of the problem is, is, you know, you try to restr- restrict the infrastructure. So it's the chips, it's the data centers right? It's, it's the kind of surveillance apparatus that I, I mentioned. And I think to that extent, you can't stop it, but you can slow it down, and I think we've seen that happen successfully to some degree when it comes generally to surveillance. So, you know a bad dictator who doesn't have access to a lot of money in a, a second or, or third, you know, third-tier country wants to get this type of equipment, so that's where you can export controls can make a difference. So you won't stop major militaries, but you can stop second and third-tier actors maybe from acquiring as easily as they would like these types of systems.

And I think, you know, I also think I, I, I mentioned is normative pressure, public pressure. Right now there's very little friction. It sort of feels like it's kind of an open space. There's few rules. There's people, the public still is confused exactly about how these systems work. The more you can start to, to create a political cost for countries that are thinking about getting them, the more that it becomes a resource trade-off question where, you know, leaders say, "Well, I could spend 150 million investing in this system, but I'm gonna, I'm gonna pay a political price, and I'm not even sure if it's exactly gonna accomplish my objectives. Maybe I'll hold off." That I think can have a bit of a, of an effect as well.

So I think normative pressure matters. I think infrastructural pressure can play a role. But I don't know that stopping the diffusion of models is really gonna, going to work. At least I can't see that.

Kate Klonick: You, I liked your normative pressure take, and I actually kind of paired it in my mind with what you just said, which was kind of actually not just normative, but diplomatic. Like a decision that has to kind of happen across nations to see the trade-offs as they're coming, and that there could be infinite amounts of money and resources devoted to basically like, you know, not a cold war, but a very, very hot war depending on whether or not you're like near a data center.

You know, and the climate kind of impact, the, the trade-off of, like, putting money into that and not into, like, improving people's lives, healthcare systems, food, like, medicine, like, all of these types of things that are very real problems. And, like, if you're… Everyone is scrambling. I gave, like, a list of just, like, sev- you gave a list of just seven or eight countries that are, like, in various wa- ways that, like, would be gearing up to get these products and to build systems that, that create these products and these weapons, that those are, you know, there has to be some type of truce. There has to be some type of agreement that we're not all going to be searching for this all the time, and maybe that's kind of a, a pie-in-the-sky type of idea right now, particularly with our current leadership. But I, I, I do think that that is something that has to happen.

And it kinda leads me to kind of the lawyer's question in this. You know, as we talk about treaties, as we talk about kind of truces, but this is, like, kind of the, in the international humanitarian law, like, kind of context, it assumes that human judgment is a point of targeting, right? And we talked, you know, we've just spent the whole time des- destroying this kind of idea of “the chain” of the kill chain. It's a much more amorphous kind of system that doesn't have a clear choke point.

But one of the things that humanitarian law and a lot of people argue for generally is to build in that choke point by having a human in the loop. This is, like, a phrase that you'll hear over and over, “human in the loop.” And people generally think that this is like, I would say, like, I will summarize the, the, the arguments against that, which are basically that if you have a human in the loop, you slow down all of the benefits that you get from a very fast, very sophisticated AI system. You get a backlog that the, the, it's like it's not a, it's not a one-way ratchet. It's not like you can have everything flowing and then, like, magically a human can, like, snatch the one bad thing out of, like, the ether.

And also, as we know, in, like, various types of experiments from, like, medical diagnosis to all kinds of pattern recognition, actually now these machines, these, now these systems are better at pattern recognition and spotting errors than even humans do, and humans often introduce errors to types, to a lot of these systems. So it's like, do we want a human in the loop, right? Or will it introduce error to, like, these, like, sophisticated machines?

But to get back to the point of autonomy and choice and international humanitarian law such that it is, and such that it has kind of power to shape this space, do you think that, you know, when Lavender generates kill lists almost automatically, and when Maven compresses, like, detection to strike, like, into minutes, right? It's just minutes.

Is meaningful human control, like, even a real doctrine that we are going to be able to deploy? Is that something we should make a choice as, like, politically, legally, that, like, we are going to necessitate that type of thing? Also, who would want that job, honestly? But, like, we're kinda like that we're gonna necessitate that type of choke point to be introduced to the kill chain. Is it still a real doctor, or is it kind of just this legal fiction that's doing a lot of work that doesn't really kind of maintain what we want to accomplish in the first place. Like, do we want these systems to be safer? Yes. But like, is a human in the loop gonna do that, essentially?

Steve Feldstein: Right, right. Well, I mean, there's actually two levels to it. I mean, I think one, one question is, do we think that there ought to be an equivalent balance between accountability, or is the new theory of war is that faster wins battles, and therefore, by sacrificing speed, pace, and scale, you are essentially causing yourself to lose?

And this is the theory that a lot of people are putting out. This is the theory behind why there's such a push between ever more powerful algorithms, bigger systems powered by AI, that can allow for a compression of identification to, to target destruction, right? And the argument that you'll, you'll hear is that yes, you might get more accountability, but then the, the war will be lost by, by the U.S.

Now, I think that's a little bit of a false choice, but I, I don't think you can dismiss those, those arguments either. And, and I, a- and, you know, the question on the kind of human in the loop and meaningful oversight, in some respects, we're already answering that question. And the answer is sort of like, no, we're, we don't really, I mean, meaningful human oversight is, is starting to become something a little more distant, certainly for the m- the, the most sophisticated militaries. Like, I don't know if 70-second reviews count as meaningful human oversight when it comes to the targeting cycle that we've described with, with Maven.

Like, I'm not convinced of that. And the idea that you would have a human who was collecting information, sifting through it and looking at it, making decisions about whether it's a legitimate target, having people check off, lawyers check off on that, presenting that to a commander, and then clicking through and, and, and deciding to destroy, which is the way it used to happen not that long ago, and may have taken, you know, several hours, if not a day, to kind of get through. Like, that's not really how war is being fought, and certainly not where we're, we're, we're moving towards.

So, you know, what I think we need to kind of keep a hold of is accountability. There needs to be accountability, period. Now, is that some combination of human on the loop combined with machine-generated recommendations that still is able to bring about accountability for violations of IHL that occur? Like, we've got to figure out a way to get there.

And I think right now, honestly, we're, we're behind when it comes to figuring out how these, how IHL applies with the new technologies on the field that are being deployed. I think there's a big gap in that. And I don't have a good answer to that, but I think we need to think carefully about it and take it seriously, because it's only proliferating in the, in the months and years ahead.

Kate Klonick: So last question for you, Steve, is that hopefully in the near future we'll get to talk to you again because you have a book coming out. Tell us what it's called, tell us what it's, you know, when it's due, and, like, tell us kind of what it's about. Is it dealing more with this kind of idea? Is it going to expand on kind of some of the really interesting points that you make in this article?

Steve Feldstein: Yeah. Thanks, Kate. So I, I do have a new book. It's called “Bytes and Bullets: Global Rivalries, Big Tech, and the New Shape of Modern War.” It's coming out next week actually in the UK, and it's coming out in September in the U.S. So, and the book really kind of looks broadly at the way in which tech is being used to exercise power, both geopolitically and war.

So, it looks at different levels of, of tech competition and rivalry between the U.S., China, and Russia on the one hand. It talks a lot about how private companies are playing a bigger role, as we've discussed here, in terms of increasingly making decisions about how products will be used and who gets access to them. And then it also talks about this, this idea of diffusion or spread. How do new actors, you know, insurgent groups and others get access to disruptive technologies that help them increase influence and attain political objectives? So it tries to sort of present a big exposition of what the landscape looks like and where we're headed. So, I'd love to talk about it when it's out.

Kate Klonick: Yeah. And we'd love to have you. Thank you so much for coming on, Steve.

Steve Feldstein: Yeah. Thanks, Kate.

Kate Klonick: And I'm looking forward to talking to you about the book in a couple of months, and thanks for coming.

Steve Feldstein: Yeah, thanks for having me, Kate. I appreciate it.

[Outro]

Kate Klonick: The Lawfare Podcast is produced by the Lawfare Institute. If you want to support the show and listen ad-free, you can become a Lawfare material supporter at lawfaremedia.org/support. Supporters also get access to special events and other bonus content we don't share anywhere else. If you enjoyed the podcast, please rate and review us wherever you listen. It really does help.

And be sure to check out our other shows, Scaling Laws, Rational Security, Allies, The Aftermath, and Escalation, our latest Lawfare Presents podcast series about the war in Ukraine. You can also find all of our written work at lawfaremedia.org. The podcast is edited by Jen Patja with audio engineering by Goat Rodeo. Our theme song is from Alibi Music. And as always, thanks for listening.