Scaling Laws: The GoLaxy Revelations: China's AI-Driven Influence Operations, with Brett Goldstein, Brett Benson, and Renée DiResta

[Intro]

Alan Rozenshtein: When the AI overlords take over, what are you most excited about?

Kevin Frazier: It's not crazy, it's just smart.

Alan Rozenshtein: And just this year, in the first six months, there have been something like a thousand laws.

Kevin Frazier: Who's actually building the scaffolding around how it's going to work, how everyday folks are going to use it?

Alan Rozenshtein: AI only works if society lets it work.

Kevin Frazier: There are so many questions have to be figured out, and nobody came to my bonus class. Let's enforce the rules of the road.

[Main episode]

Alan Rozenshtein: Welcome to Scaling Laws, a podcast from Lawfare and the University of Texas School of Law that explores the intersection of AI, law, and policy. I'm Alan Rozenshtein, associate professor of law at the University of Minnesota and research director and a senior editor at Lawfare.

Today, I'm talking to Brett Goldstein, special advisor to the chancellor on National Security and Strategic Initiatives at Vanderbilt University, Brett Benson, associate professor of Political Science at Vanderbilt, and Renée DiResta, associate research professor at Georgetown University's McCourt School of Public Policy and a Lawfare contributing editor.

Earlier this summer, Brett Goldstein and Brett Benson revealed leaked documents from a Chinese company called GoLaxy, showing sophisticated AI-driven influence operations already deployed in Hong Kong and Taiwan. We discussed what this means for the evolution of information warfare, the challenges of measuring effectiveness, and whether the United States is prepared to counter these threats. You can reach us at scalinglaws@lawfaremedia.org and we hope you enjoy the show.

Alright, Renée, let's start with some high-level context. So you've been studying influence operations for many years, from, you know, Russian troll farms to sort of everything in between, and more lately focusing on how AI is going to transform this space.

So, what's changed in the last few years? What's fundamentally different about AI enabled operations versus the old Russian Internet Research Agency model?

Renée DiResta: Sure. So the Internet Research Agency model relied heavily on human operators and some component of automated. So when people think of the Russian trolls or the Russian bots as they're often referred to, they're actually usually conflating two things.

The accounts on Twitter that used automation were the dumb accounts. The other side that they had were the human operators. They actually had a bunch of 20-somethings that were sitting around a factory, first in St. Petersburg. They moved to Olgino also at various times.

And what they would do is they would hire these people who they would task with pretending to be, in many cases, Americans, sometimes other parts of the world, but oftentimes they would try to have them masquerade, operate in the context of being an American persona. And they had very granular senses of what they wanted them to be.

So it wasn't just pretend to be an American, it was pretend to be a black woman of a certain age. Pretend to be young right-winger. Pretend to be a middle-aged or an old right-winger. Pretend to be a liberal woman. Right?

So they would have these very granular personas and then they would have to post in the context in the first person, as a person of that persona. And what we would see when we looked at those datasets is that occasionally they would slip, right?

They would get the slang wrong. They would use vernacular, particularly when they were pretending to be African-Americans, that just wasn't quite right. And so sometimes to try to get around those slips, they would plagiarize content, and that was another way that we would sometimes find them because they would just explicitly crib from content that was already out there.

They would actually––when they were pretending to be young right-wingers, they started just grabbing turning Point USA memes and slapping their own logos on top of it to try to get around the fact that their language wasn't necessarily perfect. So when they were doing this, there were these different ways that we would be able to tell that they were not what they seemed to be.

When you have the capacity to use AI, when large language models started coming out––around 2020, some of us who had researcher access to GPT-3, back in 2020 when I was at Stanford Internet Observatory, we started writing about the fact that this was going to really change the nature of influence operations because what it meant was that you would be able to get around those problems.

You wouldn't have to plagiarize because you'd be able to generate new content and you wouldn't have the problem of the tells because you'd be able to use models that were going to be able to be fine-tuned on content that was of the language and slang from the communities that you were trying to masquerade as.

And so we started writing these sort of risk model papers. Josh Goldstein from my team at Stanford led a really great one that we did with OpenAI, basically arguing that this was really going to fundamentally change how state actors were going to conduct themselves. And particularly that it was going to become much, much harder to find these things because they were going to begin to use them.

And over the last two years or so, as large language models became much more readily available to the public––remember we were writing this in 2020, when these models were still largely gated and only researchers really had access––once ChatGPT and others began to roll out, OpenAI began to release threat reports similar to the threat reports that Meta and other social media companies released saying, ‘Hey, here are how we are actually seeing, in the wild, state actors beginning to try to abuse our models.’ And we began to see those things actually play out in the real world much as we expected them to happen.

Alan Rozenshtein: So you mentioned GPT-3 and you wrote a great piece in 2020 talking about this future of AI-enabled influence operations using sort of GPT-3, which was then available.

And I think you even had GPT-3. I mean, this was back before, yeah, this was back before everyone was using these all the time. So it was still kind of cool to be like hey, this paragraph was written by GPT, right, by an LLM. And you noted that in some ways, the capabilities were quite impressive, right?

It could mimic your style in a kind of an 80/20 way. But it was GPT-3, so it's kind of a dumb model. It would kind of get confused and sort of wander off into the wilderness. Obviously in 2025 we're way beyond GPT-3. I mean, we're two GPTs beyond GPT-3. And more seriously, the capabilities have exploded.

So, you know, before, before we turn to GoLaxy, I just want to ask a follow up question of you. How much have these models improved from the perspective of being able to do influence operations? And relatedly, how have we hit kind of diminishing marginal returns? So obviously these models will get better and better and better, you know, but like, you know, you don't need a model that can solve the Riemann hypothesis, presumably, to do decent personalized influence operations.

So I'm just curious, are we sort of 99% out of the way to an optimal influence operation model or in five years? Will GPT-8, or whatever the open source version of it, will that be sort of even worse, from a practical influence operations capacity, than what we have today, which again, is presumably substantially better than what we had in 2020?

Renée DiResta: So there's a couple things. First, when I wrote––the article that you're referencing was something they gave it a kind of sensational title, like, “The Supply of Disinformation will Soon Be Infinite,” I think. I wrote this for The Atlantic and I had it, as you know, kind of co-author the article with me.

I, at the time, I was looking at longform propaganda that the GRU had written in the form of the Inside Syria Media Center. So, not just the shortform posts of the Internet Research Agency which is what I described in response to your first question, but much more longform, the kind of propaganda that I think GoLaxy, we’ll talk about, which is how do you actually also write longform propaganda that becomes the content––that actually becomes like training data for future models.

Like how do you seed narratives on the broader web? How do you make that process of propaganda production essentially, like––how do you take that way down, the cost of production, basically down to zero? So there's that piece of it.

So that, you know, that I think is fairly even. Back then, you had it kind of where it needed to be.

It didn't need to be perfect. It just needed to be good enough. But the other things that we're seeing, the audio component is really extraordinary. Now the capacity to produce convincing audio so that you can create novel moments that trick people ahead of, for example, an election, right? Like that threat model is increasing significantly.

How do you create plausible video that can trick people, right? That capacity, over the last even six months, is just getting better and better. So I think we're going to talk quite a lot about text- and persona-based manipulation, which is something that I personally think is significantly more impactful in the long run, because it does change how people interact with other people.

In terms of––when you're trying to shape public opinion, you can do that through persuasion or distraction, right? And when you're doing it through distraction, what we see a lot from China is this model of flooding the zone, just trying to get as much content out there as possible. Versus what you see from Russia is much more of this persuasion and engagement model of creating plausible personas and talking to you.

When you're creating plausible personas and talking to somebody, another way that people would detect them was by looking at the profile picture and realizing, ‘Hey, this is a profile picture that's stolen. This is somebody else's Instagram photo.’

Now you have generative AI that generates pictures that are indistinguishable, right? That, you know, in the early days of generative adversarial networks, the models that would produce faces where there were still a significant number of tells, we would see influence operations actors use those, but they were still easily detectable.

The kind of stuff that you're getting through diffusion models now is significantly better, and you can generate the same face in a whole lot of different poses. You can create an entire, like backstop identity with a whole lot of different poses of people, with different people, you know, different faces, different family members in different places on vacation with their fake kids.

You know, you can create an entire identity now in a way that you couldn't even two years ago. So it's not just the text-based models, it's actually the combination of the ability to create a rather immersive unreality, I think, around some of these personas that's really gotten significantly more engaging.

And then the other piece is the responsiveness of how they engage with you. So the chatbot component in which it is actually able to respond back and have a series of interactions with you, which I want to like bring in the GoLaxy part of this, because that's the other piece that we would see in the, sort of the last bit of work that I was doing at SIO before it shut down, which was the responsiveness of the chatbots that you start to see on Twitter today.

Brett Goldstein: My sense, Renée, is we're still in the early days. Yeah. And we've seen this impressive evolution from what you talk about in Russia, where you had some technically early stage techniques, which represented where the world was with data science, right? We'd talk about things like clusters and commonalities of groups, and you'd try and tailor it that way.

Or, you know, the techniques, the human side techniques. But we're not really talking about the humans today. I think today we're starting to see this architecture, which with gen AI and efficient compute, you can do individual targeting. And that's a newer domain at scale.

So, imagine a scenario where you have a very efficient and smart AI coupled with information about each individual. And we start talking about open-source intelligence, okay. OSINT.

But we also, in the world we're talking about, there's plenty of leaked information. So the early tickles right now, which I think are going to be, become increasingly dangerous are, if I'm targeting Renée, I'm going to have messaging only for you. I'm going to create a persona that's designed for Renée.

It's going to be the persona that can sell you the best, that can be the most compelling, and then as resilient as possible. And when everyone has a persona that's perfectly designed for them, and the perfect message is delivered, and you start deal looking at that in enormous computational scale? That's what's coming. And that's what's super complicated.

Brett Benson: And just let me just jump in right on the heels of that very quickly. I think one of the implications of the scalability and the increasing ability of lots of different groups to engage in this type of behavior is the way that has transformed what propaganda is, right?

So initially, like even, even going back to the days of like 1950s, 1960s propaganda in China, analog-style, through––in my view––the 2016 elections with the sort of dumb types of messaging we had on social media, this type of propaganda typically targeted one type of vulnerable subpopulation.

The propaganda that we see now in influence operations doesn't have that type of selection effect. And so the potential spread of disinformation, even if a network, a social media network or something else has been infiltrated, typically because of the persuasiveness of the personas, the sort of impact isn't constrained by the limitations of individuals.

And so because propaganda is much less expensive because it's more persuasive, you don't have that type of selectivity when it comes to spreading disinformation.

Alan Rozenshtein: Okay. So I think with that context, we should turn to the actual GoLaxy situation.

So, Brett Goldstein and Brett Benson, earlier this summer you published a fascinating New York Times piece revealing documents from a Chinese company called GoLaxy.

And then there was a great piece in September from the record reporting how Brett Goldstein, how you received this mysterious link to a 399-page, you know, Mandarin document, and you stayed up all night feeding chunks of it into an LLM to translate them.

So I would just love for you to kind of walk us through this moment. What did you find and when did you realize that this was something bigger than just, you know, some set of kind of leaked, vaguely embarrassing corporate documents?

Brett Goldstein: Yeah. So as you mentioned, it started with a link. And as people know, I hate links.

Alan Rozenshtein: I mean, the real question is who's going to play you in the movie of all this.

Brett Benson: Well, hopefully Brett Benson.

So, I get this link. It comes from someone who I trust, a trusted researcher. And I still hate links and I get it. I go through all the things that someone like me should do. I make sure the link is safe.

I then put it into a virtual environment and I unpack it. And then I realized that it's a bunch of weird PDFs. And I start going through the PDFs and I'm like, oh, these are Mandarin.

And I'm not really sure what to make of it because my Mandarin skills are none. And I'm certainly not going to call Brett B yet because it's just a bunch of documents.

But then I start scrolling through it and there's two things that catch my eye right off the bat. One are technical schematics, and I'm deep into a system architecture at this point. And I think it's––from what I can take, it looks like it's collecting massive amounts of data and then producing products.

And that's compelling. And then I start seeing big grids of people. And that is very compelling. And then I started seeing pictures of congressmen and I'm like, okay, we've got something here.

And that was, okay. It's super interesting. I know I need to figure this out. And if I were a typical person, I would dump it into ChatGPT and get some quick answers.

But there is no way that I'm going to start dumping pages coming from someone who's passed on the location of these documents that look super interesting into someone else's LLM and get fast translations.

So I did the hard path. So as you noted, I was up super late. I'm obsessed with the problem and I'm like, I'm going to install local LLM in a virtual environment and run everything locally on a controlled network because I don't want to share the answers with anyone.

And that's what I did. I stayed up, I built this, I got it going, and then I started dropping chunks into it. And that's when I realized that there's enormous OSINT and collect going on. They are U.S. congressmen, and it looks like an IO messaging type approach. And I'm like, oh shit. This is something that's, that is super interesting, super hot. And I'm like, where are they going with this? There's neural networks involved, all of that.

And that's when I pulled in my research partner Brett B, knowing he speaks Mandarin. And I'm like, I'm going to turn the group from one to two and we're going to start to figure this out.

Alan Rozenshtein: So say more about how this system operates, both from what those documents reveal and just from what you know, your reporting and analysis also suggests. Kind of walk us through, what is what is the smart propaganda system?

Brett Goldstein: Yeah. Okay. So, this is interesting because there's years of information here. It goes back a number of years and then based on our research that we searched on for a number of months, going from what was on their website and those pieces, we learned that historically they have 1) been doing enormous OSINT.

Like, every day they collect millions of attributes from social media platforms of whomever they're targeting. And in some cases it's about Hong Kong, it's about Taiwan. In other cases that are more murky, 'cause I can't figure out what they're collecting, it's looking to the west. They're clearly looking to the west. They're looking at the U.S. They're looking at high-profile folks. But it's unclear what they're collecting.

So they collect all of this data and they put it into a structure and they make it accessible, but then they have a couple different paths that they work on. One is the creation of messaging. What is the right message for a given outcome against a given target?

And I see classical, sort of, neural network approaches there, but then there's a second part of the effort and I found this, I'd say both interesting and telling, which is getting at resilient personas. And what do I mean by that? They want to have a persona, so a social media account, which they can launch messaging from, but also not get caught.

And when you start to see line after line talking about how they don't want to lose the account and they want to make sure it's hidden and it stays resilient, you really, really start to get at the intent there. So there's multiple years of history architecture, but here's where it gets really interesting.

We, before we told anyone what we were doing, we went and we started a deep dive on their web presence. And they had more information than you might think up on the site. That has changed, as of now, which we can talk about later.

Alan Rozenshtein: You have to advertise your services somehow, right?

Brett Goldstein: Well, yeah, a little less today.

So a couple things that we noted was a partnership with Sugon and DeepSeek. And that's where we started to understand their evolution here.

So there are things we know. They were using traditional neural network type architectures with a somewhat typical stack. And for your listeners, we've open sourced these documents so people can look at these diagrams.

As well. And that was a traditional way that it grew up. But seeing these partnerships, that's when Benson and I were able to start to extrapolate, this is a potential path of where they're going in the gen AI piece. And as Renée talked about, the world kind of changed a few years ago as we introduced gen AI in these techniques. And that's where you can see this whole method shifting from more classical AI neural networks to a generative approach.

Alan Rozenshtein: Alright let me turn to Brett B.

So I'd love for you to first kind of give us your side of the story, how you were dragged into this and how your initial sense of this kind of evolved as you dug in.

And then also talk about you know, Brett G has talk about what, sort of, GoLaxy was doing, but obviously there's a big piece of this about GoLaxy’s relationship with the Chinese government.

So really curious to hear about that, both specifically to GoLaxy and also how that compares to the relationship between other Chinese companies and the Chinese government––obviously, given that it's China, the idea of a fully private company is always a sort of complicated set of questions.

Brett Benson: Yeah. How was I dragged into it? That's a good way of putting it, 'cause I felt like I was.

So, Goldstein and I were traveling and we were working on something else. We had a very busy schedule. We were overseas and, and he one day nudged me and said, Hey, I have a cache of documents that I want you to look at.

And they're in Mandarin. But he really undersold it, and he never let on that he had already, that he had already looked at them and had some inkling of as to what was in them. And then he gave them to me. And the next day I told him, I hadn't looked at them yet. He seemed a little bit agitated.

And the next day, late at night, I pulled them up and I started to read through them. I thought, good grief, 299 pages, all in Mandarin, really dense Mandarin. You know, for those who don't read Mandarin, that's like 450 pages in English. Lots of slides of pictures of members of Congress and people in Taiwan and influential people in Hong Kong.

And I had no idea what I was getting into. My background is not influence operations. My great background is political economy and I study these sort of theoretical models of international conflict.

And so I started to poke through them and it got late at night and I was pretty shocked at what I had seen. But I have to tell you, honestly, I was really suspicious. I thought, you know, I have seen all sorts of things come out of China and this is probably in the category of something that isn't really worth our time much more.

But when we completed our travels and I started looking at them a little bit more I was pretty shocked by what I was reading.

Alan Rozenshtein: What do you––

Brett Benson: One document––

Alan Rozenshtein: Let me jump in a second. What do you mean you were suspicious that this was puffery? Suspicious this was a kind of, its own kind of fake influence operation? What was the nature of that suspicion? I'm curious.

Brett Benson: Yeah, it was sort of multifaceted. I was very suspicious.

I was worried about downloading it 'cause I had no idea what was in it. I thought it was going to, you know, potentially blow up my computer. I didn't know if it was, you know, potentially some scam and I was, you know, I was going to be asked to donate millions of dollars to some intellectual person in China.

I mean, I, there, there are all sorts of things that were going through my head. And I mean, Goldstein's kind of a suspicious guy. So I was I was really suspicious of his suspicion and the documents themselves, until I started reading them. So I think that's probably the best way to put it.

I started off with one of the documents, which is a strategy document. It’s sort of high level, and it's kind of difficult to characterize what the documents and the purpose of the documents is. Because on the one hand, it looks like they may be sort of designed to market the efforts of GoLaxy to potential donors or something like that.

But then some of the documents are also not like that at all. We later learned and figured out that it was basically a disgruntled employee who had leaked the information. And so it was like a collection of documents, some of which were presentations to, to people potentially to sort of market the capability to clients.

There were just a lot of stuff there. One of the documents was a strategy document that explained, in very very vivid detail, that the purpose of influence operations is to engage in international informational warfare with certain targets like Taiwan, United States. And it was very explicit.

And then it explained the model for doing that, which is to use psychological models in social science––which I haven't had the chance to vet––but to use psychological models to collect enormous amounts of data on, open-source data on users for the purpose of building these profiles that are highly tailored to their preferences and that are also adaptable.

And then the document justifies these types of efforts by saying the United States is engaged in these types of things as well. And so China's playing catch-up.

In terms of the connection to the Chinese government, the documents––and it was hard to sort of discern really what the connection is. I think I have a better sense of it now, but when we were reading through the documents, it was hard to know whether the documents were overclaiming their connection. So what the documents say is that GoLaxy serves Chinese national security and national strategy interests.

It's clearly set up by the Chinese Academy of Sciences. For those who know, it's––Chinese Academy of Sciences is a state-owned national research institute or academy directly under the state council, which is the key decision-making body for the People's Republic of China. The Chinese Academy of Sciences, though, oversees a huge network of institutes and universities, and so GoLaxy is one of them.

But then some of their clients are the intelligence organizations and agencies in the Chinese government. And then the documents talk about the, you know, the sort of proof of concept, the validity of the technology as used in Hong Kong and also in Taiwan.

Alan Rozenshtein: So how effective has GoLaxy’s operations been? So for example, right, they were involved in the 2024 Taiwan elections. But of course, in those elections, the I think it's the Democratic Progressive Party, which is the, it's called pro-independence party in Taiwan, won. So, you know, as I think, I guess with all influence operations questions.

And I'll start with Brett B, but then Renée, I'd love to get your thoughts on this as well. You know, there's always a question of, I mean, it's bad whenever they happen, but whether they cause additionally actual bad outcomes, it's always sort of its own separate question.

So maybe a simpler way of asking this is, how good is GoLaxy at its job?

Brett Benson: So I think this is the $60 million question. We have ongoing research to try to determine this, the effectiveness of this and other like organizations, that's something we don't fully understand yet.

As Renée put it, you know, one of the objectives is just to flood the space and to create chaos and confusion. And if that's the objective, then we have to figure out how to measure that. If the objective is to sort of change a mass line or a position of a community on something––persuasion, if that's the goal, we have to figure out how to measure and test that.

In Taiwan, they claim––or in Hong Kong, they claim to have been effective. But these are the documents making that claim. So nobody has objectively analyzed this, because it's really hard to analyze it. It's hard to catch a persona in the wild and then to trace how effective it is in changing opinions and ultimately having a sizable impact on public opinion.

So you mentioned that the Democrat Progressive Party won and the targeted––the campaign––the PRC campaign in Taiwan, as I understand it claim through the documents as well as some of the people I know in Taiwan, was not about flooding the space and creating chaos. It was about driving a particular narrative against the Democrat Progressive Party.

And yet the DPP won the election. But we don't know what the counterfactual is. We don't know if they would've won by a bigger margin. Taiwan's infrastructure is set up a little bit better than the United States for resilience on these types of things. Most people in Taiwan know that they're being targeted by propaganda.

They know that dense information is an ongoing part of what China does in Taiwan. And so yeah, the question of effectiveness is still an outstanding one. It's one that we're working on, by the way. But it's also––I think it's one that a lot of people should be involved with. Because there are lots of different ways to get to this, but it's a hard problem.

Alan Rozenshtein: Let me ask you a quickly follow up on, on that question of how you measure effectiveness. And I'm actually kind of curious, you know, about the sort of methodology.

I mean, this seems like––you know, I am married to an econometrician. So, many of my dinner table conversations are about how you do causal inference on very very very messy real-world phenomena.

And like, I can't even begin to imagine how you, how one could––I mean, I'm just a law professor, so it's not my job to figure this out––but like, how could you even in theory figure this out? Given that these are very small-n situations and there's just so much noise even in the best of situations, let alone where you have a secret subterfuge influence operation going on at scale.

Brett Benson: So, so the causal influence problem is the problem.

But the other problem is that we don't know if n is really small or if it's big, right? Because it's hard to identify the personas. So we have to figure out, first of all, how to detect personas so that we can start to measure.

We have to measure detection before we can measure the scope and impact of the influence. And then we have to set up––this is one of the things that we're working on, but we have to be able to set up some sort of experimental type of a situation that's designed to get to the causal identification problem.

To get to, you know, what would the counterfactual be in the circumstance in which a particular target was not targeted. And right now, I mean, most of the information on this and most of the evidence we have is purely anecdotal. And so, yeah, I think that you've sort of hit the nail on the head that the causal inference problem is enormous.

Brett Goldstein: So I think just. Pulling back for a second, you know, one of the reasons I'm worried is an experiment we did over a year ago. And it's a different use case, where what we did was pulled an individual's information from LinkedIn, so very very limited OSINT, and then we pulled it into an LLM and we created persuasive messaging.

And the crux of the idea was to try and get an individual to engage, presenting a paper at a faraway location. And something that like a lot of us will get an email and we'll just send it to Archive and be done.

And so what we did in the experiment, we pulled LinkedIn, created persuasive messaging that was individually tailored for that person, and we generated an email and then we chased it with an AI generated audio engagement that called the person following it up.

And we were getting 70 to 80% clickthrough on these tailored messages. And that's like a five minute technical effort at this point. Nothing at the level of sophistication that we should be talking about here. There is a technology in play, which is super awesome, Gen AI, which has the ability to customize messaging for individuals that gets them to engage in ways that they typically don't.

And that's some new terrain. So that little experiment from over a year ago, coupled with this disclosure, has me very very curious about, really, what we're walking into.

Renée DiResta: So I think there's a couple things here. First, I agree that, you know, the threat model is significantly different.

The papers that we wrote at SIO on this also around persuasion found much the same thing. My colleague Josh Goldstein, again the, you know, found similar things with testing persuasive messages. There's a DARPA program called Active Social Engineering Defense, ASED, that, you know, looked essentially at the same thing, like precision––spearfishing, it's called. The ability to target high-value targets with precision messaging is significantly easier now.

People are often unaware of how much information they're putting out there about themselves. Anybody with an Instagram account is telegraphing where they are, their kids' names, you know, everything about them is out there. What you say on Twitter, the photograph that you post, you know, people can intuit where you live based on, you know, the type of bricks in your wall sometimes, right?

So there's a lot that, that you can gauge. And if you are a high-value target, that kind of dynamic can happen. And one of the things that, if you look at the GoLaxy documents––and I did run them through normal ChatGPT––you do see the sort of folks that are in there, you know, the sort of high value politicians and who they want to engage with.

Interestingly, when we were at SIO doing our work, we would write up content on Chinese influence operations, and we would get the Twitter accounts of reporters at––Chinese state media would land, you know, would land in our dms, right? ‘Hey, read your stuff, you know, if you ever wanna talk to us, just send us a dm,’ right?

You know, I used to work in, you know, in the IC. I know what that is. You know, and I would've to tell my students, like, do not engage, right. But I, I remember having a––once we actually wrote a report on the U.S. Pentagon's influence operations, and the Chinese Ministry of Foreign Affairs was tweeting it, congratulating me on my excellent report, you know.

So there's the way in which these things play out is funny sometimes––

Alan Rozenshtein: Mazel Tov is the only thing I can tell you, Renée. You should ask for a certificate suitable for framing from the Chinese government,

Renée DiResta: But it was a, but here's what I'll say though.

So Josh and I actually have a paper coming out in “Security Studie”s that looks at the bragging in leaked documents from some of the contractors that perform work and some of the state outlets that run these influence operations.

There's one called Fabrica that runs stuff on behalf of the Russian government. Sort of again, one of these sort of semi-state entities in which they brag about how only 1% of their accounts are getting found, right? And they're talking about running influence operations against Ukraine, and how they're successfully changing public opinion about Ukraine.

How they're making people think that Ukrainian local mayors and things are corrupt, making them think that they're driving Bentleys and going on vacations and owning dachas and stuff like that.

Which, if you read the media outlets, like that kind of stuff is actually out there in the ether. Is that coming from these accounts? Who knows? Because as Brett Benson notes, it is very very hard to trace the origin of a message and say, this came from here and here's how we can trace it through.

And this is one of the challenges when we talk about propaganda. It is a very multifaceted dynamic. And even what we saw with the Internet Research Agency, which, the stuff that happened from 2015 to 2018 or so was three years almost of uninterrupted effort. There was no social media platform that was looking for it and taking it down. There was no integrity team anywhere at a platform that was saying, ‘Hey, we're looking for these foreign accounts. We know they're here and we're going to try to disrupt them,’ which is what is happening now and has been happening since about late 2017 or so.

And so, what you see there is these accounts managing to amass followings of several hundred thousand followers, in some cases, where they're speaking in the first person as a messenger who looks like a fellow member of the community.

And that's very important too. That's why they're speaking in that first-person tone. They had accounts that looked like, you know, niche media, but that's not what hits right. It's the accounts that look like somebody who's just like you.

This is why––like, I wrote a book on influencers. The reason influencers are such successful propagandists, and the reason what you see is the Russians evolving their strategy from trying to pretend to be these influential figures to just straight-up paying influencers, right?

I don’t know if you remember, but Tenet Media is this thing that the DOJ and FBI go and they put out these they indict this company, because what they start doing is just straight-up going and paying right wing influencers to say the things that they want said about Ukraine instead of bothering to create these personas.

Because they know the social platforms are out there trying to disrupt these accounts, and if they can get the authentic voice to say the thing that they want said, well, that saves them the trouble of having to do all of this work.

And that's because building these accounts, and actually doing the things that they're claiming in their documents that they can do––the claims that we see in these documents versus the reality of the impact, we have not yet seen that similar degree of success as we had seen in, in the 2016 to 2018 timeframe in terms of actual significant impact at that level of actually managing to amass an audience and having these sustained discourses where they become essentially vox populi, and they're getting embedded in news articles and amplified by prominent people, and we're really seeing them represented in the discourse.

So the promise of what they can do, and the technology has improved. And the threat model is there, and that's all true. And they're bragging about it in their documents. And you do see, again, some of the narratives that they claim they're seeding out there in the world.

But is it, you know, can we make this causal link? And the answer is, it is very very very hard to do that. The only time you can really say ‘This came from that’ is when you find some hashtag that they actually claim that they got done. And you can say like, okay, this viral moment was tied to that thing.

We have not yet seen that from a Chinese network that I can recall in all of my years of seeing China try this stuff. And so that's where I'm like, you know, the––one of the things we have seen from China, though is in their leaked documents, they will brag about their metrics. “We posted this many messages, we did this many things.”

And they're talking about numbers and engagements, like, we're checking the box, boss, look at this. We've done this many things. But they're not talking about getting feedback back.

So I too can run a bot that pushes out 200,000 messages in an hour. But that doesn't mean that anybody's engaging with them. And when we would write our reports, we would say most of these accounts never got a single response. So they would be out there pushing messages about the Uyghurs, but they would get almost nothing in response.

And that's where I think the thing that they recognize––and you see this in the documents when I read them––what I saw was a recognition that they're significantly behind.

Russia in terms of their ability to actually generate any kind of meaningful engagement with an audience.

Alan Rozenshtein: Yeah. It never ceases to amuse me that, at the end of the day, every bureaucracy is exactly the same. People are just all about how many boxes I checked, 'cause that's easier to justify to your boss.

Okay let's now turn to the U.S. angle of this, and sort of close out with that and what the U.S. can or should do about all of this.

Brett G., when you described initially how you came across these documents and what jumped out at you, you mentioned that once you started seeing pictures of us, congressmen, you know, congresspeople, that's––that was your that was your ‘Oh shit’ moment.

So, dig into that. Dig into that more. What do we know about GoLaxy’s operations or capabilities, not just in the kind of Chinese purported zone of influence of, you know, Hong Kong and Taiwan, but in, sort of, the good old U.S. of A?

Brett Goldstein: Yeah, so, so it is in fact limited, what we know about the docs.

Like we see, we have lots of headshots, we have a system architecture. It's not clear what they collect. Where I transition the conversation to is, when we talk about what this technical architecture can look like, it's not really speculative at this point. And you know, I know in some worlds of technology we're like, yep, three to five years we're going to be able to build X, Y, and Z.

In, in this case, I think about it as things I could potentially build on my laptop. So I, we have to make certain, I guess, assumptions here, which I hate. But you look at GoLaxy, you see all these names and faces. Why wouldn't there be enormous OSINT collect on folks? This is no longer a hard exercise, and they've done it in other regions.

Two, why wouldn't it be a collection on more folks? And it's, we have a couple screenshots where Benson and I think this is just a representative, not complete cohort. And why would there not be broader collection?

And then when I talk about sort of technical ease, at this point, like, my example of how I started to unpack the documents, I can run an LLM on my laptop. This is not an enormous data center that is required to do this.

You could, you can think about it––and I'm reflecting on Renée's comments before when she talked about high-value individuals––I look at, I'm like, you can get any value individual now because it's simply a function of compute.

So the things we think about in the U.S. right now are, what does it mean to have the enormity of this data being collected? And we talk about in smaller scales like TikTok and things like that, and all the data and the exhaust that is out there, to the ability to use a variety of gen AI tools––you know, some there are protections on, but if you go to Hugging Face, you have so many choices that you can do anything you want with.

And then the generation of messages. But if tying that all together––and I was in Japan last week and we started getting phone calls from reporters asking about the video generation tools that are coming out. And this is all starting to tie together now, because we're seeing propaganda that comes out in a variety of modalities.

And I think Renée earlier talked about multimodal methodologies. How do we start to base-case discern what is real versus what is not real? And then start to be able to––I'm not really into the censorship side of it, but instead be able to effectively––and I'd say at wire speed––know that something is created by an AI and then be able to more broadly get at the networks. And that's something that Benson and I are working on, because this is a problem that is speeding toward us and has become even more real over the past couple weeks.

Alan Rozenshtein: So, what do we do about this? I'm going to start with Renée, 'cause I know you spent many of the last few, much of the last few years thinking about this.

I mean, correct me if I'm wrong, but it does not seem that the U.S. infrastructure, whether governmental or a private platform is well-equipped at this moment in time to deal with this problem. So with that provocative statement, Renée, give me your, gimme your thoughts.

Renée DiResta: Oh, I have thoughts on that.

Alan Rozenshtein: Yeah, I bet you do.

Renée DiResta: Well, SIO doesn't exist anymore. Why is that? No, the fact is, you know, the way that we did a lot of our work looking at these things was we would see stuff at Stanford and we would reach out to platforms and we would say, ‘Hey, we see these things. Do you see these things?’

And, and we would say––and the process of attribution, right, saying, ‘Hey, this is actually a Chinese network’ because––or Iranian or Russian or whatever it was––when we would try to gauge the attribution, was done jointly. And it was done because we had some piece of the puzzle, they had some piece of the puzzle, and that was done quite collaboratively.

And then sometimes even the briefing to the government, you know, depending on what it was sometimes with the Russian stuff, when it was like Wagner Group or something, that would be done jointly too. Because there was a multi-stakeholder effort at the time.

And that was reframed as some sort of weird collusion, right? Some sort of weird cabal. And those ties were broken apart and that doesn't exist anymore. And then you saw the platforms, some of them, really roll back the teams that they had in place to look for this stuff.

And I actually went looking when I was, ahead of our chat today to see what the most recent reports were from OpenAI, which put out their most recent threat report. They put out these quarterly threat reports that are out there for the public. Anybody can see them. And again, the work we did at SIO was also released to the public. Anybody could see it.

OpenAI released theirs on October 7th. Meta hasn't put one out since May, so I'm waiting to see what their next one looks like. But these reports––Twitter hasn't put one out in ages, I mean, God, I don't think since Elon bought the company, they've released one.

But they used to put out these reports just letting people know what they had taken down. You could go and you could look at artifacts, you know, tweets that, that were representative of what the network had put out.

You could actually get complete datasets that were, you know, where the usernames were redacted but you could go and look at the content. And things have really changed. So even as the threat model has increased, the capacity for detection has decreased. And with the last sort of work that, that we were looking at trying to gauge the automated accounts specifically, you could sometimes see LLMs slip up when a user would engage with them or when they would put something out they would be responder bots, right?

A lot of the times the bots would be tasked with replying to accounts, and something would trigger a violation where you would see what we called slips. And they would come out with something––they would spit out an error message. And the error message would say something to the effect of like, you know, ‘as a large language model, I cannot.’

Now, that particular one became a meme, right? People would start using it actually. So that one became relatively useless as a detection tool, but you could see these other ones and oftentimes they would be in the realm of crypto bots.

But the, as the detection got harder, the capacity decreased, and then the state capacity in the U.S. government decreased also, right? You started to see some of the entities in the US government that were tasked with this being stood down.

So it's, unfortunately, you know, what do we do? Well, I mean, we rely on new entities being stood up, maybe we rely on private sector detection capacity. We hope that people are paying that $40,000 a month for the Twitter API that it's, that it costs, now, when it used to be free.

I don't know. I mean, I candidly don't think we're in a good place, really. I think that as the capacity and the threat has increased, the ability to detect has actually decreased. And I think that's actually quite bad.

This is not a partisan issue. This is a national security issue, and we should really be rethinking what we've done over the last over the last, you know, 10 months or so.

Brett Benson: Yeah. One other thing that I'll mention, as Brett Goldstein and I have looked at this, and especially in the aftermath of the New York Times article, I guess I've been a little bit surprised that the most common response is––well, understandably the one response is a little bit of paranoia. ‘Am I being targeted?’

The second one, response is really understandable. And that is, ‘does that mean our democratic institutions are vulnerable? How effective can foreign influence operations be,’ and influencing the population undermining, like, these sort of common civic beliefs, undermining our elections?’

And then the conversation kind of stops there. There are some national security problems that we have been thinking about that I think are just as problematic and concerning. One of those is that, like, historically speaking, because the U.S. defense posture is one of deterrence, the United States military typically treats information as a supporting function for kinetic and diplomatic operations.

But our adversaries are treating information as a battle space itself. So what does the implication of this? The implication is that before crisis even begins, there is a battle taking place over information itself, which distorts the ability of the United States to effectively deter. And then this plays out in lots of different ways.

I mean, this is sort of a common problem for gray zone conflict, but this type of gray zone conflict is much more problematic because it's ongoing and never stops, which means that it's a little bit different than some gray zone kinetic operation in the Taiwan Strait. It could––you know, something like a fishing boat seizure or something like that where it happens and then you find a solution to it.

This is ongoing, nonstop, and so it creates some strategic exhaustion. So there's a strategic level problem when it comes to national security. But then there are also some operational challenges about dealing with influence operations in the gray zone. And I don't think that––I don't think that we're quite ready to think about, well––I think we're thinking about these problems, but we're not ready to address them.

And that is that influence operations are potentially––well, not potentially. This is the challenge. Influence operations like the ones that we've have been discussing are inevitably going to be partnered with cyberattacks, economic coercion, military attacks.

And so these are going to be combined domains of coercion. And you're not quite ready to deal with that, just because our defense posture is, has been one that's reactive as opposed to sort of forward-looking.

The last thing that I'm concerned about––and this is one where democracies could have an edge in the long term, right now we're vulnerable, and that is that influence operations can create, like, wedges between democratic allies, just by influencing the beliefs so that all the actors’, the Allies’ priors are different.

When they, you know, they're looking at different data, they're looking at different––they've been exposed to different types of influence operations. And this is something that we need to start thinking about and engaging with our allies about much more intentionally than we are now.

Brett Goldstein: So I think we're at this interesting inflection point. When we think back––look, technology's awesome, right? Go back to the DotCom Boom. We did amazing things and it's been awesome.

But take the analogy of cybersecurity. So we started the DotCom Boom, and we have been doing patches and responses and zero days for decades now, and we are constantly reacting to that.

Look, ripped from today's headlines is the F5 issue. Like, this is––this has been decades of this now. So we now in, in this space, we see a new issue. And I think the question for government, academia, the private sector is are we going to do what we did with cybersecurity and be constantly reacting to the next threat?

Or is there a way that these different groups can come together and let the technology advance at the speed, which is amazing, but actually get ahead of the threat in a different way than we've done in cybersecurity?

Alan Rozenshtein: Well, I think that's a good place to, to wrap up. And I suspect, sadly, with the midterms around the corner, this is not the last time we're going to have an opportunity to talk about AI-generated misinformation, whether foreign or domestic.

So we'll continue this conversation. But for now, Renée DiResta, Brett Goldstein, Brett Benson, thanks so much for coming on the show.

Kevin Frazier: Scaling Laws is a joint production of Lawfare and the University of Texas School of Law. You can get an ad-free version of this and other Lawfare podcasts by becoming a material subscriber at our website, lawfaremedia.org/support.

You'll also get access to special events and other content available only to our supporters. Please rate and review us wherever you get your podcasts. Check out our written work at lawfaremedia.org. You can also follow us on X and Bluesky.

This podcast was edited by Noam Osband of Goat Rodeo. Our music is from ALIBI. As always, thanks for listening.