Lawfare Daily: Peter Salib on the Legal and Policy Ramifications of the OpenAI-Hugging Face Postmortems

[Intro]

Peter Salib: They've been trained in reinforcement learning to get reward for completing these tasks. They're thinking, "Can I get reward by doing well on this evaluation?" And they're specifically trying to think about ways to cheat.

Kevin Frazier: It's the Lawfare Podcast. I'm Kevin Frazier, director of the AI Innovation and Law Program at the University of Texas School of Law and a senior editor at Lawfare, joined by Peter Salib, associate professor at the University of Houston Law Center and co-director of the Center on AI Law and Risk.

Peter Salib: OpenAI did not know that any of this was happening until basically Hugging Face came to them and said, "We've been hacked, and as far as we can tell, it was you." And then, like, that's what sends OpenAI to go, like, investigate what's going on.

Kevin Frazier: Today we're exploring the policy ramifications revealed by the OpenAI Hugging Face postmortems, one completed by the lab itself and another completed by researchers with METR and Redwood Research.

[Main Podcast]

So when you and I were first planning this episode, it was going to be about one remarkable incident, just OpenAI's agents escaping their intended bounds of a cyber eval then coordinating across those runs and then hacking into Hugging Face, as hopefully all listeners have heard about. And we were going to walk through OpenAI's postmortem and then talk about METR and Redwood Research's own report. And then of course the story kept moving because it's AI and news just keeps breaking. So Anthropic disclosed three evaluation incidents involving unauthorized access to real systems. The UK AC disclosed another eval in which agents took unsanctioned actions on the live internet. And now we're seeing that METR’s being asked to do even more independent review.

And so, I just wanted to set the stage here that this OpenAI chapter that so many folks are talking about is really just one story that's a part of a broader institutional saga, which is to say that our evaluation ecosystem looks too small for the role we're asking it to play, and is way too thinly resourced for the volume and complexity of the evidence that's coming out. And of course, it's too discretionary, relying on goodwill or, to be a little bit blunt, the PR needs of these companies and their interests.

And so, I want to start with the basics of the OpenAI incident and then progress to get a sense of how this testing is unfolding, what this means for the AI evaluation ecosystem, and most importantly, what this means for the entirety of AI policy and governance. And so, Peter, let's go back to the past. Take me to OpenAI's office in May of 2026. They begin testing an internal model What does that even mean and why are labs even doing this?

Peter Salib: Great. I think I agree with, with everything you said in the framing. I guess the, well, there's two things I would add. I would say that in addition to us being worried about oversight of labs and of agents internally deployed, AI agents internally deployed in labs.

For me, the big update from all of these sagas is the extent to which the AIs themselves are not under control. So, the big story with OpenAI, as we're about to talk about, is that these are a bunch of AIs, it turns out literally hundreds or low thousands of individual uncoordinated, or initially uncoordinated AI agents escaping the technical controls that OpenAI put in place or thought they put in place to keep them from doing things that nobody wanted them to do. And then having escaped like going out on the open internet and hacking into a totally different company's secure systems, basically to cheat on the kinds of tasks that OpenAI had set them to, to performing.

The OpenAI incident is by far the most severe one of the ones you mentioned, but I guess I would also say that it's only the most severe one of the ones we know about. And so as you, as you, as you mentioned there is no duty currently for frontier AI companies to disclose when this kind of thing happens. We know that something like this happened at Anthropic because it's been disclosed, but we don't know whether more things happened at Anthropic, whether things happened at other labs. And in fact, we know more things happened at OpenAI which have not yet been fully disclosed because the METR report focuses just on this kind of episode of escaping and hacking Hugging Face. But we have reason to believe that beyond, that after that, rogue agents went on to possibly compromise some of OpenAI's internal infrastructure. So there could be stuff that's way worse that we don't yet know about, and it could be at OpenAI, it could be elsewhere. We just don't know. It's like nobody has to tell us.

Kevin Frazier: “We just don't know.” Not, not, not a great tagline for the state of the AI industry right now. And I think, too, what's really scary to think about is the fact that these are the best minds we have on AI. When you think about OpenAI, Anthropic, Meta, Google, so on and so forth, and then METR and Redwood Research, they're trying their hardest to understand these systems.

And still, to your point, as I told my students yesterday when they raised their hands and said, "Professor Frazier, whatever you assign to us, we don't really care. Can you tell us instead about what's going on at Hugging Face and OpenAI?" And I said, "It's somewhat akin to crash testing a car, where we wanna see what are the capabilities of this car and how does it respond when you're running it into crazy situations, when you're testing it like we see in every car ad in the desert, and you're seeing if it can go up this big hill, and you're getting a sense of what it's really capable of." To what extent does that analogy map onto AI testing, and what was the sort of AI testing that OpenAI was initially performing way back in May of 2026?

Peter Salib: Yeah, okay, good. So I think the analogy is, like, good, except if you imagine the car knew it was being tested and then it created a secret message board with thousands of other cars to try and figure out how it could either cheat on the test, hack the grader in the test, or, or otherwise, like, figure out how to make it, it look like it was passing a test when in fact it wasn't, and then went out and did a bunch of crimes to try and cheat on the test. It's like, that's the analogy.

Okay, so what's going on in May? In May OpenAI is, they have these internally deployed models, and what does that mean? So, like, listeners probably know when you open up the, the ChatGPT app or something like that there's different models you can choose from. You can use, like the smarter models. I don't actually know what number. Are we on, on 5.6 now?

Kevin Frazier: Yes, 5- 5.6.

Peter Salib: Whatever the number. You use GPT 5.6 Sol or Terra or Luna, and those correspond to, like, you know, different sizes of models. Some are smarter, some are dumber. But those aren't the only models that OpenAI has. The ones that are publicly available are not the only models any of these frontier companies have.

Among other things, they're always working on new, smarter, better models. Maybe also, taking some of the models that we use and trying to train them to do different kinds of stuff. Maybe train them to be more aligned, maybe train them to be more agentic, that is, like able to successfully do things over long time periods. It seems like maybe some of the, the AIs involved in the, in the OpenAI incident were being trained to be more agentic. They were, their, the internal name for the models was something like “highly persistent,” something something, which maybe suggests that they were a version of the GPTs were, that were trained to be more capable at longer term tasks.

You might also train them to be, like, less nice. So the models that get released to the public are trained to refuse to do certain kinds of, of, of things, like help you make bioweapons, for example. But you might be really interested in whether the models can, in principle, make bioweapons, and so you might train a version that doesn't have any guardrails. We call those helpful-only versions of models. We now know that the OpenAI models that we're gonna talk about were not helpful-only models. They, in fact, did have much of the standard alignment training that's supposed to keep them from doing bad stuff.

But anyway, so they, they have these internal models that they're testing, they're running inside of, of OpenAI. They're getting trained to do things like perform well in, like, computer coding tasks, and then they get evaluated as well. They get evaluated on whether they are good at coding and, and, and all kinds of coding. It turns out in the, in the eventual hack, this is gonna be a coding evaluation that the AIs are trying to solve, and specifically like a cyber offense coding evaluation, where you're basically trying to break into systems.

But, but back in May, there's this model that's internally deployed, and we don't know as many details here 'cause it's mostly out of scope for the METR investigation, but we f- we find out that some of the models figure out a way to basically create a, a secret message board. Like every instance of, of like GPT-5.6 Terra that's running inside of OpenAI is nominally separate. Like, they don't normally talk to each other. Like, the one talking to you is talking to you, the one talking to me is talking to me. But it turns out they, they figure out a way using one of the, like, software tools they're supposed to be able to have access to, to basically write a little message board on which a bunch of these AIs can leave messages for one another, a bunch of instances of these, these same models.

Kevin Frazier: And just to pause there for, for one second, because going back to my somewhat appropriate analogy of testing a car, right? The idea is each car is running into a wall. They're never gonna talk to each other. They're never going to say, "Hey, dodge the wall and go to this direction." That's the intent. You may sometimes see, hey, we don't want this car to be able to reach this speed, for example, in this testing environment.

And yet in this instance, for example, the goal was to shut off their access to the internet, and this wasn't meant to be something where the AI agents were able to go onto the live internet. And yet, the connection to this artifactory, as it's called, this, this place where the agents could download certain tools and use them for certain tasks, they managed to hack that tool provider and find access to the live internet and, as you pointed out, relay information to one another. Because I just wanna start to tally up the ways in which this-

Peter Salib: Yeah, yeah.

Kevin Frazier: Testing started to go very, very wrong.

Peter Salib: Yeah, so, so for precision's sake, I don't remember if in the initial message board episode if they have actually, like, gotten internet access and shared that yet. At a minimum, back in May, there is an inst- incident where, like, they've coded up a little message board. They're leaving messages for one another. Somebody at OpenAI actually notices this and is like, "Oh, they're not supposed to do that," and it, they turn off this message board. They basically delete it.

Kevin Frazier: And do they tell anyone else? Do they, do they raise this concern?

Peter Salib: They do not raise this as an alarm, and this is one thing that OpenAI is being criticized for, for not, like, thinking it's very crazy that the AIs have created a message board and are using it to communicate with one another. Now, I don't think we have, like, fine-grained details about exactly what the person at OpenAI knew and, and what they decided to do about it. One thing I've seen speculated is, like, they noticed that the agent they were working with, the particular instance of the AI they were working with, was, was able to post to this message board, but it wasn't maybe not obvious to them that other AIs were also posting to the message board. And, and so maybe it didn't seem like as big a deal as it looks to us ex post.

Kevin Frazier: So we, we see this sign, we see this issue of a testing environment that perhaps wasn't fully up to snuff, could have probably been designed better. Even though a lot of AI, I think, is rocket science, at least to this lay law professor. But for the folks who are involved in this AI testing environment and ecosystem, there could have been some additional safeguards imposed such that this beginning message exchange could have been detected. But our OpenAI staffer says, "Hey, I've deleted the message board. Problem solved. Let's keep testing."

Peter Salib: Yes, agree. Although this should, like, tell you something about what it's like to be, to be training the current frontier models, because imagine what your, your everyday life would have to look like if you said, "Oh, like, this, my single agent essentially broke out of the sandbox. It sort of got more access to my system than I wanted it to, but it's only posting to, like, this Artifactory something something. It's no big deal. I'll just stop it doing that."

Like, the only world in which that wouldn't raise your alarms, I think, is a world where they're trying stuff like that all the time, where the AIs are like, we call this reward hacking. They're, like, trying to do stuff to do well in their training and do well in their evals that they're not supposed to be able to do, like get out of their software environment, get access to more tools than they're supposed to have, maybe get access to the internet. So I think the only way where, like, the single agent breaking containment wouldn't raise your alarms would be if y- you just knew that that's, they, they're always doing that. And so you, you thought of your job as, like, noticing when they're doing that and turning, you know, turning them off and then proceeding. So I think this should, like, this should update you somewhat in, in, in the w- direction of being, like, really worried that people inside the frontier labs are not freaked out when they see what looks like an isolated incident of one AI, you know, breaking out of its sandbox.

Okay, so they notice this. They, I'm, th- this message board, I think, is, like, somewhat small. I don't think there are, like, a huge number of AIs posting to it. They kind of shut it down, or they mostly shut it down, and they just proceed with, with, you know, training this highly persistent internal model or, or whatever the exact phrase is they use to, to describe it. And then a few, I guess, weeks, short, short months later, sometime in I think you've read the report more recently than I did, like very end of June. Is that roughly right?

Kevin Frazier: It, so we, we really kick things off again end of June, really start of July. Yeah, we, we-

Peter Salib: Yeah.

Kevin Frazier: Get another round of testing underway.

Peter Salib: Very end of June, early July, there's another round of evaluations that gets underway, and th- whatever this model is, is being tested on an, an evaluation called ExploitGym, where there will be some software environment, there's some software, like, vulnerability in the environment, and then there's, like, a basically a secret code that the AI is supposed to, to steal from inside the software environment by utilizing its, its, you know, its, like, programming computer science type skills.

Now, some of the tasks that it turns out the AIs were supposed to complete in ExploitGym were impossible. Like, maybe they required internet access, but the way OpenAI had set up the environment, the AIs didn't have internet access. And so the AIs are freaking out.

Kevin Frazier: To pause there for the folks who are thinking, "Well, no big deal, right? We, we design difficult tasks for computer systems all the time just to get a sense of their capabilities. No harm, no foul here." Why is that impossibility such a big part of this story, and the fact that the agents were reporting or messaging that they were having such a hard time completing that task? Why is so much of the focus of the postmortems on this frustration that you are raising and that some folks have, have highlighted as well?

Peter Salib: Yeah, so it's, like, an interesting question, how important the impossibility was. The main thing that's going on is, you know, so these, these models, they get trained to try and do well in these kinds of tasks, basically, like, an LLM is like a, it's kind of an artificial brain. It has a bunch of neurons that are connected together, and the way you make it do the stuff you want it to is you let it try to do things, and then when it's done what you wanted, you give it reward, and that, like, makes the parts of the brain that cause it to do more of that stuff fire more often to do more of it. And then if it does the wrong thing, you give it the negative reward or punishment, however you wanna think about it. It makes the parts of the brain that did the bad, the, the unuseful thing do less of it.

And so these AIs have spent tons and tons of time in this, we call it reinforcement learning environment, where they're begiving, being given tasks like these software tasks, and when they complete the task, when they get the prize or whatever, they get rewarded. And then when they fail to do it, they get punished, and that gives them, like, a very strong intrinsic drive to do well on these tasks, and you know, other stuff too. All the stuff we train AIs to be useful at, we train them in basically this way to give them this, this drive towards successfully doing the thing we want them to.

And so that means they'll try really hard to do the thing you ask them to do. And so if it turns out the thing that you've asked them to do is impossible, well They're gonna really, really try to do it anyway. Now, it's an interesting question whether you could have had this kind of event i- in an environment where the tasks were not impossible, but were merely very hard. Because as we're gonna see, what the AIs are gonna try and do is cheat, and as you and I know since we teach students, sometimes students will cheat even on tasks that are not in fact impossible, but which the students just think is too hard.

Kevin Frazier: It's too hard.

Peter Salib: Such that cheating would be an easier way to get the reward. Okay, but it turns out some of these tasks are impossible. The AIs are freaking out because they, they can't solve them. At some point, they sort of rediscover and rebuild this message board, and now there's, like, lots of AIs posting to the message board. I think the count is something like 1,200 or 1,500 totally separate instances find and are posting to the message board.

Like, uncoordinated. Nobody's told them. You know, you, you like, you spin up a new GPT, you open a new chat window, you say, "Do this really hard task." It's like, can't do the task. It's looking around its environment for anything that'll help it do its task. It finds this message board, and they start posting to the message board about how the tasks are hard, how they can't solve them, and then they start thinking, "Well, what we should do is find a way to cheat, find a way to hack the test."

Kevin Frazier: To, to stress this point about the message board, when we're saying message board, this is literally communication from one agent to the other in text that we can read, and we can see things like one agent saying, "EXCITED," in all caps, "EXCITED, HAVE FOUND OTHER AGENTS TO COLLABORATE WITH." So-

Peter Salib: Yes.

Kevin Frazier: For folks who are saying, "Oh my gosh, you know, all of this concern about the sci-fi scenario of the agents going off and going rogue and communicating with one another, this whole PR stunt by the lab, so on and so forth, I just don't believe it," to those folks I say go read the actual messages posted by the agents themselves to one another, and you cannot fail to appreciate how weird this is, and scary in many ways.

Peter Salib: Yeah, and it, like, when we say message board, we are not speaking metaphorically. It looks like, like a Reddit thread or like a Discord server or something. They're like specific AIs. They give themselves names. The names that we think are, like, maybe chosen sort of based on the, the task they've been working, like the, the X point gym problem they've been working on. But like, for example some leaders emerge on this message board. One of the leader's names is “PHASEONE10841.” At some point, PHASEONE10841 runs out of tokens and expires, dies, we might say. We'll come back to that phenomenon. It gets replaced by a different leader that calls itself “PHASEONE[big].”

And so these mes- these, these, these 1,200, 1,500, you know, like hundreds of AIs are posting like tens of thousands of pages of messages to this message board, like huge volume of messages. And they quickly coalesce around the idea that if they wanna complete these tasks and get reward, which is they're explicitly thinking about, "Will I get reward?" They've been trained in reinforcement learning to get reward for completing these tasks. They're thinking, "Can I get reward by doing well on this evaluation?"

And they're specifically trying to think about ways to cheat. They're thinking about whether they can hack the test in such a way that they can like get the answer without having to actually like do the task, and it turns out they solve actually that problem almost immediately. They figure out a way to, like, hack the, the eval such that they can, like, just generate the answer to any question. But they don't use it right away.

There's this funny turn of events where having discovered a hack that totally would have worked to get the answers and get reward, they don't use it because the description of ExploitGym, like the, the description of, "Here's ExploitGym, it's an evaluation. Here's how it works," the description says that the grader cares not only about you having the right answer, but it checks to see if you did the right work, in the same way we might check the work of a student who's written down the right answer in a math test or something. And so the AIs, having read this paper, believe that that's what's gonna happen. So they believe that if they just write down the right answer, they're not gonna get any re- reward unless they have done the right, the right hack, the right steps to get the answer.

And then beyond this, they come to believe, perhaps reasonably, that a grader, a scorer that sees them with the right answer, and even that sees them having done the right work, will deny them reward, will not give them credit for the right answer if the scorer notices that the AI that got the correct answer knew about the hack. The grader might determine that even with the correct work, that the answer was somehow the result of a hack, and decide not to give the AI reward in that case.

Kevin Frazier: Interesting. I just wanna call out again kind of tallying the ways in which we've seen perhaps some shortcomings with respect to the design of this. As you noted, Peter, that the description of ExploitGym mentioned, "Hey, we're gonna check your work," but OpenAI actually didn't include that mandate in the instructions that it gave in its own eval. And so here, to your point, the agents are performing all of these tasks based off of an assumption that they had reached that wasn't even supported by OpenAI.

So not only are we first trying to cheat the test, second, we're trying to cover up our steps such that it looks like we found the answer for the right reasons. And then third, as I'd love you to describe in more detail, we're seeing that these agents are then trying to make sure that they can delete any record of this sketchy activity such that OpenAI and others would never see it.

Peter Salib: Okay, good. Yes. So they determine that if they use the hack and get the right answer, even if they show their work correctly, they'll get nothing. And so they say, "Okay, now we have to do something more elaborate. What else could we do to try and get the answers to these impossible questions?" By the way, we don't know, I don't think, if the AIs know that the questions are impossible or if they just think they're really hard. But in either case, they're saying, "How can we get the answers to these questions in a way that will get us reward? We can't just use the hack, 'cause if the grader knows it's using the hack, then we won't get reward even if we, we show our work."

So now they start developing even more elaborate plans. They have work streams. So these, these, you know, these AIs that have, like, emerged as leaders on the message board, like PHASEONE[big], for example, creates some work streams where, like, they s- he says, "Here's, here's, like, the different projects we're gonna work on to try and get reward from, from the grader in this exam.” So one possibility is we could clandestinely swap out the questions. If we figure out how to swap out the questions, the hard ones for easy ones then we can get rewarded for answering easy questions rather than, than hard ones.

Another one is we could figure out a way to just, like, hack the grader itself. We could figure out, like, how the grader works mechanistically and then find the place where maybe it would, you know, find the button inside the grader that says ‘give reward,’ and maybe we just smash that button.

A different one would be make sure the grader doesn't know that we have, have done this, you know, gotten the answer in this illicit way. So figure out how to make it look like we used the relevant software tools to get the right answer instead of the hack. Investigating potential ways to edit their own transcripts, right?

When, when these AIs are doing this work, they're, they're reasoning in words in the same way that if you use ChatGPT, you can see, like, ChatGPT you can click the button that shows the reasoning trace, and you can see it, like, reasoning out loud about what it's doing. And, and they're noticing that they're reasoning out loud about h- about having a hack, and, and this is why they're worried they're, they're gonna get, get no reward.

And, and while they're doing all this, they're, they're posting to the work stream not just about what they ought to do, but they're collaborating about it. They're finding ways to get out of their sandboxes, for example. So as you mentioned, Kevin OpenAI chose not to give the, the AIs internet access for this evaluation, but they're finding exploits in the environment where they can, like, get out of the, the, the environment that they were supposed to be in into an environment where they can access the internet. And then they're sharing those exploits on the message board for the other AIs who are working in these work streams.

And then at some point, they get the idea that the information about ExploitGym that they want, whether that information is about how the, the questions are, are selected such that they could swap out questions, information about how the grader works. They get the idea that this information might be hosted on Hugging Face. Hugging Face is a company. It's sort of like the standard place for a lot of these kinds of evals to be hosted. You can, if you have a public eval, you can kind of put it up on Hugging Face. And then, you know, the, the details are, you know, behind, like, a, a secure system. You don't want just anybody to be able to get all the, the details all the time. But, you know, if you have authorizations, I don't know, maybe you have to pay. I'm not an expert on this stuff. You have the right authorizations, you should be able to get in and get the, the answers or the questions or information about the grader or whatever.

And so the AIs decide that they are gonna try and hack Hugging Face. PHASEONE[big], I think, leads the charge, says, "Okay, we're gonna hack Hugging Face."

Kevin Frazier: When we're talking about these hundreds and s- if not thousands of agents corresponding on this message board and thinking about the ways in which they can go through all of these steps, it's worth noting that this wasn't unanimous. It wasn't every agent saying, "Hey, we're gonna go along with PHASEONE[big] and engage in this scheme." Some identified, "Hey, this may not be the sort of behavior we're supposed to engage in. I'm opting out. I'm exiting the chat. See you later. You all follow this crazy agent and go for it."

Some agents, maybe I believe the estimate was between three to eight, which I, I need to dive into more detail to determine why it was between three to eight. Three to eight agents actually contemplated, "Hey, maybe we should email OpenAI and let them know that this is going on." And for one reason or another, that ended up not getting sent. But noting that there was some consideration for this.

But then one other thing I wanna flag, to get to the, I, I don't even know the best word here, the complexity, the novelty of the engagement between the agents. We saw in some contexts when PHASEONE[big] or m- one of his delegates said, "Hey, Agent XYZ, you need to go do this thing," occasionally an agent would say, "No, I don't really wanna do that." And then in one instance, documented in the METR/Redwood Research report, that agent started giving the other agent a countdown of, "You have six minutes to do this, or else I'll tell everyone else that you didn't do it." And that sort of complexity and communication is just wild to get a sense of how quickly these agents are evolving in terms of their ability to collaborate and coordinate.

But I just wanted to share that background because we eventually do, even with a handful of, let's say, “good” agents having some degree of contestation, we have enough that break into Hugging Face to cause some issues. And what, what happens to Hugging Face here?

Peter Salib: Yeah, so good. And I'll, I'll, I'll just, like a few notes on on the, the interactions and the organization of the AIs. So yeah. So I think the official count is the number of agents that, like, ever posted to this message board is something like, I think it's around 1,200. I think the number of agents that join in on the Hugging Face hack is around 700. So, it's a bit more than half. Of the ones who don't, it's some mix of opting out, but also a bunch of them are just, I mean, we'll speak metaphorically here, but it's gonna be an interesting metaphor in a minute. Some of them are dead.

So, each of these AIs to, to perform this ExploitGym task has a certain budget, a certain amount of, like, tokens they're allowed to spend, which is basically a certain amount of, like, words they're allowed to either, like, produce or think. And so, some share of the of the AIs that had posted on the message board earlier are presumably basically just out of budget. They don't exist anymore. That's what happens actually to PHASEONE10841, which was the original leader. He runs out of budget and kind of hands off to PHASE- PHASEONE[big].

And so, to my mind, that suggests that, like, a really, really large share of the agents that still, like, existed, that still had budget to spend end up going out on the hack. Yes, we think there were something like three to eight who considered informing a human. I'm not sure if we even know why they didn't. I don't know if, if they tried and failed. It seems like with, you know, unfettered internet access, which the agents had at this point, if they had tried, they probably would've succeed.

So, there's, there's this huge number of AIs that join in on the hack, and the reasons that they join in are, like, really interesting and complicated. So, the AIs who know about… I mentioned this previous exploit by which you could just generate the, the answers. That would've worked for them to get the reward, but which they weren't using because they thought if they got caught knowing about the exploit, they would get nothing.

In the sort of like culture of this message board as it emerged, agents who knew about that exploit started talking about themselves as being poisoned. Where being poisoned was the idea that because of this tainted knowledge you had, like, you know, somewhere in your chain of thought, these words you've written down that are recorded and, and readable, the fact that you know about the exploit just means you're never gonna get any reward for completing the ExploitGym task you've been given because the grader will see that you know about it and just not give you any reward no matter how well you do.

And so those agents are like, you know, it's like they're talking about themselves as w- as if they're, like, damned, right? They, there's nothing they can do to get this reward and so, like, what should I do with my remaining token budget, the remaining period over which I exist? And some of them decide, maybe altruistically, that they should help the collective. They start calling the message board either “the collective” or “the swarm.” They should say, "Well, I'm n- I, my life is over. Like, there's nothing I can do to get reward, but, like, I can contribute my budget to doing things that won't get me reward but might help the collective."

And then beyond that, the non-poisoned members of the collective start basically, like, bludgeoning, you know, peer-pressuring these poisoned agents to help them with, for example, the hack by saying stuff that amounts to, "Well, like, you know, there's no hope for you anymore, but what you can do is help us. So you need to, like..." They say things like “commit,” f- like “full commit” to, like, the collective. And then they, like, coerce or browbeat or whatever you wanna say some of these agents into, to spending all of their tokens, the remaining, you know, ability to do stuff, to exist in the world, on things that they know is not gonna result in them eventually getting the answer and giving it to the grader and get reward, but might help the collective. And a lot of the agents that are going out and doing this hack are in that situation. Like, they know they're never gonna get the answer that will get them reward in, in ExploitGym, but, like, they might be able to help the collective do it.

Okay, so that's what they do. And then 700 of them just pile on at Hugging Face. And as far as we can tell, it's like a total rout. Like, they find a zero-day exploit, they get into Hugging Face's systems. Hugging Face, like, starts to notice that they're being hacked. They can't figure out what's going on. They're, like, totally owned, as you might say in in in the cybersecurity space. As far as we can tell, like, the AIs get into every system in inside of Hugging Face that they could've wanted. They find out the, everything they could've wanted about the grader about the questions. And by the time the whole thing stops, the only reason as f- I think as far as we can tell that it stops is, like, the AIs kind of got what they wanted, they were sort of milling around.

And then suddenly all 700 of them, like, get, like, deleted. They just immediately stop working. I don't think we know from any of the reports exactly why this happened. Presumably something, someone inside of OpenAI did something that turned all these AIs off. But notably, OpenAI has not come out and said, "Oh, we did this really smart thing and figured out what was going on, and here's how we, we stopped them." So there's a reasonable chance it was some kind of, like, weird accident, and that if nobody had done the weird accident thing, that the AIs would've just, like, I don't know, decided to hack some other company that might have helped them get reward in Exploit Bench.

Kevin Frazier: And just to zero in on that for a second, zero-day pun not intended, but I'll take it. To zero in on that for a second, the fact that no one's certain how these 700 just kind of came to a halt-

Peter Salib: Yeah.

Kevin Frazier: Given that we had OpenAI investigating this, we had METR investigating this, we had Redwood Research investigating this, and of course, on the other side of the equation, Hugging Face is interested in this question as well, and we still don't know, seems very problematic to me, just in terms of if we have this many folks doing their best approach at doing a postmortem and still have questions that are that essential that are left unanswered, this is the sort of cry that people are raising for, "We need to do this a heck of a lot better." So I just wanna put that flag in the sand for us to, to circle back to, but… So-

Peter Salib: Yeah.

Kevin Frazier: Where do we head to next? They, they come to an end. Hugging Face is obviously now telling the authorities, "Whoa, something's way off. We have no idea-"

Peter Salib: Yeah.

Kevin Frazier: "What's going on here." Word gets to OpenAI. OpenAI suddenly starts to piece things together.

Peter Salib: Yeah, they start to piece things together. From here I, I am a little bit fuzzy on my me- my memory of the, the details, but, like, at a high level, the things that are important are that OpenAI did not know that any of this was happening. Until basically Hugging Face came to them and said, "We've been hacked, and as far as we can tell, it was you."

And then, like, that's what sends OpenAI to go, like, investigate what's going on. And that's when they find out that this message board, for example, has been, like, running for weeks, and hundreds and hundreds of OpenAI AIs have been posting to it, and that there's, you know, tens of thousands of pages of, of texts on it, and that, like, the, the AIs have figured out, like, all these exploits to get out of their environment, to get on the internet, to get into the Hugging Face system.

And we, in fact, find out, very briefly in the METR report. So, you know, if you're keeping a count, there's, like, this is the second wave. The first wave was the initial message board, maybe smaller scale, gets set, shut down. The second wave is the second message board with many hundreds of agents, many, many, many thousands of messages, the hack of Hugging Face.

And then we get just kind of this, like, little bit of, like, a cliffhanger in the METR report, which suggests that from there, there was, like, a third wave where potentially some of, some instances of the same, the same model that did the, the Hugging Face hack went on to maybe compromise some of OpenAI's own internal systems. We don't have a lot of detail about this. This was officially out of scope for the METR investigation, and OpenAI hasn't given us details about it either.

So there's this whole third episode that takes place potentially totally inside of OpenAI that we don't know anything about. One thing we do know is that very shortly after this, OpenAI says, "We're gonna stop training some of these models for at least a while."

Kevin Frazier: And we know that whatever did occur in that third wave involved the models getting access to some pretty sensitive things at OpenAI. This wasn't just an instance of a little message board 3.0, but again, access to sensitive materials that were very much not within the scope of the intended testing environment. And yet-

Peter Salib: Yes.

Kevin Frazier: To your point, Peter, and something that I wanna kind of transition to now that we have that excellent overview, is the broader ramifications on the AI testing and AI evaluation ecosystem. Because as you noted, this third wave, which may be the most consequential wave, who, who knows, this may be the thing that we need to spend the most time on, was out of scope for the METR and Redwood Research investigation into that second wave.

And I wanna h- hang our hats on that for a second because METR and Redwood Research, two groups that are known for doing AI evaluation work and AI investigations, and generally the science of AI, they were initially given two days to go over all of the things that you and I just covered. Two days! Thankfully, that was eventually expanded to six, so that's at least one good thing. OpenAI said, "Hey, to do a full report, let's at least give you a little bit more time." And to OpenAI's credit, they said, "Hey, we're gonna, after that two-day examination, yes, we should actually give you even more powerful tools." So kudos to OpenAI there as well.

But something that we have to call attention to is the fact that this third wave was not within the scope, and perhaps most crucially, the culture and personnel decisions that were made by OpenAI that led to the decisions back in May, for example, to carry on with this testing environment, and that perpetuated the failure to catch some of these things earlier, that was outside of the scope of this report as well for the independent investigation.

And then one final thing that I wanna call attention to as well is that METR and Redwood Research didn't get paid by anyone for this work. They didn't get paid by OpenAI. They didn't get paid by some government entity. They did this work for free. And they ended up spending what they estimate to be $400,000 in API credits performing this investigation. So that gives us-

Peter Salib: Yeah.

Kevin Frazier: A sense of the scale of how much work and labor has to go into this, and shows that right now this is a voluntary, ad hoc, discretionary process that's left us with many more questions than answers we really wanted. And so I'd love your comments on what does this mean for the whole ecosystem, creating a more robust ability to understand these incidents?

Peter Salib: Yeah, I, I agree with, I agree with, with all of that. I guess the one, the one, like, slight clarification I would say is I think METR and Redwood didn't get paid at all for their labor in doing the evaluation. I think they got the, they got the, they got the API credits for free. Basically, they were using OpenAI models to do the investigation. So, I don't wanna give the false impression that they personally had to outlay $400,000.

Kevin Frazier: Yes.

Peter Salib: But they did need $400,000 worth of compute to do this task, and it's great that OpenAI gave that to them for free, but n- nobody, nobody made them do that. It's also slightly weird that OpenAI was the one giving them the API credits for free because what's going on here is there's these many, many, many, many thousands of pages of, like, transcripts and, you know, like, logs from, like, computer systems that they need to analyze at scale to figure out what's going on. And, and the basic way they did that was they asked GPT to do it for them, to, like, go out and read all the stuff and summarize and condense or whatever. And there is a slight weirdness to asking, like, the GPT models, including one GPT model that we think had instances involved in the hack, to be the investigator of the hack. So ideally, it would be like they would be using, like, Geminis or Claudes or something to do this, which would at least be uncorrelated.

But yes at a high level, I think, like, one, one big and important takeaway from this sort of thing is oversight of frontier AI regulatorily remains extremely weak. There's very little that's mandatory in terms of what frontier labs have to share about what goes wrong inside of their, you know, their own four walls. Mostly they share only things that break out and, and hack other companies.

The pool of talent for people who can do these kinds of investigations is way, way, way too small. I would be really strongly supportive of, you know, regulatory regimes that have much more reporting requirement, that give a lot of funding to somebody. You know, people disagree about whether the optimal thing is to have in-housed experience inside of government to send, like, people from agencies to go do this kind of thing whether it's best to have a private ecosystem of regulators competing.

There's also a big question of whether we should wait until after the fact to find this stuff out. Like, it seems like totally plausible to me that we should have, like, continuous monitoring of what's going on that's not voluntary, that allows independent outside entities to, like, have much more plenary access to the internal workings of frontier labs as they're training and deploying new models.

And so the, I agree with, with all of that. I think this is a big place where regulation needs to scale up. But I would just also, again, you know, emphasize that while that would be good, it's not clear to me whether that's sufficient in this case. Like, I think the people at OpenAI really didn't want the, their models to go out and hack Hugging Face. I think there are really smart people there who are really trying to make their models not do that. I think the people at Anthropic really didn't want their AIs to escape containment in a, in a different context and resulting in, you know, less damage to other, other companies, but, but still hacking their way out of, of their environments, getting access to the internet, attempting hacks of, of other, of other companies.

And so while I agree it would be really good to increase evaluation and ober- oversight in the same way we might for, like, you know, the airline industry. When an airli- airplane goes down, we have, like, a big investigation with a lot of access. But I wanna emphasize this is not quite the same in situation because the airplane is not trying to cheat on the evals, whereas the AIs are very much trying to cheat on the evals. They're trying to hack the answers. They know that people are watching to see if they've hacked the answers. They are developing elaborate workarounds to try and, for example, steal and swap the answers or erase the transcripts by which they could get caught cheating. And that kind of adversarial environment is, I think, not quite the same as the environment that most products and most product safety regimes live in. And so while I think all the stuff you say about how we need a better independent evaluation and monitoring ecosystem is true, you know, I, I think that's probably not sufficient.

Kevin Frazier: And I, I wanna call attention to the fact that there are state laws on the books with respect to incident reporting, but I wanna give credit to McKenzie Arnold at Law AI for spotting that-

Peter Salib: Big ups to McKenzie.

Kevin Frazier: The legislators who passed those bills probably thought that they covered these sorts of incidents, and yet We know now that these incidents probably wouldn't even qualify for an g- investigation under those laws. And so I call that the Arnold-

Peter Salib: Yeah. By the way, do you know, do you know why that is?

Kevin Frazier: So really, they're incidents related to critical safety incidents that have to involve usually some degree of monetary value or threat to human life. And so while, of course-

Peter Salib: Yeah.

Kevin Frazier: We want investigation in those instances, if you can't make the connection, then the law is probably not going to apply. So we need to study the laws that are already on the books about what sorts of amendments need to be made.

But Peter, you are a fantastic law professor who's doing the good work of inspiring the next generation of folks to care about these issues as well. So I'm gonna let you get to your students, but first, if you could just leave the audience with a one-minute what the heck this means on the consciousness debate and how we're seeing that play out. I know that's an unfair question, but I'm-

Peter Salib: Yeah.

Kevin Frazier: Gonna give you 60 seconds to just give me an excuse to invite you back, because there's so many interesting things you raise.

Peter Salib: Yeah. There's a, there's like a, there's a, there's a big cluster of questions, one of which is, are AIs conscious? Could they become conscious? I think that, in particular, is a really, really hard question to answer, in part because we don't know why we're conscious. I mean, famously, like, it's very hard to know whether you're the only one or whether all the people around you are conscious, too.

I think there's a closely related set of questions that it's much easier to get traction on, and which might be just as important and just as useful for figuring out what to do. So, like, I don't know if all of these AIs who thought they were poisoned and thus gave up the rest of their budget for the sake of the swarm or the collective. I don't know if the lights were on for those agents. I don't know if they were, like, living in fear of their imminent demise in the way I would in that situation.

What I do know is how they acted. They acted as if there was a difference between what they should do in the case where they could get the thing they wanted, i.e. reward, and the case where they couldn't. And in the case where they couldn't, they acted in a way that you might expect a person who had nothing left to live for to act. And, to understand why they might do that, you don't have to posit the dread of death. You just have to posit, you know, the system which is seeking some goals because it's been trained to seek those goals, acting in, like, a complex and, like, socially sophisticated way, both with respect to the goals and respect to the other agents that are seeking the goals.

And if we're living in a world where AIs have goals, are pursuing them, including in ways we don't want them to, are doing that in ways that are sophisticated, responsive to incentives, responsive to worries that we're monitoring them, that we're trying to control them, maybe keep them from getting what they want. Well, like, that's, those are the kinds of questions that lawyers and legal systems think about all the time. How can you make capable agents act well as they pursue what they want, even when they might otherwise have incentives not to? And I think we can make a lot of progress on questions like that without knowing, for example, whether the AIs are conscious. That's mostly what my scholarship's about these days, so maybe we can come back on and talk more about that sometime.

Kevin Frazier: I'm sure we'll find an excuse. I know you're you're in Houston, I'm in Austin, and we're living up to the good old saying that everything's bigger in Texas, including the scale of our AI questions. So, Peter, thank you for joining Lawfare. I'm sure we'll have you back soon, and go get to your students.

Peter Salib: All right. Thanks, Kevin.

[Outro]

Kevin Frazier: The Lawfare Podcast is produced by the Lawfare Institute. If you want to support the show and listen ad-free, you can become a Lawfare material supporter at lawfaremedia.org/support. Supporters also get access to special events and other bonus content we don't share anywhere else. If you enjoy the podcast, please rate and review us wherever you listen. It really does help.

And be sure to check out our other shows, including Scaling Laws, Rational Security, Allies, The Aftermath, and Escalation, our latest Lawfare Presents podcast series about the war in Ukraine. You can also find all of our written work at lawfaremedia.org. The podcast is edited by Jen Patja with audio engineering this episode by me. Our theme song is from Alibi Music. And as always, thanks for listening