Lawfare Daily: Sam Altman with Ronan Farrow and Andrew Marantz

[Intro]

Andrew Marantz: The articles of incorporation under which OpenAI was set up as a nonprofit make very clear that they have a legally binding fiduciary duty to do certain things. And what we chart in the piece is this kind of years long process of the company kind of converting itself out from under those restrictions.

Kate Klonick: It's the Lawfare Podcast. I'm Kate Klonick, senior editor at Lawfare with Ronan Farrow and Andrew Marantz, writers at the New Yorker and authors of a huge 16,000 word piece published last week in the New Yorker titled “Sam Altman May Control Our Future—Can He Be Trusted?”

Ronan Farrow: Sam Altman is one figure. He's a consequential one, but he's especially consequential as a window through which to view a wider dynamic that critics in the piece describe as a race to the bottom.

Kate Klonick: Today we're talking with Ronan and Andrew about the legal and national security implications of a number of things that are revealed in their article and discussing what some of the policy options might be going forward.

So I wanna start for—just to take a moment, for the better's rule of headlines, which is that if you ask a question in a headline as you do, the answer is always no. So in theory, this could be a very quick interview. No, we can't trust Sam Altman, but there is actually so much here, much more than just kind of a profile, so to speak, of Sam Altman, you guys spent over a year reporting on this, conducting a hundred plus interviews, and it really—brings the receipts on a lot of information while putting it into a full narrative.

[Main Podcast]

And so there's a couple of things that I think have been under reported as news and might have legal ramifications. So I wanted to unpack those because it is law fair first, the general volume of lies or the lack of disclosure to the corporate board and the possibility, if not legal fraud, certainly fraud in the colloquial sense to the public. Second, I wanna talk about alignment and trust and safety issues if we have some time. And then third, the national security implications. Obviously that re—and the reveals that you bring up.

And so let's start with the fraud, not necessarily in a Theranos style. It's not like you have a product that doesn't exist here, right? But what you do have is a product that Altman has been putting out there with OpenAI that continuously makes promises on a level of safety and a commitment to safety as you expose over and over again in the piece that he doesn't make good on or backs away from. And there's comparisons in your piece to SBF or to Elizabeth Holmes.

So what is your takeaway and what you were hearing about potential kind of public accountability for the way Altman was saying one thing and doing another, both in terms of product safety and then also in terms of conversations with regulators?

Andrew Marantz: Yeah, I mean there, there are many layers of this and maybe I'll just sort of take a couple and then Ronan, you can take whichever ones I don't get to, but I think I should just say really plainly, we are not alleging in the piece or here that we like know that OpenAI or Sam Altman broke this or that law at this or that time, right.

The purpose—And I think the reason I bring that up is that this is sort of why this, a lot of this stuff has been in the realm of kind of, I mean. The accusations, as you say, of borderline fraud or potential fraud are many in the piece. And it's an interesting, it's an interesting mix of like, what you don't have is this sort of smoking gun thing where, you know the reason Altman was fired in 2023 is that he was caught red-handed, you know, embezzling funds or you know it, it's not that kind of allegation.

What you have is this kind of very slow buildup and accumulation of facts such that they're very hard to kind of put together in one place, which is why this took so long. And it is, I mean, since you point out the fact checking and since you have experience with it, we literally did have the best fact-checkers in the world on this, and we really did get it so bulletproof that it's kind of hard to contest.

But to the question of like which frauds rise to the top where all allegations of fraud, one of them is we talk to a lot of economic experts who talk about, you know, what they call circular deals in the industry. And this is not unique to OpenAI. This is kind of a lot of these, what are called hyperscalers, you know, there's really just a few of these companies building this incredibly, highly leveraged, risky product that requires massive, unprecedented amounts of capital expenditure in order to build the infrastructure to make the next model.

So even if you are extremely bullish on the economic potential of the product, you could still think this is a bubble. And in fact, Sam Altman at times says, this is a bubble. Because even if the product is, as you say, you know, it's not a figment of the imagination. It's not like a tulip craze. It could still be a cause of this frothy investment.

And, you know, you can get into the details of how when OpenAI and Nvidia strike these deals, for example, they're kind of both investing in future products that don't really exist yet and that depreciate very quickly. And so there are all these sort of microeconomic allegations of what people call borderline fraud or circular deals.

And then to your question about the safety stuff, you know. I think as a baseline, it's very easy for people to look at a tech company and say, oh, are we so naive that we're shocked by, you know, a tech company saying they won't do, you know, bad thing X and then they do it, right.

And I think this is a bit of a different case where, you know, the articles of incorporation under which OpenAI was set up as a nonprofit make very clear that they have a legally binding fiduciary duty to do certain things. And what we chart in the piece is this kind of years long process of the company kind of converting itself out from under those restrictions such that now they're one of the biggest for-profit companies in the world.

That is not something you can do without angering a lot of people, including people who you recruited and who took pay cuts to work at your company.

Ronan Farrow: I'd also point out on the sort of fraud question, and I want to be clear, you know, we're wearing our lawyer hats here and you know, I'm not going to talk about fraud as a matter of law, but when you talk about in a colloquial sense, things that feel materially deceptive and may have real legal ramifications of one kind or another, it's really important to note the way in which this company was behaving amidst this backdrop that Andrew just described, right of you have promises that don't always line up, with what's being delivered.

You have that on a scale that critics in this piece are saying is beyond even the baseline of Silicon Valley being founded on that kind of a hype model. You have, you know, material assurances to board members about real, concrete safety testing requirements that are just turning out to not be true, and then you have a super structure of handling of this that also raises question marks according to many people around its stakeholders and outside legal experts that we spoke to.

So one great example of this that hasn't been previously reported is that a part of the mechanics of Sam Altman coming back after board members fired him several years ago for this alleged pattern that we document of apparent, you know, deception and manipulation, is that he got rid of some of the board members that had moved against him, but as a condition of their exit, they demanded an outside review and he was reluctant.

We report that people at the time recall Altman saying, you know, I don't want any review. Any review just by its existence, regardless of its conclusions, could make me look guilty, an understandable concern. But ultimately, after a lot of tense negotiation, he acceded to that demand. The way that it went down was that he got rid of two key board members who had moved to fire him.

He, in close consultation with their major outside financial backer, which was Microsoft, basically selected two new board members, Larry Summers, which is the name we can talk about. But at the time, you know, he was designed to confer legitimacy 'cause he had been a cabinet member and he had been the president of Harvard. He's since stepped down because of his role in the Epstein files.

But you know, he was a big cheese who was brought in alongside Bret Taylor, who had been at Facebook and was also viewed as a steady hand. But, you know, we uncover how during a previous round of Bret being considered for that board, actually, there had been people who were concerned that he was too close to Altman to be independent.

So you have these two people that Altman selects, you know, pretty closely. He's in all the conversations, basically proposing these names. And then you have a process where they go to, you know, a very—legit law firm, WilmerHale, to do an outside investigation. And the interesting thing is Wilmer's legitimacy for these kinds of investigations, it comes from a lot of transparency in many of the prominent cases where they've done it. You know, Enron, WorldCom, these are cases where they publicly released voluminous details of investigations.

In this case, we report there were a ton of people around this investigation. I think we cite six people close to it, who really felt it was designed to limit transparency, and as one put it, to reach a foregone conclusion that they went in the perception of these critics. And of course, you know, we have the lawyers involved defending this. They say, no, it was comprehensive, it was independent, but there were a lot of people around this who felt that they were not interested in the core questions of integrity that were underlying the firing.

It seems they did deal with them because people wouldn't stop raising them. You know, I wanna be careful about this. It's not that they totally scoped it away from that, but ultimately it really seems like what outside investors who wanted Sam back were looking for was, did he sexually assault someone? Did he embezzle? Much more bright line things.

And with respect to the actual findings we actually have someone involved saying, well, you know, on the honesty and integrity question, the review, I'm only lightly paraphrasing here, did not find that Altman was a George Washington's cherry tree of integrity.

So, so you have that underlying content of a review. And then the really extraordinary thing in my view, which is yes, private companies sometimes keep these things out of writing. It's worth noting that there are plenty of legal analysts who say that's a red flag. It's a way to prevent liability and avoid scrutiny.

When you get to a case like this where it's a high-profile scandal that engulfed Silicon Valley and where all of the stakeholders around it, senior executives at this company, forget the public, expected to see findings. They, on the advice of those two new board members that have been selected in close consultation with Sam and their personal attorneys, you know, the Wilmer lawyers obviously were also in on this decision, but this particular private advice fed in, kept it out of writing. It was only oral briefings. And look, the piece is very sober about appraising all the arguments some of the lawyers make that this is fine and appropriate.

We actually have one of those new, new board members who oversaw it, Taylor saying on the record, there was no need for a written report. So they're acknowledging this. Now, there are a lot of people around this who understandably say that defeats the entire purpose it was supposed to achieve. They released only 800 words of essentially a press release on the website saying there had been a breakdown in trust. Nobody knew what the hell had happened.

So I hope this can bridge some of that divide of misunderstanding. And this brings us full circle to your question. Under Delaware corporate law, there's like Section 220 obligations for any nerds that are watching and know the mechanics of this. If this company IPOs, I think one reason why you can imagine the pushback and comment-seeking and fact-checking process on this point was very intense with the parties involved is there is real concern.

We have, you know, board members telling us, well, if there were these deficiencies, there may need to be another review. You have a law firm that does gigantic business on the basis of the legitimacy of these kinds of reports, and you have a legal system that entitles future shareholders to go back, as they often do in these kinds of scandals where things are insufficiently documented and demand the internal underlying records, and you know, judges have often ruled in favor of those demands. So that is something of a potential powder keg, I think, in the view of a lot of people around it.

Kate Klonick: I mean, I read this and that was exactly what I was thinking. I was like, this had to go through so many lawyers to be able to basically put this stuff into writing because for exactly the reason that you state, I mean, what Andrew talked about is exactly correct.

Some of this is Silicon Valley business as usual, you know, like lying about your ping pong score, like who doesn't do that? Right? But some of this is just, you know, is salesmanship. No one's saying that is the stuff that people are like, it is the actual rubber meets road of putting these things into a corporate structure, binding board members with fiduciary duties to those ideas, and then kind of backtracking and like reverse engineering the entire nature of your company and everything that investors kind of had bought in on, while simultaneously actually deceiving your board.

I mean, I was kind of baffled and thought that, I mean, and it was again, and this is a Lawfare podcast so we have the right audience for it. It was a nerdy kind of point, but like the idea that Wilmer did not release these details, the idea that they gave these oral briefings, like how bizarre, like why give briefings at all?

Ronan Farrow: And we really sort of stumbled into the full extent of this. Like we had Sam on the record saying, ‘Oh you know, I don't know about it being just an oral briefing to those two new board members that I helped pick, you know, I, it was I believe very strongly it was given to all of the subsequent board members who joined in the aftermath of this.’

I'm paraphrasing, but you know, that's, you can find his exact quote in the piece and then, you know, we looked into it and that's simply not true. And we have, you know, other people around us saying that is a lie. And really, it was just this extremely limited, deliberately undocumented briefing process over the course of it.

And, you know, Wilmer is also very careful about saying that their findings were not exonerative. You know, that their role was just, all they'll say is as summarized in that press release, which is based on whatever they found, those two board members Sam helped pick, decided it was appropriate to have him continue.

So it's all very cagey. It's very deliberately undocumented, and as you say it really in the view of many around this, defeats the purpose of bestowing the lit legitimacy that you would get from a proper investigation. I think it's a great parable about the need for transparency in these kinds of corporate situations, especially with these vast existential and safety stakes for all of us, right?

This isn't even normal business. This was a 501(c)(3). You know, it actually is a disservice to Sam Altman that the investigation was handled this way. Because that's how you wind up with a 16,000 word New Yorker investigative piece years later.

And there's a lot of legal authorities that talk about this very question in this very way, that if you keep things out of writing for short-term game, you know, and ostensibly it's to protect privilege and whatever, you know, to avoid as much liability as possible. But it has these longer term costs, that people can speculate that there are much worse things that they found.

So I again, hope that we've done something to A) restore a little bit of the transparency there about what the underlying complaints were, and B) send out this message loud and clear that there are these costs that companies and law firms doing this kind of work should reckon with, I think in a more acute way.

Kate Klonick: Andrew, do you have anything to add?

Andrew Marantz: You do have some instances of like, for example, Greg Brockman writing in a digital diary.

Like, I would like to make a billion dollars or something. So you do have a few people sort of writing down the thing that they—

Ronan Farrow: Yeah. And also Greg Brockman in a diary writing, you know, essentially like wondering whether it's a lie that they're saying that they're a nonprofit and then turning around and maybe becoming a B Corp, which they were planning to do very early on.

It's truly remarkable. And the fact that all of these guys keep such voluminous journals. This piece is so full of journaling.

Andrew Marantz: I know.

Kate Klonick: It's insane. I was re—I'm like, who is telling these people to keep a diary?

Andrew Marantz: I know.

Kate Klonick: Do they think that they're like kind of someone is going to like be finding their notes like Archimedes someday or something and like wrapping them into like that they're going to be part of this genius historical record.

Andrew Marantz: I mean, as you know, they absolutely do think that and—

Ronan Farrow: Yep.

Andrew Marantz: And that's why, you know, this is not, I mean obviously this is a piece about a person, you know, sort of being used as a lens onto these structural things. But I don't think anyone should walk away from this piece thinking like, oh, Sam Altman shouldn't be the AGI dictator.

Like Elon should, or Demis should, or Greg Brockman should. Like obviously I think the structural takeaway here is closer to like, who is vesting these dudes with this amount of power? How could that possibly end well? And yeah, as you say, like. A lot of people, I think, sometimes take the easy way out sort of mentally from this, people who are dismayed by the accelerating power of these technologies and they sort of say either it's a parlor trick or it's stochastic parrots, or like, this is all hype and an attempt at regulatory capture.

And I think among many things what that misses is, you can think that sort of sitting at home and watching from the sidelines, but the people involved in this do not think that. They think they are playing like a ring of sauron level competition and their actions back that up.

Kate Klonick: Yep. There is a huge piece of this and we'll kind of get to this kind of in a little bit.

Which is just that it's happening. Like it doesn't matter whether you think it's real or not, it is like there is enough money and enough buy-in that this is just becoming the reality that we're swimming in. And it doesn't matter if they're, they are like stochastic parrots there. If there's a trillion dollars invested in it, then we're all going to be run by stochastic parrots.

Andrew Marantz: And if the autonomous military murder drone is run by a stochastic parrot, like that's still a problem.

Kate Klonick: Right? And so it does hit me—And Andrew, I know that you have spent a lot of time kind of, you and I both have spent a lot of time and the speech platforms kind of looking at social media and it just feels, to a large extent, exactly the conversation we were having in like late teens, early twenties, which was like, and this feels like AI is just speed running these issues.

Like all of a sudden we're gonna have these incredible mega corps, these transnational companies that transcend state power, right? Running these things and there's no one to curb that. I think that we should just like kind of skip to the country plan, which is the idea that I guess Page Hedley, who has come in, was a former nonprofit public interest lawyer, and she was gonna be OpenAI's policy and ethics advisor, and she made a talk about, OpenAI could basically avert a catastrophic arms race by building a coalition of AI labs that would eventually coordinate with an international body, akin to NATO, to ensure that the technology was, you know, deployed safely.

And you have in like your piece that Brockman, Greg Brockman just like could not rock this. He could not understand why someone would engage in this type of like new governance in this type of multi-stakeholder-ism, in this type of like, do-the-right-thing-ism. He was like, where's the money? Like how do we make money in that?

And then begins this thing called like that you call the country plan, which is essentially like, why not pit selling this technology between China and Russia? Why not?

Andrew Marantz: What could go wrong? Yeah,

Kate Klonick: Yeah. What could possibly go wrong? And then you obviously, the rest of the piece is this like incredibly detailed conversation about how Sam's already in the process of doing that to like the, of selling the technology to the Gulf States.

And so, I don't know. I guess we'll kind of skip to the big question, which is like, is there any way to stop this if you have this kind of coalition of the willing between these giant moneyed countries and the rule of laws is falling around our feet and like at the very least it feels—I like, I, you know, you are hoping that the IPO gets canceled.

You're hoping, but like I have to say, I've finished your piece and I was like, in another world in another time, this would've canceled the IPO. And I'm like, does it even, is this gonna matter?

Ronan Farrow: This is for me the bigger point. You know, Sam Altman is one figure. He's a consequential one, but he's especially consequential as a window through which to view a wider dynamic that critics in the piece described as a race to the bottom.

And that's a kind of confluence, right, of what you described of legislative and regulatory guardrails on private industry and particularly big tech falling away more widely, an anti-regulation bent in American politics under the current regime, and a backdrop of money in politics that increasingly allows Silicon Valley to really have its hands on all of the levers of policymaking and power.

If you are running for office right now in the United States, you are contending with a flood of PAC money that is specifically focused on quashing AI regulation. Greg Brockman, Altman's second-in-command has contributed to one of these major Trump friendly anti-regulation PACs. So there is just a massive amount of machinery that is pushing away from safeguarding us from any recklessness, any lack of trustworthiness of the type that we write about in the piece.

And the tragedy of it is, these companies and individuals are the ones best positioned to clock what the risks are. They keep telling us the risks are grave, right? You know, autonomous weapons in war zones. You know, the potential for mass surveillance, the devastating misinformation potential of this technology in electoral contexts.

All of the kind of more individual mental health stakes where you have all of these wrongful death suits against OpenAI, alleging that ChatGPT contributed to suicides and a murder potentially. The list goes on and on, and yet, there is just very little framework to deal with any of it and the inflection point of the structures these companies themselves set up, right?

OpenAI being this safety focused nonprofit, having a board that could fire its CEO if they thought he was lying serially about material things. That all fell away when Sam Altman came back. Sam Altman is not the only one participating in these broader dynamics, but very often OpenAI has been a first mover.

And I think we both viewed the story as one that just tells the bigger story as well. And you know, we can talk about some of the kind of shortfalls in policymaking, steps in terms of regulation and legislation that get at a whole range of the issues that we've already raised. Some of them are being kind of trialed elsewhere in the world.

Some of them have come up in proposed state regulation, though that, too, faces an uphill battle. We talk about that a little bit in the piece, how there's also a lot of machinery to get rid of any state efforts, but I think, I was eager to have this conversation because that is the part of it that I feel is most gravely urgent, that we all need to marshal whatever resources we have as Americans to get at these broader shortfalls of accountability for companies that really do, as one person said of the Sam question in this piece, you know, it's people with their finger on the button in a very consequential way.

And, you know, forget the more acute example of Sam and his alleged issues with honesty. You know, you look at the, in the, in their own eyes anyway, good guys at Anthropic, right? They've got breaches right, left and center. They're also weakening their safety commitments.

So I think the point is we are placing all trust in unaccountable private sector actors, and I fear that we're not going to wake up to the consequences until something really devastating happens.

Andrew Marantz: Yeah. There are also major public sector actors. They just tend to be like shake to noon on his yacht in the UAE.

So I should say like two things on this. Just to piggyback on what you're saying, Ronan, like, you know it, it's often sort of painted as naive or kind of, you know, like why are these sci-fi nerds even talking about all of these risks and like for one thing, these mega dystopian sci-fi risks are being discussed because people like Sam Altman told us to be afraid of them. Right?

So this is not something that Ronan and I invented one morning to scare readers about, and we're very kind of, I think, sober about it in the piece. The people who founded these companies told us to be afraid of this. And moreover, like again, there are people who think that's just a move to try to attempt regulatory capture or to attempt to, you know, gain attention for the next round of investment. And there may well be some of that.

But you know, you bring up people like Page Hedley, these are not all competitors who work for Anthropic or who work for Google or who are trying to scope the next round of investment funding. I don't know, maybe Page will, you know, have an IPO before the end of the year and I'll be, you know, proven, not cynical enough, but like there really were many people who we talked to in the a hundred something people we spoke to who were not, you know, were very careful to not be just laundering competitor gossip through our piece.

Like there are a lot of people who really are concerned whistleblowers now their fears may be proved right or wrong, and unfortunately we're gonna live to see whether they're right or wrong, but it's not just all competitor hype and you know, the normal stuff of capitalism.

Kate Klonick: Yeah, so I wanted to say that's another thing that I absolutely value about this piece is that it is soup to nuts.

It is, you talk to people on the factory floor of these companies and you talk to people—I have a list actually, of just like all of the different ranges of perspectives that you got in this piece, and one of the things that I think would have been a disservice and made this piece less valuable is if you had just focused on the Game of Thrones aspects of like these kind of companies, the C-Suite, the people who are trying to be the narcissists and to like run the world and who are writing their diaries so that one day people will study them and talk of their genius.

But one of my really big takeaways from my time in Silicon Valley and my time inside companies is that the majority of people, like outside the top 100 people in a company, are there because they think that their job matters and that they're trying to do the right thing. You talk about how in late 2022, 4 computer scientists published a paper motivated in part by concerns about quote unquote deceptive alignment in which sufficiently advanced models might pretend to be behave well during testing, and then once deployed, pursue their own goals.

That this moment actually compels Altman to create like a super alignment project where he like purports to care about safety and like gonna set up this huge team and recruits on it, and then of course it just falls away. It just goes away.

Ronan Farrow: And he, and the announcement actually says specifically like, we need to create this to prevent the extinction of humanity.

Kate Klonick: Right?

Ronan Farrow: Yeah. So, so again, to Andrew’s point, like it's not us introducing those stakes. I think that's a great example. There are many others in the piece. I was struck by the thing you just said about the gap between the kind of C-suite leadership and their sort of writing down their own history for the books and then, you know, the working level people where I share your impression.

You know, again and again, Andrew and I were encountering people who really came to this company, this nonprofit at the time that they joined for the early ones, because they believed that there was a need for the thing that Sam Altman was pitching, which was something that was not just governed by profit and growth, and that would stand really defiantly against that, that had a corporate structure that was uniquely designed to prioritize safety.

And very often those people at a working level are much less heard. And you know, that's one of the most important equities in this reporting to carry the concerns of these researchers, which continue to this day. And, you know, some of the lower-profile, safety-minded people who had concerns, they actually, you know, raised them or attempted to raise them within the corporate structures in formal ways.

We have a lot of emails complaining to people from safety leads saying, you know, the company is going off the rails on its mission. You know, essentially sort of whistleblower material that we surface. And we even talk about, there's a case where in terms of the corporate transformations into a more for-profit mode there's a moment where there's a board vote on that conversion.

And there's a board member who appears to vote against it, you know, according to the materials and sources that we have, and argued that the way this was being done was not appropriate, that the nonprofit was being severely undervalued and said, I can't do this in good faith.

Kate Klonick: And then it's memory hold—

Ronan Farrow: Right. An attorney or rewritten says point blank. If we record this dissent, it might be a flag to investigate further the legitimacy of this new structure. And the vote then gets recorded as an abstention apparently without this board member's consent. Now there's a disputed factual record here. We carry OpenAI's pushback.

They say several employees recall it being a deliberate abstention, but I found that to be a fascinating case. You know, we don't, as Andrew rightly pointed out, come to legal conclusions here. You know, I'm a lawyer, but I'm not your lawyer, OpenAI. And I'm not going to come to conclusions on it, but I think it's worth raising because it has meaningful legal ramifications.

Whatever the conclusion might be, it is certainly something that raises questions about potential falsification of business record charges under state law. You know, that would be a misdemeanor in most jurisdictions, but more practically, the lawyer for the board made exactly the right point, which is that vote was consequential.

It authorized a corporate restructuring worth hundreds of billions of dollars, and a tainted vote could ultimately create grounds to challenge the legitimacy of that conversion, which is really relevant to pending nonprofit litigation, the Musk suit, and to IPO due diligence. So all of these cases where you know, to use your frame kind of very often, the little guy, the less profiled people sound alarms in one way or another.

And then to use your term again, sometimes it does seem like there's some memory holding. It just, I don't think the math on it is smart, even from a pragmatic standpoint for the people who are trying to make the complaints go away, I think it creates more problems.

And I really do want to talk before we part ways about the failures of structures around this company and the regulatory and legislative failures that just, they don't create any recourse.

Like to use some of these examples, there should be a more robust, federal AI whistleblower protection legislation. You know it, it's crazy at this point when you have these companies themselves saying, this is the most dangerous technology ever to not have the kind of regulatory regimen that we have for, you know, pharmaceuticals or for food even, you know, and for whistleblower protection associated with that, there should be something modeled on Sarbanes-Oxley for AI whistleblowers.

You know, people were very fearful to talk to us for this story. And people who are sounding alarms about this technology are doing a public service and they deserve to be protected. You know, that's, I think just one of the many regulatory steps that would make, or legislative, you know, you could come at it in multiple ways, that could really make a difference.

Andrew Marantz: Yeah. I mean, just backing onto that, like Kate, you were talking about the ways in which this is continuous or discontinuous with the trust and safety debates around social media. I think it's interesting, you know, there's this one moment in the piece where I think to some people it maybe seemed like we were doing a gotcha thing where we were saying, okay, let's talk to existential safety researchers at the company. And a representative from the company is like, what's that? That's not a thing.

And first of all I think just to be clear, like we absolutely did not do that as a gotcha thing. Like we, we went in saying, please, let us talk to the research teams that you have working on the original ostensible, raison d’être of this organization.

And the reason that we talked about existential safety versus, you know, trust and safety is that the word does mean different things, and there are legitimately different people who work on different aspects of safety. Some of it is anti-doxing. Some of it is privacy stuff. Some of it is making sure the models don't, you know, turn into mega-Hitler and start spewing racist stuff at you. And some of it is this stuff that is in the more hypothetical sci-fi realm of making sure they're aligned and don't kill us.

All that, to your point, Kate the deceptive alignment research, we kind of just very quickly say in the piece, this all sounds crazy, except that under certain experimental conditions it's already happening and like. Again, best fact-checkers in the business, like no one disputes that as a factual claim. These things are starting to happen under certain experimental conditions.

Now we don't know what that means. We don't know how fast this stuff is gonna accelerate, but as Ronan is saying, the fact that the people closest to this technology are trying to make everybody wake up and start to panic a little bit, and we have a political system that seems incapable of that is a, a deep structural problem that—

I mean, we just document in the piece like a California bill comes up and as we report in the piece, an investor, Ron Conway, who's close to OpenAI, calls up Nancy Pelosi, calls up Gavin Newsom, the bill gets killed, right? So this is the current system. It's not shocking to anyone that's the system we live under, but it is unprecedented the scale of the acceleration.

Kate Klonick: No, and I think that we should switch to kind of end—talking about this, because I do think that we should have a conversation about what next does the law have any type of, you know, recourse? Can we push back on these things? I mean, after reading your piece I am incredibly bullish on, and I was before, but like, on kind of regulatory potential legislative potential, doing something to curb some of this, and to make it so these companies are accountable to anyone besides their kind of megalomaniac CEOs and like.

I just don't understand how that's possible in this current government. It seems like a perfect storm of the closeness and the coziness between kind of this current administration and these companies—

Andrew Marantz: Well, and in fact when there was almost a moratorium on all state regulation for 10 years. This isn't even in the piece, 'cause that we had so much to get to, but it seems like Steve Bannon and Mike Davis and a lot of MAGA-ish people were standing in the way of that.

So it's a very strange bedfellows kind of situation.

Kate Klonick: Yeah, I mean, I think that's completely right. And then like, you know, I watched in Europe just from like afar, like people try to kind of create the AI Act and Brussels and everything and create real kind of strictures. I mean, the other thing is that the terminology that is being used in this is so vague and so untethered to like meaning, technical meaning, or, you know, or even non-technical meaning—

To be totally frank, guardrails, if I had a nickel for every time some asshole said guardrails to me, like, we should just have guardrails. Like, okay, well that's really nice. Like on what, for what does that mean?

Ronan Farrow: Well, let's talk about what guardrails could mean, by the way.

Yeah, because—

Kate Klonick: Let's talk about it.

Ronan Farrow: The U.K. had an AI safety institute long before we've tried anything of this type, the U.S. tried to do this under Biden to basically have like some form of mandatory pre-deployment safety testing that's actually meaningful.

Andrew Marantz: And Sam Altman was all for it at the time, but that seemed to change.

Ronan Farrow: Right? And part of the pattern with Sam, as Andrew's pointing out is that we document how he was, the regulators’ “regulate us” guy and he proposed a federal agency to AI. And he talked about wanting this, these kinds of testing requirements. And you know, then simultaneously was very often working against these very kinds of provision in regulation.

To your point about what is actually possible, you know, even thinkable in the current environment, I mean the flood of money from AI into politics is enabled by this post-Citizens United landscape where you can have Greg Brockman putting $50 million into the leading the future PAC, and he and his wife putting 25 million into MAGA, Inc. And that's just free speech now.

And I think we are learning that the world we're living in where this is possible and where the flood of money into politics really makes it unthinkable to meaningfully push back. That has real deleterious consequences for all of us. That is endangering all of us.

So, that to me is why it's worth us putting a year and a half of our lives into pieces like this. I had just come off of when Andrew and I started this a year and a half ago, a body of reporting about Elon Musk, which published at a time when people were not raising the red flags that they now openly raise about him.

And the reason there was also not that I was, you know, so fascinated by this one man, it was that Elon Musk is a really leading example, sadly, of the ways in which these tycoons in Silicon Valley have developed super governmental authority. And that story was about someone who had, you know, real geopolitical ramifications where he was the sole vendor in a lot of key areas to the defense sector, and where basically, you know, senior officials had to scurry around doing his bidding and he could add a whim in that case, turn off communication access through Starlink in the battlefield in Ukraine on the front lines and people's lives were in the balance and so that's the bigger story. And it's what we see in AI too.

Andrew Marantz: I mean, one way to kind of connect these threads, right, is like exactly like you're saying about this kind of super governmental power.

When you go back to these early days of people sitting in a conference room with a whiteboard saying, what if we were to auction off AGI to China and Russia, or you know, what if we were to, you know, grasp the ring of sauron and become an AGI dictator, right? Or what if this, you know, wakes up one day and kills us all?

I, in some sense, like the best-case scenario is that was all hype and bullshit, and it was just trying to get investors' attention. Anything other than that is kind of the worst-case scenario, right? What if there's some even slim chance that something like that is true? And so I think to Ronan's point, it's like, yeah, it would be one thing if it was just someone like Elon or Sam Altman kind of having these sort of like self-aggrandizing conversations about, one day, I'm gonna own all these satellites that will be useful in war zones or one day, I will build a technology that the Pentagon can use to maybe conduct mass surveillance. Or maybe one day I'll build a technology that can be interwoven into markets and transportation infrastructure and hospitals. And but like the scarier thing is what if some portion of that comes to pass? And obviously that's the timeline we're currently living in.

Kate Klonick: Yeah. So I'm gonna propose one thing as long as we're on the topic of kind of what we can possibly do and who will save us from this, which is, I mean, it's a little idealistic, but the courts. And I, you know, I would point exactly to kind of, they are less viable, or at least were, than, you know, the other kind of branches of government and these other types of structures.

I think one of the things that really struck me, and you know, you don't, you know, you just said before, like that Anthropic has its problems as well. Like it's not as if it's like a perfect company, but you do have, and you end the piece with, and it must have kind of been one of the provocations to finally get this out the door.

And a perfect coda to end on is the supply chain risk designation that the administration let like came down on with Anthropic. I have to say that reading everything that you wrote, I am just like, how on earth is Anthropic the problem? Like, I mean like how could they possibly pivot to OpenAI as a substitute?

You go through everything from like Altman failing to get a clearance, right, to all of his dealings with the UAE with Saudi Arabia. You know, these back and forths and promises of safety. And then not making good on them. And then you have like these kinds of conversations about, you know, geopolitical maneuverings, launching an unchecked safetym OpenAI products in India.

You know, just various types of things. Are you following kind of closely the Anthropic case in the northern district of California? Are you following closely the case in like that? We just got kind of, Anthropic lost at least initially with the,

Ronan Farrow: In the D.C Circuit.

Kate Klonick: In the DC circuit. So like, what are you thinking? Do you think that there is hope with the courts on any of this?

Ronan Farrow: Well, you know, it was striking to see after the Pentagon standoff that we narrated in the piece that the initial ruling from a California judge is this Judge Rita Lin, who wrote actually a really powerful decision and talked about the designation being classic illegal First Amendment retaliation and called it orwellian and Anthropic was restored to federal procurement schedules.

The latest ruling, you know, I would not overstate its significance just yet. What it does is create kind of a mess. It's a procedural ruling. I mean, basically the argument that the circuit came up with was it's primarily a financial standing that Anthropic has, and that the equitable balance is in favor of the government's interests on this.

So it just. The D.C. Circuit didn't say the blacklisting was legal. They just said Anthropic didn't meet the very high bar for emergency relief while the case plays out. So the merits are still live. There's an evidentiary hearing on the 19th. It'll be interesting to see. I think you're exactly right in the big picture question.

The courts can play a pivotal role at this time when, for all the reasons we just discussed, the legislative branch is so disempowered. You know, the courts also have, in some ways, been denuded of a lot of the legal thinkers that would be into holding private sector actors accountable. There are real problems with the judiciary right now as a source of accountability for a Silicon Valley run amok, but.

I do think you're right to identify that as one of the core areas of hope. 'cause you do see rulings like the one I just described where someone said, you know, no, this is irregular and wrong and violates the First Amendment.

Andrew Marantz: Yeah. And I think just to add to that, like all the conversations we're having about guardrails and regulation, right?

I think all of us would not want to give short shrift to how complex, all of these discussions are, and how there are good faith arguments. I mean, you heard a lot of arguments around when this supply chain risk designation was happening. Like, okay, for all you pro-regulation people, is this the government you want regulating this technology in this way?

Like so there are good faith arguments that need to be worked out by people who are approaching these things from first principles. The problem is when it's all drowned out by industry fueled, you know, muddled, sort of propagandistic talking points. You can't have that good faith conversation.

Kate Klonick: Yeah. Well, and I guess we can end on the fact that the muddled-ness, this is an incredible piece of journalism.

It was a tremendous amount of work at just full of, you know, just such information that the public needs, frankly, before you have a public company that is valued at a trillion dollars, which is more than the GDP of many countries in Europe, for example.

And it's just it's just, it's amazing that like, you know, that all of this wasn't available from like, everything from the Wilmer report to, you know, knowledge about kind of some of how these alignment goals have been abandoned and the safety goals have been abandoned to like how this is getting released in other countries to the country plan, kind of the nat sec and geopolitical maneuverings. I think that this is going to be a piece that has incredible effect for a very long time.

So,

Ronan Farrow: And I just wanna say also in terms of the impact, I really appreciate the policy discussion, which very few people take the time and space for. And I hope anyone listening to this, I mean, this is a specialty audience of people who are smart enough to care, but they all have friends in the wider public too, where they can explain why this matters.

So I hope people start a meaningful conversation about this because the stakes really are that vast, and I hope it extends to how we engage with our representatives. You know, you talked about the hope that the judiciary will play a role in creating, to use the term you love so much, guardrails. But also I haven't given up hope about the legislative branch.

You know, you see politicians already across different parts of the government clocking that it can be advantageous to play a role in creating accountability. Florida today just announced, their AG’s office just announced new investigation into OpenAI around some of this stuff we've talked about today.

So I think that people are looking at the numbers in Washington as well, and there was polling last month from I think NBC showing that a majority of the public now views the costs of AI and the risks as outweighing the benefits. So I do think we're all in this moment where we're kind of mesmerized by our screens and in the thrall of these companies.

But if Americans join arms and really tell their representatives, we want you to do your job and make sure that we're kept safe from this technology that could eliminate our jobs and threaten our safety and all the other things we talked about—I do think the simple math of Washington can still lead to meaningful oversight and results if we really, really press for that.

Kate Klonick: We have the midterms coming up. I mean, like when we've talked about how this is gonna affect the IPOs, but like, I absolutely think this piece could play an AI existential risk and like fear, basic consumer and citizen fear of some of these technologies could absolutely play a role in and how people vote.

Ronan Farrow: Before you vote, find out whe whether your representative is getting AI money and, you know, do scrutinize their policy positions related to that.

Kate Klonick: Yeah. Well guys, thank you both for coming on. Andrew, Ronan, it was wonderful to have you.

Andrew Marantz: Thank you, Kate. This is awesome.

Ronan Farrow: Thanks, Kate. Great to be here.

[Outro]

Kate Klonick: The Lawfare Podcast is produced by the Lawfare Institute. If you want to support the show and listen ad-free, you can become a Lawfare material supporter at lawfaremedia.org/support. Supporters also get access to special events and other bonus content we don't share anywhere else. If you enjoyed the podcast, please rate and review us wherever you listen. It really does help.

And be sure to check out our other shows: Scaling Laws, Rational Security, Allies, The Aftermath, and Escalation, our latest Lawfare Presents podcast series about the war in Ukraine. You can also find all of our written work at lawfaremedia.org. The podcast is edited by Jen Patja with audio engineering by Cara Shillenn of Goat Rodeo. Our theme song is from ALIBI Music.

And as always, thanks for listening.