Lawfare Daily: ‘The Russian General Staff’ with Oscar Jonsson, November 12

[Intro]

Roberto Tallarita: It’s very, very difficult and probably impossible to even come up with some criteria on how to assess whether the balancing is happening or not.

Kevin Frazier: It’s the Lawfare Podcast. I’m Kevin Frazier, director of the AI Innovation and Law Program at the University of Texas School of Law, and a senior editor at Lawfare, joined by Roberto Tallarita, assistant professor at Harvard Law School.

Roberto Tallarita: The kill switch is in the hands of people who have a lot of wealth at stake, and the big question is will they care about humanity?

Kevin Frazier: Today we’re exploring what Anthropic’s impending IPO and novel corporate governance experiment means for AI regulation.

[Main podcast]

So, Anthropic is competing to develop some highly capable AI systems, and its founders have warned that those systems could cause quite serious harm, and they spelled those harms out in their recent S1 filing ahead of their IPO.

Those harms include the potential damage to people around the world, as well as to financial systems. And yet, Anthropic’s stated mission is to develop advanced AI systems responsibly for the long-term benefit of humanity. This leads to some obvious tensions, recognized again by Anthropic’s founders themselves.

But I wanna come to some terms that you spelled out with this novel experiment that’s underway from Anthropic with respect to corporate governance. So, you pointed out way back in 2024 that these AI companies are experimenting with various forms of non-traditional corporate governance. And you noted that the boards that they’re relying on, the different structures they’re imposing, all boil down to some questions about the extent to which the individuals in those positions will actually adhere to those missions, and that’s something that’s always going to be quite variable and quite uncertain.

And those experiments are coming to the fore in the near future as Anthropic IPOs. And so, I wanna walk through, Roberto, why you think the corporate governance that Anthropic is experimenting with may or may not live up to its role in making sure the company adheres to its safety mission, and what safeguards you would impose if you were in their shoes instead.

But to level set for everyone, what does traditional corporate governance look like when a company goes public?

Roberto Tallarita: So, the way a normal, let’s call it normal corporation or business corporation operates, makes decision, is that it has a board of directors. They are appointed by the shareholders the investors who actually put their money in the company.

And directors supervise and monitor what’s going on in the company. Their most important job is probably to hire a CEO and to make sure that she does her job properly. And from time to time, the board also is called to make important strategic decisions. Now, the relationship between the board and the shareholders is at the core of what a good corporate governance model looks like.

The board, the directors are fiduciaries of the corporation, and there’s been a century-long debate of what this means in practice, whether directors should care only about the stock price and shareholder value, or if they must care about social goals and social responsibility. But at the end of the day the reality in the real world is that directors are fiduciary for the corporations for the benefit of the shareholders.

This means that if shareholders are disappointed, if shareholders don’t think that management is doing a good job and directors are doing a good job monitoring management they can fire and replace the board. Now, this doesn’t happen very often because shareholders are fragmented, dispersed, and every, you know, a single individual shareholder doesn’t have enough at stake to care about what’s going on in a corporation.

Suppose you have some stocks in a company it’s probably... it’s probably not worth your time, energy, and resources to even, like, read the materials you receive for vote… for voting at a shareholder meeting. It doesn’t make sense for you economically to pay attention to what’s going on.

But there are mechanism out there. There’s a market for corporate control. There are shareholder activists. There are hedge funds who have the resources and the incentives to pay attention to what’s going on, and they can step in, buy stock, and agitate for change, including firing a board.

So, this is what normal corporate governance looks like. The structure is designed in a way that the board is responsive to shareholder and shareholders’ financial interests, and there are mechanisms out there to discipline and punish the board whenever they deviate too much or… from these objective of caring about shareholders’ financial interests.

This is what normal corporate governance looks like in a big public company.

Kevin Frazier: Okay, so let’s walk through a couple of the innovations that Anthropic has attempted to impose both pre-IPO and post-IPO to try to make sure that the financial imperatives of building AGI, of making AI go well aren’t overriding.

So, let’s start with the idea of a public benefit corporation. What the heck is a PBC, and how does that differ from the traditional corporation that people may know of when they hear, again, your traditional retelling of what corporate governance looks like?

Roberto Tallarita: So, a public benefit corporation is essentially a normal corporation, but for this concept of balancing a financial goal with a social public objective.

So, as I said before, like the typical standard goal of a business corporation is to make money. This won’t be surprising for any listeners. And making money means making money for the shareholders, who are the investors who actually make money when the company makes money. But the public benefit corporation put, can put in its charter a public benefit clause that says, well, directors, our directors won’t care just about profits, but they have an explicit mandate, an explicit obligation to balance profits with this public benefit, which each public benefit corporation must spell out in its charter.

So, the idea behind this kind of particular corporate form which has been around for a few years now is that some corporations, like the business model of some corporations, tries to blend, in a sense, traditional profit making and some kind of social on public or public mission.

And this is precisely what Anthropic decided to become years ago before the IPO. Anthropic, it’s a public benefit corporation, and the basic idea behind this choice is that they have a social mission. They have a public goal they care about. It’s not just about selling models, making money and creating good financial return for shareholders, but their identity, in a sense, that’s what they tell us, is also some very important social mission, which is to build a safe AI for the benefit of humanity.

This is not something that is 100% at all times compatible or consistent with the profit-making mission. That’s why they chose a specific corporate form that allows them to do this, allows directors to be aligned, in a sense, not just with financial goals, but also with this social mission.

Kevin Frazier: So, we’ve got multiple forms of alignment going on in our AI conversations.

And in this case, the idea is, as you pointed out, the board can consider both the financial incentives, adherence to the mission, and the impact on the broader set of stakeholders. Here, not referring to shareholders, but stakeholders, and who may be impacted by Anthropic’s activities. And so, we’ve seen, for example, Dario Amodei, the CEO and one of the co-founders of Anthropic, took to X recently and detailed all of the ways in which he regarded Anthropic had taken steps that may not have been in the company’s financial interests, but may have been furthering its mission.

So, Roberto, for the board that’s going to form for Anthropic post-IPO, how do we assess whether they’re actually weighing those three factors adequately? If I’m a shareholder and I wanna challenge, hey, it looks like you’re not actually adhering to this mission as specified by the Public Benefit Corporation statute in Delaware, what recourse do I have to be able to say, hey, board, you know what? You’re actually not adhering to the law?

Roberto Tallarita: So, it’s very difficult. I’ll say it right away, right? It’s a nice aspiration, and it’s certainly part of what Anthropic, I think, genuinely believes they’re doing. But it’s very difficult for someone, whether a shareholder or an outsider in general to even get a sense of whether this balancing project is working or not.

So first of all, as you said, there are three different goals here two of which are somehow overlapping. And they say in the proposed charter, we are required to balance, one, the pecuniary interests of our stockholders, which is the traditional profit-making goal. Two, the best interests of those materially affected by our conduct.

This would be employees, customers, communities… local communities where they build plants or maybe data centers or whatever. And three, the company purpose, which is this broader kind of social mission of building safe AI for the benefit of humanity. Well, so first of all, it doesn’t tell us what balancing means.

Even assuming that we know we can actually investigate and measure in some way whether a specific decision is good for humanity or not, it doesn’t say whether this is 33% each or maybe 80/10/10 or whatever else kind of balance. So, the... It must be a balance, like meaning if the directors decide, well, from now on, we don’t care at all about one of these things, and they say it out loud, and they start bragging about it, well, that might be a problem, right?

But short of that, any kinds of hypothesis about how one specific decision affects these three factors is potentially fine, unless we put like zero on one of some factors, which is a very different estimate to do even theoretically let alone in court. So, this is the biggest, and we could just...

I mean, this is why… this is probably the biggest problem with this kind of structures, public benefit corporations. It’s it looks nice on paper, but it’s very difficult and probably impossible to even come up with some criteria on how to assess whether the balancing is happening or not.

And any kind of balancing is fine. So, these two problems undermine greatly what this kind of structure is supposed to accomplish.

Kevin Frazier: Yeah. And it’s fascinating also to note that under Delaware law, the shareholders who are actually able to even bring a suit with respect to potential divergence from the PBC, you’ve got to be a pretty massive shareholder.

This isn’t gonna be your average Joe or Jane who has one or two shares of Anthropic. You have to have at least a 2% holding of the company, which if the IPO is going to be as large as people expect, that’s not nothing, right? That’s gonna be a lot of money. And even if you brought that suit, as you raised, Roberto, how a court would interpret the proper balancing is up in the air.

Now, interesting, I will say as an Oregonian, I feel compelled to point out that in Oregon, their PBC statute allows companies to designate third parties who can hold them accountable to adherence to that mission. So, you know, to the extent Dario or any of the co-founders at Anthropic are listening, if you wanna refile or recharter in Oregon and designate a third party to hold you accountable, that could be another layer of governance to add here.

But Roberto, we’ve covered PBCs as one potential safeguard for adhering to Anthropic’s mission post-IPO. The other layer that I wanna get to next is that the board itself is comprised in a very interesting fashion. So, we know we’re going to have seven board members. Can you walk through how each of them is going to be appointed?

And then we’ll get into the weeds of the Long-Term Benefit Trust.

Roberto Tallarita: So first of all I wanna add something about shareholders trying to enforce the PBC thing. And you’re totally right that this balancing obligation can be enforced, first of all, only by shareholders. And so, it’s not like even if the benefit is towards humanity or the employees or customers, they have no standing in enforcing this kind of obligation.

And shareholders must be some kind of not super small shareholder. So, there’s a 2% threshold that you mentioned. An alternative way to get there is they have to own, like, 2 million, I guess, dollars worth of stock, which is, like, more manageable, but still it’s a bar that they have to clear. But again, there’s a question there before getting to the composition of the board why shareholders would even care about even if you find, and you might find like some plaintiff lawyers are— they’re very entrepreneurial, very inventive.

They might find some maybe socially oriented asset manager that has a few millions in the company that actually want to enforce the balancing clause alleging that maybe the board actually cares too much about money and too little about humanity. It’s gonna be, as we said very hard to demonstrate any of this.

And it’s also in terms of case law and precedents, a very novel kind of area of law. And the very few cases that have been out there, unsurprisingly, were about shareholders complaining that the boards cared too little about them and about profit-making. So, weren’t really cases about enforcing the social purpose.

Which is not surprising. That’s what shareholders are there for. They’re trying to make some money with this investment.

Kevin Frazier: But that’s just such a great thing to point out, though, about this design of PBCs to begin with. The only people who can enforce it are most interested in one prong of the balancing

Roberto Tallarita: Exactly. Exactly. That’s a big weakness, right? But I mean, it’s understanding from a governance standpoint. It would be a total mess if anyone could have standing to sue the company because they’re not doing the things that are good for humanity. It would be completely unmanageable.

But this is what it is, and these are the limits structural limits of this kind of form. Now, to your question about the board, this is the most interesting and creative aspect of Anthropic’s corporate governance structure. And I must say, from the very beginning, I will you know, most of the conversation I will say— I said, I will say critical things about the structure and say, look, this has this weakness and you know, people, a beneficiary won’t have standing, and it’s gonna be very difficult to enforce, and this and that, which I totally stand by.

But at the same time, I can see, I can perceive here by reading the charter, by going through the filing, that they gave a lot of thought to this problem. There— it’s not a marketing ploy as many people believe that PBC or this kind of social purposes sometimes are. And I also took a look, and I did some textual analysis with the help of AI, of course, going through some of the IPO filings, the registration statements of other public benefit corporations.

There have been a few that have gone public in the last several years, and Anthropic is an outlier in terms of both the creativity of their structure, but also the space they devote and in the prospectus to the potential trade-offs between profits and their social mission. They’re saying it there, right?

They’re keep hammering the point that, look, we care about social mission too, and this might mean that you will lose money. And they spend many words, way more than any other public benefit corporation has done before in trying to convey this message pretty clearly. And lots of people think, I mean, this is marketing, but I’m not sure.

I mean, I have no insider information into the company, but I’ve you know, reading the effort here and talking to people who have been closer to this effort I’m pretty confident that they are sincerely trying to pull off this very difficult balance between purpose and profit.

But the, the reality is that it’s very hard, and I’m not very reassured. Despite the, an A+ in effort, I’m not reassured that this is gonna work. If we are really worried, as they are, that this is this technology is the biggest threat we are facing as a society right now, I’m not reassured that corporate governance will take care of this risk.

Kevin Frazier: So, we know that there are certainly going to be shortcomings with a sort of corporate governance alone as the mechanism to avoid some of those worst-case outcomes. And as you noted previously, boards traditionally have been thought of as having a hyper-focus on the financial interests. We mitigate that, or Anthropic attempts to mitigate that, with the PBC structure.

We also know that there’s this Long-Term Benefit Trust, which is a formal purpose trust under Delaware law, that is somewhat exclusively focused on perhaps those other two prongs of the balancing, right? They’re not focused so much on the financial interests of the corporation, but really making ensure that there’s adherence to the mission.

Assuming you agree with me on that, how does that transfer into and inform the company’s corporate governance post-IPO? How are they going to empower this trust of five individuals to actually shape the direction of the company?

Roberto Tallarita: So, this is the most interesting and creative part of Anthropic’s governance structure. But let me immediately say something that will, it’s kind of surprising and the long benefit trust is not 100% focused on safety. Even the trust has a triple mission, as it were. So, it has to balance profit, stakeholders, and the big mission of building safe AI for the benefit of humanity.

Which is interesting, and we can talk about why they decided to do it this way rather than have a trust that is lesser focused on AI safety. But the fact is that the trust itself has a triple mission as well. It’s precisely a mirror of the mission of the board. So, the way this works is that, and I think the idea is here is the corporate planners here, the Anthropic advisors were 100% aware that public benefit corporation in itself, I mean, it’s a nice tool, but it’s not that effective, right?

That it would have been that credible in reassuring people that the social mission was at the center of the governance. So, they came up with this creative idea of giving this trust, which is a different entity the power to appoint a majority of the board. Now, there’s a wrinkle there, and I’ll go back to this.

But the basic idea is that there are seven directors, and typically in a normal corporation, all seven directors are appointed by the shareholders, like regular investors. This is a multi-class company, which is pretty usual in Silicon Valley. So, the founders will have actual control, voting control, so they will have a majority of the votes, even if they will not have a majority of the shares.

Which is a, it’s not normal corporate governance, but it has become a new normal for tech companies in Silicon Valley. But-

Kevin Frazier: And just to pause there for one second, for the folks who don’t read S1s for a living, right? This is the dual class structure, or I guess in this case, multi-class structure.

Whereas normally we would have one class of shares, every share would get one vote, and this is sort of the traditional, and then all of the shareholders can keep the board accountable in equal part, so on and so forth. But to your point, we have this multi-class structure for Anthropic with varying degrees of voting power.

Some classes have no voting power, some have only obligations with respect to the board, and there’s this whole smorgasbord of activity going on here. And so, what is it about that choice for a multi-class structure that informs the board’s configuration?

Roberto Tallarita: So, the, you have all kinds of different shares here.

It’s a very unique structure. You have A shares and T shares and E shares and N shares, all kind of alphabet letters.

Kevin Frazier: And let’s just to make sure that folks feel somewhat able to keep pace with this, because listeners, I gotta be honest, this is so complex, I highly recommend you take this transcript, put it into ChatGPT or Claude, and ask for a diagram.

But we’ve got so many things. Just to lay the land out quickly, we’ve got our Class A shares. So, there are traditional Class A shares that you and I will attempt to purchase or your broker will purchase. We’ve got Class E shares, which are going to be allocated to employees at a future date in 2027, and I’m sure we’ll talk about the powers afforded to the Class E shares.

Then we have Class F shares, which is a single share that will be dictated by a LLC, the founder LLC, which is made up of the seven founders, and that single share will have 50.1% of the voting power. So, when there is a shareholder vote, 50.1% of those votes will be afforded to this single Class F share.

And for folks who are reading between the lines, that means that Class A, you and me, the average Joe shareholders, cannot outvote the Class F shares, the vote cast by a majority of the founders who make up the founder LLC. One more Class to get to. There is then Class T. Class T is assigned to the Long-Term Benefit Trust.

They don’t have voting power, but as we’re getting into the weeds of, they do have a substantial say over the majority of the board seats. Okay, Roberto, did I lay that out okay? Anything to correct here?

Roberto Tallarita: Yeah. That’s great. A plus-

Kevin Frazier: Okay. Oh, thank you ...

Roberto Tallarita: on corporate-

Kevin Frazier: Thank you ...

Roberto Tallarita: governance.

Kevin Frazier: I only took one class at Harvard Law during my years at the Kennedy School, and I think I earned an H.

So, I’ll take an A plus. That’s an improvement.

Roberto Tallarita: That’s excellent. Excellent summary. And what we really care about for our topic of mission and purpose is not really that the founders have disproportionate voting power. I mean, this is not the normal corporate governance that I described at the beginning of our conversation, but it has become a new normal in Silicon Valley among tech companies.

Founders like Zuckerberg in Meta or the Google founders in Alphabet, and many, many other tech companies typically have super voting shares that allow them to have a majority of the votes or at least a disproportionate voting power compared to the fraction of the company they truly own, the number of shares they have.

So, this is, like, kind of normal for tech companies. The founders will be in control even if they won’t no— they will no longer own a majority of the company. But the new thing is the T shares, which is the class of shares in the hands of the long… whatever it’s called.

Kevin Frazier: The Long-Term Benefit Trust. Yes.

Roberto Tallarita: Long-term benefit. So, the T shares, which is the class of shares in the hands of the Long-Term Benefit Trust. So, this is... The idea behind the trust is that the trust, the trustees are the guardians of the social mission. They should care about building AI for the benefit of humanity, and they have this unusual power, despite having these shares with no other kind of voting rights and no economic rights.

So, they’re fake shares in a sense, but they have the power to appoint a majority of the board, four directors out of seven. So, the idea is that Anthropic is trying to tell the world we really care about this. We really care about our social mission. We not only choose a public benefit corporation but we know that, you know, the public benefit corporation, I mean, it’s an interesting tool, but it’s not that effective.

So, we actually give the power to appoint a majority of the board, not to the founders, not to the general shareholders, but to this entity, which is a self-perpetuating outside entity, in a sense, a trust… we don’t appoint these people, shareholders don’t appoint these people, but they get to appoint a majority of the key decision-makers in the company.

I mean, this is, if you think about it, it’s a pretty radical, at least put this way, and we can get to all the wrinkles and weaknesses and loopholes that are out there. But in, in the general idea, it’s a pretty radical move, and OpenAI has a different but similar, but there, a lot has happened over the last couple of years, and I won’t get into that, where there’s a charity, a nonprofit controlling the company.

So, the basic idea going around AI labs when they started thinking seriously about governance was precisely this. We want to be serious about this, so we want to give actual control rights to some outside external decision-makers who are effectively insulated, at least aspirationally, from financial pressure.

That’s the basic idea behind the trust.

Kevin Frazier: So, we’ve got this Long-Term Benefit Trust. They control the majority of the board. The other three members will be made up of some mix of the founders, and they’ll have an opportunity to weigh in on those decisions as well. But again, not a majority. The LB… Long-Term Benefit Trust will have that.

But there’s a wrinkle, and let’s start to get into the further deeper layers. This is a sort of Shrek episode of corporate governance, just going deeper and deeper into these layers. The fourth seat on the board is somewhat special, right? This is the swing seat that’s going to matter most for casting votes.

There’s a weird provision here where suddenly the employees, the folks who own the Class E shares, they can somehow become a part of this corporate governance puzzle. So, help us put the pieces together on what the Class E shares mean and how this is yet another form of check and balance that Anthropic devised.

Roberto Tallarita: So, this is very interesting because if you start digging deeper in the structure, you start seeing details that might become very important in case of a disagreement between profit and the social mission. So, it is true that the trust, as we said, appoints four directors out of seven, so a majority. But the fourth director, which is clearly the most important, the pivotal director, if there’s a disagreement, if there’s a fight in the board the founders against the trust about a key decision, maybe a release of a super powerful model, and maybe the trustee directors think that it’s too early, that they have to hold off because of safety concerns, and the other directors decide that it’s actually ready to go and it’s the competition is so fierce that for the benefit of the company they have to release the model.

If there’s a fight like this, the pivotal votes will be this fourth director. And if you dig deeper in the documents, the fourth director is not entirely under the control of the trust. So, in order to appoint this fourth of their four directors, they have to get some support from the founders or the employees, the E shares, right?

So, it’s interesting because the whole mechanism here is— the whole mechanism protecting the social mission is based on the people chosen by the trustees. And of course, also the people choosing the trustees, and we can talk about this later. And one of the four directors chosen by the trustee needs to be some kind of, I would say, broad consensus person.

They must be someone supported not only by the trustees, but by some other constituencies, whether founders or employees. There are all kinds of— I mean, the mechanism is quite complicated. I won’t get into the pers-percentages or alternative ways of appointing. But the basic idea is that it’s more of a, like, bipartisan, if you want, director.

And so, if you were already skeptical that the trustees could be effective guardians of this at some point in my explanation, you were kind of surprised and reassured, well, but they have four directors, so maybe this is really radical, and it’s mission-oriented. But now we find out that these— of these four directors, three are truly entirely controlled by the trustees.

And by controlled, I mean just the appointment is controlled by the trustees. Then they are real people, and they will make their decisions, which is another question, right? And but the fourth must be in advance kind of negotiated in some way with the founders or the employees. So, it will be some kind of cross-constituencies appointment.

Kevin Frazier: So, one of the things that you point out in your scholarship on corporate governance in the tech space in particular is that all of these people are humans, and humans are unpredictable. Humans cannot necessarily always be constrained, and they may have odd interpretations of the balancing of the factors that we’ve been discussing.

And so, in the event that the Long-Term Benefit Trust somehow starts to go a bit off the rails, right? Right now, members include people like Ben Bernanke, which I think most folks would regard as, and this is meant as no offense to Ben, a sort of graham cracker of a board member, right? He’s not viewed as, as radical in any way necessarily, or as particularly devoted to one view or another, but someone who could take the mission of a trust very seriously and implement it.

But the method for selecting future members of the Long-Term Benefit Trust is they themselves will select their successors. And so, we could imagine a sort of cascading effect, and eventually the trust starts to look a little opposed or at least contrarian to Anthropic’s underlying mission.

What’s the recourse there? What’s the sort of, for lack of better phrase, escape clause to get out of the Long-Term Benefit Trust having such incredible control over what may be one of the most powerful companies ever created?

Roberto Tallarita: So, let me first say that this is the choice of the actual individual, the people who will do this is crucial here, right?

Because the legal mechanisms, as we saw, the enforcement of the mission is pretty weak, is very weak. And so, this boils down to who are these people? Who are the decision makers? Who will be the trustees? What kind of decisions what kind of personal costs, what kind of efforts, energies, willingness to fight or they will have over the years and what they think about AI risk, what they think about the threat of super intelligence.

And we know that we have all kinds of opinions out there, and some people think that this is going to destroy humanity with a double digit probability, which is probably the scariest, most dangerous threat ever faced by the human race, ever. And lots of people think that this all conversation about AI doom and AI risk is just craziness and madness.

And so, people have all kinds of different opinions. People have all kind of different temperament and independence and willingness to pick fights. So, the key question here, as you said, is the people their characters their beliefs, their ideas. So, the trustees are a kind of self-perpetuating body.

So, whenever someone is… retires or resigns or is no longer there the other trustees replace them. So, there’s formally there’s no involvement of Anthropic, the founders Anthropic shareholders, et cetera. This is a powerful mechanism to insulate these people from all kinds of ties or pressure from the company management and the big investors in the company.

The reality is a little bit more complicated, right? Because, first of all, the trustees were put there at some point by the key Anthropic people. And, as you said, I mean, over time they can change, and it will be very important to see whether they will change in a more kind of traditional boardroom financial culture or in a more kind of AI safety direction.

And this is just— it will depend on individual choices of the current trustees. But also, these are typically high-profile people who have other jobs as well, of course, who do not have potentially the staff and the apparatus, to do the self-perpetuating I was talking about.

And in fact, the filing pretty transparently acknowledges that so far the trustees and Anthropic management, it says, have collaborated closely, and I’m quoting here, on sourcing, interviewing, and selecting trustees. Which is pretty understandable, right? You know, it’s not like Ben Bernanke is going out to put ads for new trustees and screening people and interviewing people.

Of course, they rely on the existing structure of Anthropic and management is involved. So, this might sound like an innocent-like sentence explaining just some technicalities of sourcing and recruiting, but this is crucial because as we said, people are crucial, and who these people are is the most important element of this all complicated corporate governance structure if you care about AI safety.

So, the involvement of management, I mean, it doesn’t mean that when they are appointed, they are— they have independence and they can be fired, or at least not that easily by the founders and by the company. And so, they are effectively insulated from pressures. But again, where they come from and the selection is just as important.

Because at the end of the day all this complicated baroque structure is designed to insulate from financial pressure. But another important question is, will they have other kinds of pressures, internal, philosophical, moral, scientific, political? But it... And I don’t mean external pressure, I just mean beliefs and values and character and temperament, all kind of factors that make the difference in terms of what kind of choices a person makes.

Kevin Frazier: Yeah. And just to riff on that for a second, it’s profound because in a lot of these AI governance conversations, there’s a lot of mention of concentrations of power, and folks are concerned about concentrations of power in several domains. There’s the idea that perhaps there are too few labs that are exercising control of this frontier, highly capable technology.

There’s concerns that AI may accelerate constants of… concentrations of power within governments themselves, right? We may see authoritarian regimes take over as a result of being able to leverage AI faster and for more nefarious ends. But then to zoom out and think about concentrations of power in this context is quite considerable because as we’ve outlined, you have the Class F shares, which have a majority of the voting rights in shareholder decisions, all held by seven founders.

Then you have the Class T shares held by the Long-Term Benefit Trust that is just made up of five individuals. And so, cumulatively, you can see that around 12 folks are going to have incredible influence on the direction of Anthropic. And so long as Anthropic remains on the vanguard, that to me has to be highlighted as a concern with respect to a new kind of concentration of power, which is just the sheer governance regime that’s going to be imposed here.

And so, Roberto, there is some sort of, just to borrow another phrase from other AI discourse, a kill switch of getting around the Long-Term Benefit Trust in the event that it did go in a poor direction. What is that mechanism in brief? Because I know folks won’t need to go fully into the S1, but at a high level, how do you get around the Long-Term Benefit Trust if it becomes a problem, for lack of a better phrase?

Roberto Tallarita: So, there is a kill switch. There is the possibility for the founders and employees to band together and kill basically the trust. So, to get rid of this control of rights and control powers of the trustees. It’s not easy to activate. There are all kinds of percentages based on different situations, and the important thing is that the founders cannot do this by themselves, which is an important protection.

If you’re looking at this structure and you’re worried that trustees will be allowed to be guardians of the social missions only so long as they don’t bother the founders too much, well, you will be reassured to know that the founders cannot fire the trustees or cannot get rid of the trustees unilaterally.

They need to get substantial support from the employees as well, the owners of the E shares which I guess stands for employees.

Kevin Frazier: Right. Or some combination of either the employees or of the other shareholders. But the fact of the matter remains that the founders alone aren’t going to be able to flip that kill switch.

Roberto Tallarita: Exactly.

So, you need some broad consensus across the various constituencies, or at least some combinations of those constituencies to get rid of the trust, of the trustees. Which means that I mean, in a sense, you need some mechanism like that. I don’t see it as a loophole of the social mission structure or the social mission protections because trustees can go rogue, and they’re people just like other kinds of people, and they can start misbehaving or making decisions that are neither in the interest of the shareholders as investors nor in the interest of AI safety or whatever.

So, you need some kind of mechanism, and having a mechanism designed in a way that requires support from different constituencies is a way to tell the world, tell the market that if something really bad happens, if the trustees start misbehaving, there is a way to actually affect this. On the other hand, though these constituencies who can activate the kill switch are all vulnerable, to some significant extent, to financial motives and financial pressures.

The founders, for sure, the employees owners of e-shares, they have significant personal wealth tied up in the companies. These are shares, and they will represent very likely a significant part of the wealth of the assets of these people. And the normal, let’s call them shareholders, they are just investors who are saving for retirement or managing money for their customers.

So, they certainly care about the bottom line and financial results. So, it is true that the kill switch requires some kind of alliance across constituencies, but these are all constituencies that whether deliberately or subconsciously or whatever, they will be very sensitive to stock price and financial results.

And when push comes to shove and there’s some, like, very difficult conflict between purpose and mission, and assuming that the trustees will be really true to their triple mission and will try to enforce the social mission against the profit mission. The kill switch is in the hands of people who have a lot of wealth at stake.

And the big question is, will they care about humanity? Will they care about safety so much has to bear personal financial cost, maybe a significant one, and decide to side with the trustees and against management? That’s a big question. And once again, it boils down to people and their psychology, their values, and their individual decisions.

Kevin Frazier: Well, just as we say that we’ve got a democracy or a republic if we can keep it, it sounds like we’ve got a safety-forward form of corporate governance if we can keep it. So, this will be a very interesting experiment. Roberto, thank you so much for covering a new alignment issue, rogue trustees new kill switches.

This has been a fantastic overview of one of the most important experiments in corporate governance, and I’m just so glad you came on the podcast to help us understand all of the checks and balances and the myriad layers.

Roberto Tallarita: Thank you. Thanks so much. It was very fun.

[Outro]

Kevin Frazier: The Lawfare Podcast is produced by the Lawfare Institute. If you want to support the show and listen ad-free, you can become a Lawfare material supporter at lawfaremedia.org/support. Supporters also get access to special events and other bonus content we don’t share anywhere else. If you enjoy the podcast, please rate and review us wherever you listen.

It really does help. And be sure to check out our other shows, including Scaling Laws, Rational Security, Allies, The Aftermath, and Escalation, our latest Lawfare Presents podcast series about the war in Ukraine. You can also find all of our written work at lawfaremedia.org. The podcast is edited by Jen Patja with audio engineering this episode by me.

Our theme song is from Alibi Music. And as always, thanks for listening.