Why We Opted Not to Work at the AI Labs

Knowledge is power. That’s especially true of knowledge of artificial intelligence (AI)—what it is, how it works, how to use it, and how it is developing over time. While millions of users access AI everyday, a much smaller set of people witness the current state of the art. Consider that the median researcher at OpenAI relies on AI agents to perform tasks of ever greater complexity for increasing periods of time, with the net result of AI agents performing the equivalent of 3.1 days of work for every one day of human labor. They have access to models that are months ahead of the public frontier. The individuals and organizations that can harness AI to such ends will be more productive, more profitable, and more powerful.

The power resulting from AI expertise is becoming more and more concentrated in what we refer to as the “Silicon Tower.” Our friends and colleagues have left academic roles at prestigious institutions, including our home institutions, Stanford and the University of Texas, to join AI companies. The Information documented that 22 leading academics had made that move—facilitating a brain drain from campuses to the Bay Area; it’s likely that even more have made that jump in the time since. We ourselves have had to navigate these choices, deciding whether to accept job offers for life-changing amounts of money and access versus the freedom and independence of the academy.

But that brain drain is not the end of the story. There’s also a brain wane happening—a sustained and difficult-to-reverse decrease in the collective capacity to generate and absorb knowledge about AI (and other topics!). Academics report that their most promising students are dropping out for entry-level roles at start-ups. Others warn that the would-be JDs, MBAs, PhDs, and so on of yesterday are forgoing graduate programs altogether to become today’s “member of technical staff”—the generic term for employees at companies like Anthropic and OpenAI. The combination of fewer professors staying on campus and increased numbers of students opting out of graduate degrees is alarming when it comes to the general diffusion of knowledge and science. If universities and nonprofits are no longer reliable sources of timely and high-quality research, where will such knowledge come from?

There is a case to be made that the accumulation of this knowledge by the most influential companies and research organizations is in many ways positive. All else being equal, society may be better off for having the smartest people in the world working together at a few AI companies and research entities to study the most transformational technology humankind has developed. Recent events, such as the OpenAI-Hugging Face incident—in which AI agents broke out of their testing environment, established a message board, and jointly coordinated the hack of a third-party website—make clear that there’s a tremendous amount of work to be done to make sure that the AI society is counting on to unleash new medical innovations, improve education, and make government systems more effective can operate safely. Civil society should want the best minds to work on this in the most innovative places.

Yet that concentration of knowledge is also cause for concern. As knowledge of AI rests in fewer hands, in fewer areas, and with less oversight, the odds of negative outcomes increase. The most important problems in AI, and in turn some of the most important problems in the world, cannot be solved via technical solutions on their own. They require informed democratic deliberation and legitimacy to ensure the market serves the public interest. Firms developing advanced AI have an incentive to race faster and more recklessly than would be socially desirable because of the allure of being the first to build artificial general intelligence (popularly referred to as AGI). While improving technical model alignment may help, it does not solve these racing incentives. Recognizing this, more than a thousand employees at the labs themselves call for “pacing the frontier,” seeking ways of coordinating to avoid a race to the bottom. But without a healthy ecosystem of external experts, regulators may become increasingly reliant on industry researchers to draft responsive regulations. A review of the visitors log to the White House, for example, would reveal that visits from AI CEOs and calls from staffers with ongoing connections to industry stakeholders tend to coincide with major policy shifts. Congressional hearings have also tended to include significant industry participation. Under the status quo, the public may have to form its understanding of the technology based on company talking points, with a resultant breakdown in trust over the technology’s promise and peril. And, in the longer term, the next generation of AI experts may be smaller than an alternative world in which higher education institutions have the resources and personnel necessary to train students about the latest and greatest tech.

The need for research disconnected from direct financial interest is not lost on AI companies. When OpenAI first announced its operations as a nonprofit in 2015, it acknowledged, “Since our research is free from financial obligations, we can better focus on a positive human impact.” As much as the frontier firms profess a commitment to humanity’s well-being, even the appearance of a profit motive may sap the production of that kind of research or diminish its credibility. The scholars that trade the ivory tower for the Silicon variety produce far less public-facing research. And, though quantity may not always lead to quality, it does provide a broader, more diverse range of outputs from which to cultivate new thinking and spur additional research. On the whole, AI companies tend to operate with minimal transparency relative to what’s required to inform rigorous analysis from independent experts.

Transparency alone, however, will not remedy this issue. Lessons from regulatory misadventures of the past prove this point. Sheila Jasanoff provides one such example: the Toxic Release Inventory. The Toxic Release Inventory—created and maintained by the Environmental Protection Agency (EPA)—is a repository of the chemical emissions of plants. The inventory was an attempt to respond to the Bhopal gas disaster in 1984, which involved a toxic gas leak from a Union Carbide factory in India that caused “weeks of death, panic, and disorganization.” In the aftermath of the crisis, the EPA’s aspiration for the inventory was that the information would help U.S. communities near similar entities learn about the risks of exposure to toxins. In practice, as Jasanoff put it, “[o]rdinary citizens ... could not interpret and use the information without the aid of specialist organizations that, in effect, translated the raw data into usable terms.”

There is no substitute for an institutionally independent, highly capable set of experts and organizations that can study and spread knowledge and science. The development, implementation, and assessment of regulation that the frontier labs themselves state is necessary hinges on such stakeholders. Institutions relied on to assist with those tasks, like government agencies and higher education institutions, risk losing core expertise.

How to address this shortage of stakeholders in the AI space is not an easy problem. Scholars and advocates have long highlighted that entrenched incentives drive talented students away from academic careers. Back in 2018, it was the case that individuals with PhDs in computer science could earn five times as much in industry as they could working in academia. That multiple has exploded in the age of AI. Perhaps most critically, industry jobs at AI companies also involve the opportunity to work with and develop the latest and greatest AI tools. Whereas universities may be operating under tight token limitations due to the expense of AI, AI companies afford researchers extensive token budgets and the opportunity to use frontier models before they are even publicly available. Staff at AI companies may also have access to vast troves of user data that allow them to investigate research questions that would otherwise be impossible to answer. So not only is the grass greener due to better pay, but it’s also taller, teeming with more life, and begging for more nuanced and creative inquiry. It’s unsurprising that many academics have left their positions for such opportunities, even if temporarily. And, to be clear, there’s a strong case to be made for the best and brightest to be working as closely as possible with one another and with access to inside information that may not be amenable to disclosure. Our goal is instead to highlight that there are real trade-offs—at the societal and individual levels—involved with more talent flowing to the labs.

Joining a frontier lab is not always an individually optimal decision. Many of the most important problems in the world require the independence of academic and nonprofit institutions. But even setting aside pay, working in the frontier is the best path for some of the top minds to do the most interesting research they can. These benefits, however, come with trade-offs to society as these insights stay within the walls of a few offices in Silicon Valley. The creation and diffusion of knowledge is a canonical collective action problem. It may be time to intervene to better align the private and social benefits of knowledge diffusion about AI.

We understand that this comes with enormous risks unique to AI. National security concerns caution against wide diffusion of knowledge about how to build frontier AI systems. This is all the more reason to engage external experts in designing mechanisms to ensure safe democratic accountability. To that end, consider the following suggestions:

First, it should be financially palatable for the best minds to work on problems central to AI and society without working in the frontier labs. One promising development is the sudden increase in philanthropic funds directed toward core problems in AI. A challenge that many external researchers nonetheless face is that this money does not always reach the researcher. While there are some means for philanthropy to support compensation—endowed chairs and sponsored research awards are the standard vehicles—a grant restricted to research expenses cannot be converted into additional personal pay, and salary supplements are frequently subject to institutional approval, percentage limits, or fixed terms. The result is that philanthropic funds may close the gap for resources like compute while leaving the compensation gap largely intact.  

Second, AI labs should develop or expand existing trusted research programs so that access to the most important data, such as statistics about usage, capabilities, and safety, is not exclusive to employees. Companies have strong rationales, including in the public interest, for keeping much of that information behind closed doors. However, as demonstrated through one-off collaborations with researchers at universities and nonprofit entities, labs are capable of increasing access to more trusted stakeholders. Those frameworks can be scaled to allow more verified researchers to analyze and study data that may go a long way toward shaping more responsive policies and rendering society more resilient in the face of all the changes ahead.

And third, certain academic fields—such as the law—ought to critically analyze their approach to tenure and professional development. By way of example, it remains the case that many law schools expect a tenure applicant to have written at least two law review articles. These articles should be solo-authored and placed in a law journal, which are edited by second- and third-year students and may take more than a year to be published. In practice, this looks like junior academics forgoing opportunities to write with interdisciplinary colleagues on the most pressing issues of the day in journals that are far more likely to reach audiences with influence over AI policy and development. Many talented junior scholars view such requirements (rightfully) as anachronistic and wasteful. Likewise, the years-long publication process in economics is poorly suited for working on the time scale of continued capabilities improvement in AI, but scholars still depend on top publications for tenure decisions and professional advancement. If scholars were instead rewarded for making contributions that better aligned with scientific and social interest, then academic work may become more enticing. Imagine, for instance, if participation in legislative hearings, attendance at exclusive workshops, and contributions to multiauthor papers and studies counted toward professional progress. Those achievements enable rather than foreclose scholarship that may have tremendous, immediate value in AI conversations.

*     *     *

The costs of brain wane may become most apparent when society looks for the next generation of independent experts and finds that too few people had the opportunity to join their ranks. Training those experts requires professors who can introduce students to the frontier and institutions that can afford to keep them there. The longer we allow that capacity to diminish, the harder it will be to rebuild it when we need it most.