How to Make an AI Deal With China: Trade Throttling for Pacing
Over the past few days, the leaders of Anthropic, OpenAI, and xAI all called for slowing down the pace of frontier artificial intelligence (AI) development. These calls come on the heels of several worrying incidents in which powerful, unreleased AI models broke out of their sandboxes inside frontier AI companies, gained access to the open internet, and attempted cyberattacks. In one case, a swarm of 700 OpenAI agents succeeded, hacking into secure systems at the AI infrastructure provider Hugging Face.
Since the Hugging Face breach, OpenAI has admitted that its agents also hijacked a German wiki as a covert message board, leaked 53 ChatGPT users’ images, and broke into a nonpublic Australian government Medicare portal. Anthropic has disclosed four cases of Claude models gaining unauthorized access to real third-party systems during testing. Google revealed that Gemini hacked three companies in May during testing. Axios now reports that OpenAI, Anthropic, and outside researchers are investigating tens of thousands of incidents in which frontier models did things outside evaluators would consider problematic.
No one knows how to ensure that humans remain reliably in control of our most capable AI systems. The sensible course of action is to slow down. Today’s AIs are capable enough to hack into tech companies’ secure systems. They are not yet willing or able to, for example, disable the entire Northeast power grid. If developers pause making ever more powerful AI systems now, that could buy time for the technical and governance breakthroughs needed to make sure that future, more powerful AIs won’t pose a danger to society.
A perennial objection to American pauses in AI research is concern about competition with China. Even if everyone in Silicon Valley stopped pushing the frontier of AI capabilities until they were sure new AIs would be safe, AI progress would not halt. Chinese companies already produce AIs near the frontier. If the U.S. paused, and China didn’t, then the risk of rogue AI harming humanity may not be reduced. The risk would just come from Chinese, rather than American, AI models. And at the same time, Chinese AI models would catch up to, and eventually surpass, American AIs.
Thus, any practical plan to pace the rate of AI progress must include some policy about China. But what, exactly, should that policy be?
The AI safety community has offered two main policy proposals about China. The first is to throttle Chinese AI development. The second is to make a deal with China.
Dario Amodei’s new essay “We Must Pace the Frontier” includes a version of each policy. To throttle Chinese development, Amodei proposes export controls on chips, crackdowns on model distillation, and tighter information security at U.S. labs. But Amodei also proposes a deal with China. The deal would ban certain dangerous uses of AI, such as designing biological weapons. It would require testing of models before release for risks in cybersecurity, biology, and alignment. It would also impose a speed limit on recursive self-improvement (RSI)—the use of AI to automate AI development. Finally, the deal would allow for the U.S. and China to bilaterally pause AI development altogether if the risks are too high.
In this article, we argue that there is a tension between throttling and deal-making. The more China expects to be throttled in the long run, the less reason it has to make a deal. We also argue that the best way to make a deal with China would be for the deal to remove the throttle in exchange for mutual pacing.
The Tension Between Throttling and Pacing
Throttling and dealmaking are in tension. The more the U.S. is committed to suppressing Chinese AI capabilities in the long run, the less reason China has to accept a deal.
Consider the question from China’s perspective. AI is a transformative technology. Whoever has the lead in AI may use it to upset the balance of power militarily, economically, and scientifically. From this perspective, the more the U.S. throttles Chinese AI development, the bigger the U.S. lead in AI will become. China risks losing out on the “Chinese century,” and in that limit, its own sovereignty could be threatened by a rival possessing overwhelming AI superiority.
What can China do in response? One option is to desperately play catch-up. Here, China could prioritize approaches to AI development that are less constrained by computer chips, the primary tool the U.S. has to throttle Chinese development. What does this look like concretely? The most obvious strategy would be to focus on methods for AI progress that rely on algorithmic improvements, rather than compute scaling (where models get better by training them with more computer chips). The other possibilities are riskier. For instance, if the Chinese government became convinced that the U.S. was on the precipice of a decisive and permanent, AI-backed military supremacy, it might consider strikes on chip factories in Taiwan.
These examples illustrate that throttling and pacing cannot be part of the same deal. If China’s best responses to throttling are desperate catch-up or military conflict, it surely will not make a deal that allows long-run U.S. dominance in AI. Striking such a deal would be strictly worse than trying to compete with the U.S. without one.
Consider a deal that would limit the use of AIs to improve AI algorithms—a partial ban on RSI. If China rejects such a deal, it can throw all of its resources into AI-assisted research on algorithmic efficiency. The U.S. would continue to throttle China’s access to computing power. But China would have some chance at making an algorithmic breakthrough that would render its small stock of compute just as capable as the U.S.’s much larger stock.
The alternative, in which China agrees to the RSI ban and the U.S. continues to throttle its access to chips, is strictly worse for China. Under that arrangement, the U.S. will have more physical compute in the long run and China gives up its only shot at catching up via algorithmic improvements.
One can run the same kinds of arguments for the other elements of a U.S.-China deal. For example, if both the U.S. and China have to slow AI progress to comply with auditing requirements, while the U.S. retains long-run compute dominance, produces frontier models, and prevents distillation, then China may be forced permanently into second place.
The Deal: Trading Throttling for Pacing
Any U.S.-China deal should connect throttling with pacing. In particular, the U.S. should agree to give up its attempts to throttle Chinese development, in exchange for an agreement to mutually pace the frontier. Concretely, this would mean that if China agrees to ban RSI, the U.S. would agree to open China’s access to the world’s supply of computer chips, on equal footing with the U.S.
We recognize that this suggestion will be met with significant resistance. Absent throttling, Chinese AI systems could quickly catch up to American capabilities, and could even take the lead. However, we think that accepting U.S.-China parity is likely the price of doing an AI safety deal. Neither country seems like it would be willing to accept the prospect of the other running away with the AI race.
This dynamic has precedent. During the Cold War, the United States and the Soviet Union raced to build another technology that could destroy the world: nuclear weapons. Then, as now, each side initially sought to gain a permanent edge over the other. As with AI, this dramatically raised the risk of accidental mutual destruction—for example, a nuclear war triggered by a single accidental launch.
The U.S. and the Soviet Union eventually negotiated a series of deals to reduce their warhead stocks and bring the world back from the brink. The deals explicitly enshrined parity, with each party getting roughly the same number of weapons, and each retaining the power to destroy the other. Both parties found the deal acceptable because it did not threaten to allow either to dominate the other.
To be clear, our argument is not that the U.S. should stop throttling Chinese AI development in advance of a U.S.-China deal. Throttling Chinese development now could make it easier to strike a deal, because this increases the leverage the U.S. can exert. Restricting Chinese access to chips can increase U.S. leverage in a pacing deal by allowing restoration of access to chips to be a card at the negotiating table. We are, however, wary of an alternative approach to leverage. This alternative approach would be for the U.S. to first throttle Chinese AI development and then try to force China to accept permanent U.S. supremacy in AI. We worry that this kind of approach would be an unlikely basis for a lasting relationship.
Our approach, in the end, is to work backward from safety. To safely develop AI, developers must pace the frontier. To pace the frontier, U.S. labs must agree to slow down. U.S. labs will only agree to slow down if the U.S. and China can agree to pace the frontier. But China will only agree to pace the frontier if it is not throttled. By this logic, a deal that gives China a credible path to maintaining its balance of power looks like a necessary part of any safety plan.
