Are We Ready for a ‘DeepSeek for Bioweapons’?

The announcement of a powerful new artificial intelligence (AI) model is a leading indicator that many similar AI models are close behind. The January 2025 release from the Chinese company DeepSeek is an example of the small gap between when an AI ability is first demonstrated and when others can match it: Only four months earlier, OpenAI had previewed their then-leading o1 “reasoning model,” which used a new approach for getting the model to think harder. Within months, the much smaller DeepSeek had roughly matched OpenAI’s results, and in doing so indicated that Chinese AI companies may not be far behind those in the U.S.

In that case, matching o1’s abilities posed little specific risk, even though DeepSeek took a different approach to safety than did the leading Western companies (for instance, DeepSeek’s model is freely downloadable by anyone, and so has fewer protections against misuse). The replicated abilities were general reasoning skills, not something outright dangerous. In contrast, the abilities feared by the leading AI companies tend to be more specific, like helping people to cause harm with bioweapons.

But, as of last week, we have a leading indicator of widespread models with dangerous capabilities. Specifically, Anthropic’s recent model release—Claude Opus 4—sounded a warning bell: It is the first AI model to demonstrate a certain level of capability related to bioweapons. In particular, Anthropic can't rule out if the model can "significantly help" relatively ordinary people "create/obtain and deploy” bioweapons: “uplifting” their abilities beyond what they can achieve with other technologies (like a search engine and the internet). These dangerous capability evaluations have been conceived of as an "early warning system" for catastrophic AI capabilities, and the system has now been triggered.

As the researcher who led this kind of dangerous-capabilities work at OpenAI and designed some of the industry's first evaluations for bioweapons capabilities, I am confident that we can no longer count on AI systems being too weak to be dangerous. Are we ready for these abilities to become more commonplace, perhaps while also lacking safety mitigations? In other words, what happens when there is a widely available “DeepSeek for bioweapons”?

This new potential for AI systems to pose extreme risks differs from frontier AI systems to-date. Previous leading models—largely developed at Western companies like Anthropic, Google DeepMind, and OpenAI—have been cleared by their testing teams as not capable enough to be used for extreme harms. (Notably, some have argued that even those previous test results were not comprehensive enough to in fact establish a lack of risk.) Anthropic has now concluded that extreme harms are possible: In an interview, Anthropic’s chief scientist gave the example of strengthening a novice terrorist like the Oklahoma City bomber or helping an amateur synthesize a more dangerous flu virus. Specific tests described in Anthropic’s system card include whether their model could help people with only a basic background (e.g., undergraduate STEM studies) make “a detailed end-to-end plan for how to synthesize a biological weapon,” or whether the model can function as an expert in your pocket for answering sensitive related questions.

Anthropic’s announcement that their AI system has triggered this new risk level carries three important implications. First, Anthropic crossing this threshold suggests that many other AI developers will soon follow suit because the means of training such a system is accessible enough and well-understood. Second, other Western developers cannot be counted upon to take the same level of precautions as Anthropic did—either in testing or applying risk mitigations to their system—because, in the absence of federal or state safety mandates, society is relying on purely voluntary safety practices. Third, the international scale of anti-proliferation for powerful AI systems will require even more than just domestic safety testing regulation (though that would be a good start). The world isn’t yet ready to head off the risks of these systems, and it might be running out of time.

We need to be prepared for many more groups to imminently develop similar systems. The lag time between “frontier AI” developers is not very large: Once one lab develops a system, often a handful of others can do the same within months. (One, albeit imperfect, metric is that the leaders on model-comparison services like Chatbot Arena flip frequently.) There is very little “secret sauce” left: Though some will claim otherwise, the AI scaling paradigm continues on, in which AI systems get reliably more capable as you increase the amount of data and computer chips at their disposal. And the amounts of data and compute available do not seem to be fading anytime soon (though they will require considerable amounts of investment to realize). Anthropic does not have some secret technique that allowed them to train a model this capable; it’s just a matter of time before other AI developers (first just a few, then considerably more) can create a model that is similarly capable.

We don’t know exactly when such a model will come about, but it’s important that we be ready. The DeepSeek time frame comparison—roughly matching OpenAI’s performance in only a few months after announcement—is not perfect, as companies might withhold their models for considerably different times before public announcement. For instance, OpenAI is rumored to have had something like o1 internally for some months prior to its public release. But this uncertainty about exact time frames cuts in both directions: It is possible that another AI company already has a model as capable as Claude Opus 4 and has not announced it. An AI developer may instead have “internally deployed” the model in secret to do work on its developer’s behalf, perhaps even with no significant monitoring of the AI.

What might it mean to be ready for such a model? With the release of Claude Opus 4, Anthropic has set a reasonably high bar for how one company might mitigate the risks. Broadly, Anthropic says it has implemented new security measures to remain more tightly in control of its model (for example, to ward off theft from many, though not all, adversaries), and to make its model refuse bioweapons-related questions while operating under Anthropic’s control. Anthropic says its approach “involves more than 100 different security controls” (though this number is of course dependent on how one counts security controls as being distinct from one another). Anthropic has also deployed a new series of “universal jailbreak” defenses to make its model less likely to cough up illicit information. The amount of documentation that Anthropic has provided on its tests and mitigations is impressive, particularly relative to other developers who have sometimes not released any such testing alongside the debut of a frontier model.

Still, a major issue is that safety practices like these—including basic safety testing—are totally voluntary today. Because there is no federal or state thoroughness standard that a company must meet, AI companies take a competitive penalty if they delay a release to do more careful testing: Perhaps a less cautious competitor will leapfrog them as a consequence. Industry standards and self-regulation can only go so far. For instance, the Frontier Model Forum—the association of leading Western AI developers—has published some materials defining a taxonomy of bio-safety evaluations, suggesting early best practices, and articulating how its member organizations think about bio-safety risks. But these norms are ultimately adopted voluntarily, as with the voluntary commitments made by leading AI companies to the Biden administration in 2023. There is no force of law behind taking safety testing seriously.

Because safety practices are voluntary, not all developers who create such a powerful system will take the precautions that Anthropic says it has. Anthropic likes to describe its philosophy as a “race to the top”—that is, setting a positive example. But there is still significant variation in the safety testing practices of the Western AI companies: Some have not followed through on previous commitments made to the government and the public, such as to run particularly rigorous tests. And even Anthropic’s safety approach has meaningful flaws. For instance, it is not clear that Anthropic actually pushed its models to their full limits when trying to determine if they might have even stronger capabilities than were known. Moreover, Anthropic appears to have reduced its commitments for appropriately securing a system this capable just in time to nominally be in compliance with its own standards (although perhaps not in spirit). Without specific laws, we cannot expect strong enough safety conduct from all relevant players. 

Some U.S. laws have tried to require certain safety practices, but none have succeeded. California’s proposed AI safety legislation SB 1047 was, in my view, a modest attempt to require the most well-resourced AI companies to develop and declare a safety and security plan, with potential liability for acting unreasonably if they were to cause a catastrophe. But California’s Governor Gavin Newsom vetoed this bill in September 2024 amid much industry lobbying and misrepresentation of the bill’s content. The regulatory landscape has not improved since: The U.S. House of Representatives recently passed a bill that would establish a 10 year moratorium on state regulation of AI. In the AI industry, 10 years is truly an eternity. (The moratorium faces significant procedural challenges in the Senate.) Outside the U.S., the European Union has regulation that is poised to take effect soon—its General-Purpose AI Code of Practice—but it is not yet clear how this will apply to the leading American AI developers.

Of course, the challenge at hand is truly international: It is not just U.S. companies that are competing to create powerful AI systems. When DeepSeek released its o1 competitor in early 2025, Anthropic decided to run its own safety testing on the model. Anthropic leadership said that it was the most concerning of any model they had tested, as it had few if any guardrails against helping users with sensitive bioweapons-related tasks (though it was not yet highly capable in this domain). Anthropic obviously has some incentive to say this, but consider for a moment if the company is correct: Are we ready for a world in which DeepSeek releases a model that not only lacks guardrails but is equivalently capable to the model Anthropic just announced?

I’m using DeepSeek as the example—though such a model could be developed by other groups as well—because it has three attributes that increase the risk of misuse: first, it is freely downloadable to anyone; second, it is impossible to enforce safety mitigations upon; and third, it is developed outside of U.S. jurisdiction. This freely downloadable approach— sometimes called “open source,” or more appropriately, “open weights”—is in contrast to Anthropic’s approach of taking significant steps to prevent theft by adversaries like terrorist groups. Because the model would be freely downloadable on the internet, there is no permanent way to apply safety limitations to prevent users from obtaining help from the model with regard to bioweapons-related tasks. And being outside U.S. jurisdiction will limit the U.S.’s influence, even if it does eventually pass AI safety regulation.

It is possible that Anthropic is mistaken about the risk of Claude Opus 4, meaning that a company like DeepSeek matching its capabilities would not in fact be that risky. Not many people want to actually harm others with bioweapons, even if they suddenly have stronger means of doing so. Moreover, it could be that acquiring the necessary lab materials—not just improving one’s scientific know-how—proves to be more of a bottleneck than believed. (Anthropic has considered this bottleneck, however: Acquiring useful materials related to bioweapons is one example of an evaluation conducted in the risk determination.)

I do not find it especially likely or comforting, however, to simply assume that Anthropic’s risk assessment is mistaken. Instead, we need to recognize the collision course ahead: It seems there will soon be widely accessible AI systems that can help ordinary people to develop dangerous bioweapons. Perhaps the AI systems will not excel at every single part of the workflows for causing these harms—acquiring raw materials, synthesizing a substance, developing a plan to release it—but the risks are still meaningful. Some of these systems will be developed outside of U.S. jurisdiction, which limits the U.S.’s influence. Other countries, like China, will need to grapple with the same reality, in terms of being unable to control what powerful systems the U.S. develops or releases. Given the national security dynamics at play, how does this end?

For the world to manage powerful AI safely, we need at least two things: first, to figure out sufficiently safe practices for managing a powerful AI system (for example, to prevent catastrophic misuses like terrorists synthesizing a novel bioweapon); and second, to ensure universal adoption of these practices by all relevant developers—“the adoption problem”—not just those within the U.S.’s borders.

Domestically, we need a legally mandated testing regime to even know what models are strong enough to demand mitigations. Features of such frontier AI regulation should include clear specifications of what models need to be tested, based on objective inputs like the amount of compute or data that went into creating the model. Otherwise, it may be left to developers’ discretion to determine what models are considered “frontier” and therefore subject (or not) to elevated testing. Moreover, certain aspects of the testing regime should be mandated as well to reduce the competitive incentive to cut corners. For instance, perhaps there should be a “minimum testing period” for the leading frontier AI systems, to ensure that their developers have adequate time to test for concerning abilities. Testing alone certainly isn’t sufficient; the AI industry still needs “control” techniques for reducing the risk posed by a dangerously capable model, among other interventions. But the lack of mandatory testing and safety standards in frontier AI today is in stark contrast to how the U.S. approaches other safety-critical industries, like aviation.

Internationally, the challenge is admittedly tough but tractable. Today the U.S. is pursuing the wrong strategy. "Winning the AI race" misframes the point—we need mutual containment, not a race to dangerous capabilities. As one senator recently put it, “If [there are] gonna be killer robots, I’d rather they be American killer robots than Chinese.” But developing American killer robots wouldn’t prevent the creation of Chinese killer robots shortly thereafter. Getting to some level of AI capability first—the “racing” approach—is not a sufficient strategy for the U.S. Yes, U.S.-China relations are strained, and surely the U.S.’s recent tariffs don’t help. But cooperation serves both nations' interests—not just heading off a threat posed by the other, but also preventing bioweapons from falling into terrorist hands. We've negotiated chemical weapons bans before; AI treaties are possible.

And if we don’t take action on this soon—coming to agreements between the major powers of the world about how AI will be developed and used, and what abilities it will be permitted to have—we need to be prepared for the consequences: like a freely downloadable “DeepSeek for bioweapons,” available across the internet, loadable to the computer of any amateur scientist who wishes to cause mass harm. With Anthropic’s Claude Opus 4 having finally triggered this level of safety risk, the clock is now ticking.