The Data Is the Target
The rise of AI analysis based on large amounts of data opens the door to data corruption as a military strategy. However, this also has potentially profound implications for civilian usage of data.
Militaries and intelligence agencies around the world are integrating artificial intelligence (AI) technology into their strategic and tactical decision-making processes. The U.S. military uses the Maven Smart System for a variety of tasks, including targeting decisions in the Iran conflict. Russia, China, and Israel are also adopting sophisticated AI decision support systems (AI-DSS) that guide everything from logistics to disinformation to cyber operations. Even Iran has reportedly recently used commercial AI models to find and target U.S. bases. These tools are spreading rapidly because militaries believe they are working.
That value also makes AI-DSS increasingly juicy targets. Adversaries will still go after bases, ships, and tanks in a war, but they will also attack what makes AI decision-making possible: networks, data centers, undersea cables, and, importantly, the data that AI systems are trained on and analyze to make wartime choices.
As militaries come to depend on AI-DSS to understand and react to international events, attacks on data could become increasingly dangerous. The more data an AI-DSS holds, the more useful it is—and the more exposed. AI even supplies the means of its own spoiling: synthetic imagery, forged documents, and disinformation at scale. A military or intelligence service that poisons an opponent’s data may intend only to fool a targeting system. What it actually corrupts can lead to economic losses, civilian harm, and, potentially, escalation.
What’s at Risk and Why?
Today’s AI-DSS rely on myriad information sources. Some of these, such as returns from space-, air-, ground-, or sea-based sensors, as well as signals and human intelligence, are developed and managed exclusively by militaries and intelligence services. But AI-DSS are generally more effective when they ingest more data. This creates an impetus to mingle data from military and intelligence sources with commercial and open-source information.
While such an approach could allow AI-DSS to become more capable, there are tangible risks to combining disparate data sources. For example, the U.S. military nearly boarded a Chinese commercial vessel after an AI system, with access to government signals intelligence and open-source data, led analysts to believe that it was carrying nuclear components to Iran.
The Russia-Ukraine conflict provides other useful examples. Before the war it would have been unthinkable to use open-source social media posts to inform military operations, but they now have surprisingly effective defense applications; the Ukrainian military used social media to help find targets and geolocate Russian equipment at the start of the full-scale invasion, leading to many documented strikes. Commercial satellite imagery has proved important to both military operations and public understanding of the war. Militaries also depend on civilian-accessible signals such as GPS for military maneuvers and drone operations.
As commercial and publicly available data becomes more plentiful and useful for military decision-making, the impetus to degrade that data is growing. Social media is now rife with disinformation about the Russia-Ukraine war; Russia has repeatedly disrupted GPS signals in Ukraine and elsewhere in Europe; malign actors are generating false satellite imagery and sharing it widely; and the U.S. military is planning on spoofing soldiers’ phone signals to prevent adversaries from tracking troop movements.
Top U.S. officials have also noted the potential utility of messing with the information environment. In February, Secretary of Defense Pete Hegseth made a jab about ordering pizzas from local restaurants just to deceive a rudimentary open-source tool, the Pentagon Pizza Index, which tracks slice delivery activity in Northern Virginia to predict the occurrence of military activity. Such a move may alarm anxious citizen watchers more than foreign armed forces, but not all militaries can afford exquisite intelligence. Unfortunately, any open-source data could be taken as an indicator of military action, even the purchasing of dozens of pizzas at random.
China’s vision to use AI to improve its global situational awareness can help illuminate the outsized value of corrupting data. As our research has shown, the People’s Liberation Army (PLA) is requesting strategic-level AI-DSS that will hoover up news articles, social media content, equity market information, and academic papers and analyze them for clues to better predict and respond to international events. The Chinese military’s requests seem to take for granted that the data will be reliable. If the data is not altogether correct, however, it could lead Beijing to misinterpret global trends, potentially triggering poor decisions.
What makes the prospect of attacks on data even more concerning is the fact that nobody knows the relationship between corrupted data and model output. Even the poisoner of a data stream cannot confidently predict how it will affect a model’s decision-making. While deception has always come with a risk of provoking miscalculations, AI’s ability to process decisions at scale makes the risks of miscalculation far greater.
Strategic miscalculations would be dangerous, but they are not the only risk posed by corrupted data. As valuable as each of these data sources can be for militaries, they are arguably more valuable to civil society. Social media has played a critical role during past crises in guiding civilians to safety and helping them find family members or resources. During Hurricane Harvey in 2017, stranded Houston residents posted their locations on social media platforms, which helped rescuers find them as 911 systems were overwhelmed. Commercial satellite imagery is a booming business that is at least as valuable for managing crops as it is for tank spotting. Jamming GPS doesn’t just disorient drones but also confuses civilians trying to drive away from danger. To make the risk more clear, imagine an entity manipulating location data to direct thousands of vehicles into traffic jams to block or at least slow down military forces. To make this happen, a government or malign actor could manipulate phone or telemetry data to trick map apps or tamper with stoplights.
Businesses and the data they rely on will also be in the crossfire. Many militaries that lack indigenous capability, such as Ukraine, now rely on commercial providers for data services including, for example, satellite communications, satellite imagery, and critical event response information. Russia has indicated it views SpaceX as a legitimate military target.
Is This Any Different From Traditional Cyber Threats?
Open-source and commercial information have been manipulated in the past. Social media platforms have long been hotbeds of mis- and disinformation, including fabricated footage and video from other conflicts, thus rendering suspect much of the content related to wars. Falsifying military data also has a long history. The Fast16 malware, active in 2005 but fully analyzed only recently, corrupted nuclear testing software not by ruining it altogether but by subtly corrupting test data to confuse scientists. In 2021, spoofed signals from the Automatic Identification System (AIS) showed two European warships approaching a Russian naval base in Crimea. The vessels, however, were docked in Odesa.
Seasoned intelligence analysts and military operators know when and how to question suspicious data, or at least they used to know. AI is changing analytical methods in several concerning ways. On the one hand, AI allows for the integration of vast amounts of data that help find the proverbial needle in the haystack. On the other hand, all that data opens up the risk of bad data. Where the source and sanctity of the data is unknown to the end user, the risk of accidentally following faulty recommendations based on corrupted information increases. To make matters worse, without strong executive leadership, the speed of AI analysis can pressure analysts to make faster decisions, taking away time normally spent interrogating the data in a system that can facilitate more decisions in ever shorter periods of time. In other words, such a scenario could lead to bad decisions at speed and scale.
Unfortunately, AI can also speed and scale the manipulation of data. Deepfakes are a common example, but they go beyond fake videos of national leaders. The open-source community revolted when Google tried to introduce generative AI for Google maps, as the tool could quickly create manipulated images of military bases, nuclear plants, or refugee camps. Google wisely reversed course within 24 hours, but not every user needs Google to manipulate a map image. It is easy to envision a future where adversaries flood consumer, government, and commercial datasets with sophisticated deepfakes, forgeries, and corrupted information that imperil military operations, civilians, and businesses alike. Investigators have already uncovered networks of websites that use AI to produce huge quantities of content aimed at mucking with AI model training data.
Compared to strikes on tanks that are easily visible, data poisoning attacks are difficult to observe in real time. Corrupted datasets can lead today’s models to keep confidently generating incorrect outputs that look plausibly correct. This corruption happens to training data as well as to operational data being used in conflicts. Manipulating data before model training can skew AI’s performance or create vulnerabilities that are triggered only under certain circumstances. Recent research suggests that it takes only a small amount of poisoned data—in one study only 250 documents in models of up to 13 billion parameters—to weaken a model’s performance.
Implications
Countries may turn to data manipulation because it is an ideal instrument in the age of constant cyber operations that fall short of outright war. Corrupting data sources is relatively cheap and deniable, and its effects may be subtle in useful ways. Over years, a country can degrade an adversary’s information environment without provoking a violent rejoinder.
There are no easy ways out of this emerging era of data vulnerability. The U.S. military should, at minimum, assume its models are targets. Preparing for diverse data attacks will be critical, as will creating automated systems that monitor for changes to model outputs, due to either adversaries’ actions or data drift, which occurs when the real-world data a model uses gradually diverges from that it was trained on. Regular reliability checks against accidents are also needed. Furthermore, it would be useful to establish robust data provenance standards and norms around verification and validation of data sources, as well as standards for reinspection, monitoring, and defense.
Alongside these technical approaches, nurturing greater capacity outside of government would help. A strong cadre of open-source intelligence sleuths, journalists, and humanitarian organizations could document data corruption and share information on its consequences. Greater corporate awareness is necessary to defend commercial datasets. Many companies that operate satellite constellations, mapping tools, social media platforms, and data services have likely never thought of their products as militarily relevant targets. The government should explore working with companies to treat their information as such and better protect it before the next crisis.
The U.S. government should also explore setting norms around information manipulation with countries such as China and Russia. Of course, this will be difficult and the chances of success slim. But it would serve the U.S. government well to clearly communicate the risks associated with data manipulation, which could lead to escalations that are in no country’s interest. The law of armed conflict applies to all means and methods of warfare, nonkinetic ones included, and the principles of distinction, military necessity, and proportionality govern information operations that amount to attacks. The government should therefore clarify its position on whether, and when, data constitutes a military objective and how to assess foreseeable incidental harm, including knock-on effects, when targeting it.
As militaries use AI-DSS to speed decision-making, they are elevating the value of commercial and open-source data. While operators are incentivized to adopt these tools to act quickly, they need to address the potential issues with incorrect and corrupted data. Fabricated information fed into military AI-DSS via any pathway could lead to deadly miscalculations.
