More on the Defense Science Board -- And the Proliferation of Data

Like Jack, I was struck by the Defense Science Board’s report on the autonomy of military systems, but it was a different aspect of the report that caught my eye – enough so that I thought it was worth noting here. Buried in the report is an acknowledgement of an increasingly concerning phenomenon:  the rate at which data is collected is outstripping our ability to analyze it – even with massive data storage capacity and new, sophisticated analytical tools for processing unstructured data. Often, privacy and civil liberties advocates bemoan the increased collection of data on the theory that analytics allow the correlation of data to create new information and profiles on the data subjects.  This phenomenon, which is quite real, goes by many names – in US v. Jones it was called the “mosaic theory.”  Others liken it to finding needles in a haystack.  More commonly we simply call this idea “Big Data” – the possibility that data processing can reveal hidden attributes of an individual or group.  [Note:  I’m not taking any sides on the normative question here, just being descriptive.  As a factual matter this sort of analysis can be and has been used to uncover clean skin terrorist connections.  But it is also capable of misuse and abuse.  Whatever the use, the fact of Big Data analytics is real.] But, apparently, the flood of data is getting too overwhelming even for sophisticated analytical programs.  According to the Defense Science Board:  “Big data has evolved as a major problem at the National Geospatial Intelligence Agency (NGA).  Over 25 million minutes of full motion video are stored at NGA.”  And  the DSB report notes that the proliferation of ever more autonomous systems with ever more sensor arrays will produce a near exponential growth in the amount of data that is collected (think video + infra red + sound + whatever else we might try to capture).  They are concerned that this will simply overwhelm current analytical capabilities.  As the Board put it: “Today nineteen analysts are required per UAV orbit.  With the advent of Gorgon Stare, ARGUS, and other Broad Area Sensors, up to 2,000 analysts will be required per orbit.”  And, the government “can't hire enough analysts . . . to close these gaps.” In short (and this is really an amusing thought for me – one I want to think about a bit more before fully endorsing) the report suggests that massive data collection is actually privacy protective!  We are getting better at finding needles in haystacks, so to speak.  So, if you are concerned that finding needles is invasive of privacy the current best answer has been to prohibit needle finding.  Now, the DSB report tells us that maybe the answer is to just pile on hay more quickly than our needle finding capabilities can be improved!