Today, a revolution in software technology that allows for the highly automated and instantaneous analysis of enormous volumes of digital information has transformed the N.S.A., turning it into the virtual landlord of the digital assets of Americans and foreigners alike. The new technology has, for the first time, given America’s spies the ability to track the activities and movements of people almost anywhere in the world without actually watching them or listening to their conversations.
The partnership between the intelligence community and Palantir Technologies, a Palo Alto, Calif., company founded by a group of inventors from PayPal, is just one of many that the National Security Agency and other agencies have forged as they have rushed to unlock the secrets of “Big Data.”
While once the flow of data across the Internet appeared too overwhelming for N.S.A. to keep up with, the recent revelations suggest that the agency’s capabilities are now far greater than most outsiders believed. “Five years ago, I would have said they don’t have the capability to monitor a significant amount of Internet traffic,” said Herbert S. Lin, an expert in computer science and telecommunications at the National Research Council. Now, he said, it appears “that they are getting close to that goal.”
The agency’s ability to efficiently mine metadata, data about who is calling or e-mailing, has made wiretapping and eavesdropping on communications far less vital, according to data experts. That access to data from companies that Americans depend on daily raises troubling questions about privacy and civil liberties that officials in Washington, insistent on near-total secrecy, have yet to address.
“American laws and American policy view the content of communications as the most private and the most valuable, but that is backwards today,” said Marc Rotenberg, the executive director of the Electronic Privacy Information Center, a Washington group. “The information associated with communications today is often more significant than the communications itself, and the people who do the data mining know that.”
When separate streams of data are integrated into large databases — matching, for example, time and location data from cellphones with credit card purchases or E-ZPass use — intelligence analysts are given a mosaic of a person’s life that would never be available from simply listening to their conversations. Just four data points about the location and time of a mobile phone call, a study published in Nature found, make it possible to identify the caller 95 percent of the time.
“We can find all sorts of correlations and patterns,” said one government computer scientist who spoke on condition of anonymity because he was not authorized to comment publicly. “There have been tremendous advances.”
Mr. Rotenberg, referring to the constitutional limits on search and seizure, said, “It is a bit of a fantasy to think that the government can seize so much information without implicating the Fourth Amendment interests of American citizens.”
So a few summary points, so far, are:
1) The original scare stories from the media were a “big fail” as they were based on hyperbolic rhetoric based on imprecise, inaccurate or even fabricated claims. This had an even greater effect on a public that doesn’t generally understad the technical capabilities and related issues.
2) “Big Data” techniques have, in recent years, increasingly allowed investigators to infer things about an individual from “metadata” (phone call; #’s dates. durations etc or email; same) that it isn’t clear are protected under Constitutional guarantees, without requiring those agencies to access data (actual content of phone calls and emails) that are protected.
3) With the ever increasing amount of data available over the internet (doubles every 2 years) and the ever more effective analysis available from tools that use this data, has or is a turing point been reached where the use of this data by the government reached a point where it does infringe upon our Constitutional Rights? For example, with the growth of “the Internet of things” the use of these tools will enable the government to infer much more about us. Having access to the data from my internet connected home thermo stat and home security system will allow the government to infer when I and my family leave the house.