I think that the general point here is that if the government manages to develop such software in secret, without rigorous oversight, it will be abused. (Two cases in point: warrentless wiretaps, and the FBI’s national security letters) My personal worries are not that the technology is inherently uninvasive or secure, but that the implimentation and use will invasive.

Here’s a quick thought experiment: Let’s say that we had some database (assume for the moment that it is fairly rudimentary) that is used in some fashion to pick out suspects. Let’s say that the information in the database is also perfectly secure, and you encripted pointers (about which I know very little) are used to prevent operators from “snooping” through the data. One day, Joe Citizen buys tickets to visit a friend in some exotic locale, and he gets flagged as a “person to keep a close eye on.”

What happens now? Will the government warn him that he is being watched, or ask for him to explain himself? Or will he be scrutinized more closely without being informed? What form will the scrutiny take? Will the government be able to get a warrent to tap his phone, or search his home? Will the scrutiny end after a time, or will he be watched indefinitely? Will the information gathered by this scrutiny be used elsewhere?

It seems unlikely that we will be able to build a “Threat Identification Program” that will never have a false positive, and still have a acceptable false negative rate. How can we prevent abuse?

None of these are technological problems, they are social and political ones. However, they need to be resolved before the technology is allowed to mature.

What disturbs me is that the term data mining is being applied to projects where there is very little to no development of models for preemptive detection of future terrorist attacks or in the screening of cargo or travelers and their risk scoring.

What is also disturbing to read is that the press, congress and the public have no clue about new technologies developed recently which make use of ‘encrypted pointers’ which basically means the original data is never moved or exposed. This encrypted technology means millions of records can be screened for the matching of known terrorists’ names or for suspicious patterns identifying terrorist-related crimes, such as money laundering and identity theft – without violating citizen privacy.

I am the author of five books on data mining and was the data mining consultant on the first department-wide audit of DHS.

Anyone with common sense can tell you the answer to that. It’s NO! You dont have to have a debate, discussion or long legal process to understand that someone using data on you without your permission or knowledge invades your privacy and civil liberties.

Then again we are talking about our government which shows only very rare individuals with Common Sense.