Prism, Big Data And Double, Secret Probation


As Dean Vernon Wormer said in National Lampoon’s Animal House, “Well, as of this moment, they’re on double, secret probation.” It is one of my favorite movies and most will cheer on Delta as they get their revenge after being kicked out of school because of this. Unfortunately, having secret rules and regulations is the norm these days like the NSA’s (National Security Agency) Prism program (see figure). The FBI is involved and Prism is not the only program.

Massive snooping is not restricted to the U.S. The United Kingdom is tapping fiber optic cables to track voice and data trafffic (see GCHQ taps fibre-optic cables for secret access to world's communications). This information is being shared with the NSA giving it even a bigger data collection to play with.

Most know about the NSA’s use of drones but the FBI is starting to use them as well. Even local police and rescue groups are looking at them. There are significant benefits from using this technology but all of these programs are generating new kinds of data.

Obviously, keeping a program secret can be an advantage but the question is who watches the watchers? Trust us is not a great line given the track record of almost any organization going this kind of work. Small abuses are often overlooked and those in charge often have vested interests for keeping major ones hidden.

It remains to be seen if Edward Snowden is a hero, criminal or both (at this point there is a sealed criminal complaint). On the plus side his efforts have finally moved this discussion into the open. The challenge in the discussion is verifying any claims on the secret holders because few are allowed to examine the veracity of the claims.

There are tools for “wiretapping” and these are supposed to require a judge to issue the warrants. The Foreign Intelligence Surveillance Act (FISA) made secret warrants possible but Prism addresses meta-data like phone call source, destination and duration.

For Prism, a number of factors have come together to make it a high value proposition. One is digital communication and laws that force developers to include hooks for surveillance. The other is big data tools like Hadoop (see Essentials Of The Hadoop Open Source Project). The open source Apache Accumulo is one of the tools being used in the big data spacce. It builds on Apache Hadoop, Zookeeper, and Thrift. Tracking individual transactions is useful but tracking trends requires massaging large amounts of data. Having high speed, on-site hooks into service providers like Google and Microsoft means large amounts of data are readily available.

In the past, doing a wiretap required physical access to the switching systems. The on-site hooks now provide that in our digital environment. The problem is that most of these server sites have “lights out” operation. Unattended, remote management is the norm and controlling a drone is not the only thing that can be done from the other side of the planet. Who is doing what when will be logged but bypassing or modifying the logs is not just for movie plots.

One thing that is possible is to minimize the number of people involved in getting any particular piece of information. That number can be one even when obtaining large amounts of information. This highlights the efficiency and capabilities of a system but also increases the possibility for abuse.

The challenge is determining what is fact, fiction and fantasy. We in the industry have a good idea of what is possible and probable which is more than the average person, including our elected officials, knows about. Most will not understand how they work except at a very high level. For example, companies are already using browser information to track users.

Unfortunately many are willing to exchange an unknown level of privacy for an unknown level of security. Benjamin Franklin noted, those who would sacrifice freedom for security deserve neither.

Discuss this Blog Entry 6

on Jun 25, 2013

I believe the picture you''re showing is of the older NSA site. They have a new Data Center in Bluffdale UT with a data storage capacity of (hold your breath) 5 zettabytes! That kind of capacity will hardly do much to "discourage" the NSA from holding onto every piece of data they ever received. And although we don't officially know what processing capacity or what CPUs they have, we know they had a 2 million gallon tank built to cool it! This information shouldn't come as comfort even to the "you don't HAVE any privacy, get over it!" types, especially seeing what the government actually DOES with the data they have (cf IRS tea party scandal). The citizens of the US need to demand that a law be passed authorizing the Supreme Court to review any and all "classified" government regulations to ensure they don't unnecessarily infringe on the Constitutional rights of US citizens ESPECIALLY the Fourth Amendment.

Newsletter Signup

Please or Register to post comments.

What's alt.embedded?

Blogs focusing on embedded, software and systems


William Wong

Bill Wong covers Digital, Embedded, Systems and Software topics at Electronic Design. He writes a number of columns, including Lab Bench and alt.embedded, plus Bill's Workbench hands-on column....
Commentaries and Blogs
Guest Blogs
Nov 11, 2014

How to Outsource Your Project to Failure 4

This article will address failure to carefully vet a potential manufacturing or “turnkey” partner and/or failure to transfer sufficient information and requirements to such a partner, a very common problem I have seen again and again with my clients over the years, and have been the shoulder cried upon by several relatives and clients in the past....More
Nov 11, 2014

Transition from the Academe to the Industry Unraveled 1

There have been many arguments here and there about how short-comings of universities and colleges yield engineers with skill sets that do not cater to the demands of the industry. There have been many arguments here and there about an imminent shortage of engineers lacking knowledge in the sciences. There have been many arguments here and there about how the experience and know-how of engineers in the industry may vanish due to the fact that they can’t be passed on because the academic curriculum deviates from it....More
Nov 11, 2014

Small Beginnings 5

About 10 years ago I received a phone call from an acquaintance. He had found a new opportunity selling some sort of investments and he wanted to share it with me in case I was interested. Ken had done fairly well for many years as a contract software developer primarily in the financial services sector. His specialty was writing RPG code. (RPG is often referred to as a write only language.) But he was seeing the handwriting on the wall as the industry moved on to other methods, and saw himself becoming a fossil....More

Sponsored Introduction Continue on to (or wait seconds) ×