Prism, Big Data And Double, Secret Probation


As Dean Vernon Wormer said in National Lampoon’s Animal House, “Well, as of this moment, they’re on double, secret probation.” It is one of my favorite movies and most will cheer on Delta as they get their revenge after being kicked out of school because of this. Unfortunately, having secret rules and regulations is the norm these days like the NSA’s (National Security Agency) Prism program (see figure). The FBI is involved and Prism is not the only program.

Massive snooping is not restricted to the U.S. The United Kingdom is tapping fiber optic cables to track voice and data trafffic (see GCHQ taps fibre-optic cables for secret access to world's communications). This information is being shared with the NSA giving it even a bigger data collection to play with.

Most know about the NSA’s use of drones but the FBI is starting to use them as well. Even local police and rescue groups are looking at them. There are significant benefits from using this technology but all of these programs are generating new kinds of data.

Obviously, keeping a program secret can be an advantage but the question is who watches the watchers? Trust us is not a great line given the track record of almost any organization going this kind of work. Small abuses are often overlooked and those in charge often have vested interests for keeping major ones hidden.

It remains to be seen if Edward Snowden is a hero, criminal or both (at this point there is a sealed criminal complaint). On the plus side his efforts have finally moved this discussion into the open. The challenge in the discussion is verifying any claims on the secret holders because few are allowed to examine the veracity of the claims.

There are tools for “wiretapping” and these are supposed to require a judge to issue the warrants. The Foreign Intelligence Surveillance Act (FISA) made secret warrants possible but Prism addresses meta-data like phone call source, destination and duration.

For Prism, a number of factors have come together to make it a high value proposition. One is digital communication and laws that force developers to include hooks for surveillance. The other is big data tools like Hadoop (see Essentials Of The Hadoop Open Source Project). The open source Apache Accumulo is one of the tools being used in the big data spacce. It builds on Apache Hadoop, Zookeeper, and Thrift. Tracking individual transactions is useful but tracking trends requires massaging large amounts of data. Having high speed, on-site hooks into service providers like Google and Microsoft means large amounts of data are readily available.

In the past, doing a wiretap required physical access to the switching systems. The on-site hooks now provide that in our digital environment. The problem is that most of these server sites have “lights out” operation. Unattended, remote management is the norm and controlling a drone is not the only thing that can be done from the other side of the planet. Who is doing what when will be logged but bypassing or modifying the logs is not just for movie plots.

One thing that is possible is to minimize the number of people involved in getting any particular piece of information. That number can be one even when obtaining large amounts of information. This highlights the efficiency and capabilities of a system but also increases the possibility for abuse.

The challenge is determining what is fact, fiction and fantasy. We in the industry have a good idea of what is possible and probable which is more than the average person, including our elected officials, knows about. Most will not understand how they work except at a very high level. For example, companies are already using browser information to track users.

Unfortunately many are willing to exchange an unknown level of privacy for an unknown level of security. Benjamin Franklin noted, those who would sacrifice freedom for security deserve neither.

Discuss this Blog Entry 1

on Jun 25, 2013

I believe the picture you''re showing is of the older NSA site. They have a new Data Center in Bluffdale UT with a data storage capacity of (hold your breath) 5 zettabytes! That kind of capacity will hardly do much to "discourage" the NSA from holding onto every piece of data they ever received. And although we don't officially know what processing capacity or what CPUs they have, we know they had a 2 million gallon tank built to cool it! This information shouldn't come as comfort even to the "you don't HAVE any privacy, get over it!" types, especially seeing what the government actually DOES with the data they have (cf IRS tea party scandal). The citizens of the US need to demand that a law be passed authorizing the Supreme Court to review any and all "classified" government regulations to ensure they don't unnecessarily infringe on the Constitutional rights of US citizens ESPECIALLY the Fourth Amendment.

Please or Register to post comments.

What's alt.embedded?

Blogs focusing on embedded, software and systems


William Wong

Bill Wong covers Digital, Embedded, Systems and Software topics at Electronic Design. He writes a number of columns, including Lab Bench and alt.embedded, plus Bill's Workbench hands-on column....
Commentaries and Blogs
Guest Blogs
Mar 6, 2015

Programming Efficiency: Part 2 2

A number of years ago I did some work on a PABX. The device was electromechanical and far from state of the art, but the owner had been talked into spending a bunch of money expanding it and couldn’t justify scrapping it....More
Jan 21, 2015

How To Dumb Down Smart Electronics 7

In our generation of microcontroller-enhanced everything, it has been frustrating at best to see how often the capability of smart hardware has been foiled by the design of poorly purposed software....More
Dec 15, 2014

Who Are You? (I Really Want to Know!) 7

Borrowing a stanza from The Who’s hit song seemed like a good way to bring attention to a critical topic often ignored by engineers. When finding information on a company that has an interesting product, what do we do? Of course, we look at its Web page to learn more....More

Sponsored Introduction Continue on to (or wait seconds) ×