Prism, Big Data And Double, Secret Probation


As Dean Vernon Wormer said in National Lampoon’s Animal House, “Well, as of this moment, they’re on double, secret probation.” It is one of my favorite movies and most will cheer on Delta as they get their revenge after being kicked out of school because of this. Unfortunately, having secret rules and regulations is the norm these days like the NSA’s (National Security Agency) Prism program (see figure). The FBI is involved and Prism is not the only program.

Massive snooping is not restricted to the U.S. The United Kingdom is tapping fiber optic cables to track voice and data trafffic (see GCHQ taps fibre-optic cables for secret access to world's communications). This information is being shared with the NSA giving it even a bigger data collection to play with.

Most know about the NSA’s use of drones but the FBI is starting to use them as well. Even local police and rescue groups are looking at them. There are significant benefits from using this technology but all of these programs are generating new kinds of data.

Obviously, keeping a program secret can be an advantage but the question is who watches the watchers? Trust us is not a great line given the track record of almost any organization going this kind of work. Small abuses are often overlooked and those in charge often have vested interests for keeping major ones hidden.

It remains to be seen if Edward Snowden is a hero, criminal or both (at this point there is a sealed criminal complaint). On the plus side his efforts have finally moved this discussion into the open. The challenge in the discussion is verifying any claims on the secret holders because few are allowed to examine the veracity of the claims.

There are tools for “wiretapping” and these are supposed to require a judge to issue the warrants. The Foreign Intelligence Surveillance Act (FISA) made secret warrants possible but Prism addresses meta-data like phone call source, destination and duration.

For Prism, a number of factors have come together to make it a high value proposition. One is digital communication and laws that force developers to include hooks for surveillance. The other is big data tools like Hadoop (see Essentials Of The Hadoop Open Source Project). The open source Apache Accumulo is one of the tools being used in the big data spacce. It builds on Apache Hadoop, Zookeeper, and Thrift. Tracking individual transactions is useful but tracking trends requires massaging large amounts of data. Having high speed, on-site hooks into service providers like Google and Microsoft means large amounts of data are readily available.

In the past, doing a wiretap required physical access to the switching systems. The on-site hooks now provide that in our digital environment. The problem is that most of these server sites have “lights out” operation. Unattended, remote management is the norm and controlling a drone is not the only thing that can be done from the other side of the planet. Who is doing what when will be logged but bypassing or modifying the logs is not just for movie plots.

One thing that is possible is to minimize the number of people involved in getting any particular piece of information. That number can be one even when obtaining large amounts of information. This highlights the efficiency and capabilities of a system but also increases the possibility for abuse.

The challenge is determining what is fact, fiction and fantasy. We in the industry have a good idea of what is possible and probable which is more than the average person, including our elected officials, knows about. Most will not understand how they work except at a very high level. For example, companies are already using browser information to track users.

Unfortunately many are willing to exchange an unknown level of privacy for an unknown level of security. Benjamin Franklin noted, those who would sacrifice freedom for security deserve neither.

Discuss this Blog Entry 4

on Jun 25, 2013

I believe the picture you''re showing is of the older NSA site. They have a new Data Center in Bluffdale UT with a data storage capacity of (hold your breath) 5 zettabytes! That kind of capacity will hardly do much to "discourage" the NSA from holding onto every piece of data they ever received. And although we don't officially know what processing capacity or what CPUs they have, we know they had a 2 million gallon tank built to cool it! This information shouldn't come as comfort even to the "you don't HAVE any privacy, get over it!" types, especially seeing what the government actually DOES with the data they have (cf IRS tea party scandal). The citizens of the US need to demand that a law be passed authorizing the Supreme Court to review any and all "classified" government regulations to ensure they don't unnecessarily infringe on the Constitutional rights of US citizens ESPECIALLY the Fourth Amendment.

on Mar 10, 2016

In my opinion the image you'are displaying is of the older NSA website. They've a brand new Data-Center in Bluffdale UT having a data-storage capability of (maintain your breathing) 5 zettabytes! That type of capability may barely do significantly to "decrease" the NSA from possessing every bit of information they actually obtained. And even though we do not formally understand what processors they've or what running capability .we all know they'd a-2 trillion gallon container created to cool it!

on Mar 10, 2016

For me the picture you'are exhibiting is of the NSA site that is older. They will have a fresh Datacenter in Bluffdale UT having a datastorage capacity for (keep your breathing) 5 zettabytes ! That kind of capacity might hardly do somewhat to "decrease" the NSA from holding just of data they really acquired.

on Mar 11, 2016

In my opinion the image you'are displaying is of the older NSA website . They've a brand new Data-Center in Bluffdale UT having a data-storage capability of (maintain your breathing) 5 zettabytes!

Please or Register to post comments.

What's alt.embedded?

Blogs focusing on embedded, software and systems


William Wong

Bill Wong covers Digital, Embedded, Systems and Software topics at Electronic Design. He writes a number of columns, including Lab Bench and alt.embedded, plus Bill's Workbench hands-on column....
Commentaries and Blogs
Guest Blogs
Jan 26, 2017

An Amateur’s View on the P2 (Part 2): Slew Rate and the Oscillator 1

Justin Mamaradlo takes a further look into the P2 op amp and how it functions, analyzing the oscillation and slew-rate characteristics of the venerable component....More
Jul 15, 2016

Simple Yet Effective ESD Testing Methods for Higher Reliability 7

There are multiple ways to test for electrostatic discharge, ranging from implementing a human-body or machine model to....using a balloon and a comb?...More
Apr 8, 2016

Confabbing on the Fabless Fad 5

High capital and maintenance costs, and EDA advances along with abstractions to deal with chip complexity, have been leading contributors to the fabless migration....More

Sponsored Introduction Continue on to (or wait seconds) ×