Articles

Life before Big Data

dataHeadlines

“Big-Data Computing: Creating Revolutionary Breakthroughs in Commerce, Science and Society.”
— Randal E. Bryant, Randy H. Katz, and Edward D. Lazowska (Dec 2008)

“Big data: The next frontier for innovation, competition, and productivity.”
— James Manyika, et al (May 2011)

The term “big data” was included in the most recent quarterly online update of the Oxford English Dictionary (OED). So now we have a most authoritative definition of what recently became big news: “data of a very large size, typically to the extent that its manipulation and management present significant logistical challenges.” The term, however, may have appeared as early as 1944. [1]

Background

Predictive modeling and analytics are getting a lot of attention these days, perhaps in light of corporate America’s “discovery” of Big Data. Well, I have news for you, “big data” has been around longer than you think and corporate anyone did not “discover it”.

NASA’s Bigger Data

The Space Race and Cold War gave us big data, and NASA and DoD had the supercomputers to amass and store incredible amounts of data. Of note, on February 11th, 2000 NASA, launched the shuttle Endeavour on an 11-day mission. The mission, The Shuttle Radar Topography Mission (SRTM) was a joint project between the National Geospatial-Intelligence Agency (NGA) and the National Aeronautics and Space Administration (NASA). The objective of this project is to produce digital topographic data for 80% of the Earth’s land surface (all land areas between 60° north and 56° south latitude), with data points located every 1-arc second (approximately 30 meters) on a latitude/longitude grid. The absolute vertical accuracy of the elevation data will be 16 meters (at 90% confidence). [2]

One of the products, Global (Longitude 180 W-180 E, Latitude 90 N-90 S), including SRTM data for Longitude 180 W-180 E, Latitude 60 N-56 S region, contains 173 GB for the elevation raw data mosaic with gaps filled. Follow-on missions collected additional data and certainly “bigger data” in terms of storage (hundreds of gigabytes).

Other Bigger Data

While working at the Missile Defense Agency’s Threat Modeling Center we would build 6 degrees of freedom (DoF) threat missiles and a single missile could generate over a terabyte of data for a simulation. Most scenarios included more than one threat missile. I think we once had a data file that was on the order of 50 terabytes.

Surveillance data from space also exceeds most people’s concept of big data, and we have been collecting it since the Cold War. I once had to write an algorithm that would take the enormous amount of data and filter it (along with some data reduction) using things like value of information (VOI), or else it would over-saturate our networks and be useless.

Conclusion

The point is, “bigger data” has been around a lot longer than “big data”. The growth of data corresponds with the growth in storage capacity, which corresponds to the growth in technology, and if NASA had not raced to get men on the moon, we might still be drawing histograms on graph paper.

Works Cited

  1. Gil Press, “A Very Short History Of Big Data”, Forbes, May 9, 2013
  2. Strickland, J. (2011). Using Math to Defeat the Enemy: Combat Modeling for Simulation. Lulu.com. ISBN 978-1-257-83225-23

 

Jeffrey Strickland

Authored by:
Jeffrey Strickland, Ph.D.

Jeffrey Strickland, Ph.D., is the Author of Predictive Analytics Using R and a Senior Analytics Scientist with Clarity Solution Group. He has performed predictive modeling, simulation and analysis for the Department of Defense, NASA, the Missile Defense Agency, and the Financial and Insurance Industries for over 20 years. Jeff is a Certified Modeling and Simulation professional (CMSP) and an Associate Systems Engineering Professional (ASEP). He has published nearly 200 blogs on LinkedIn, is also a frequently invited guest speaker and the author of 20 books including:

  • Operations Research using Open-Source Tools
  • Discrete Event simulation using ExtendSim
  • Crime Analysis and Mapping
  • Missile Flight Simulation
  • Mathematical Modeling of Warfare and Combat Phenomenon
  • Predictive Modeling and Analytics
  • Using Math to Defeat the Enemy
  • Verification and Validation for Modeling and Simulation
  • Simulation Conceptual Modeling
  • System Engineering Process and Practices

Connect with Jeffrey Strickland
Contact Jeffrey Strickland

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s