BIG Data? How about GINORMOUS Data?

Picture

The “cloud” just isn’t big enough to store the amount of data that I want to store.  I want to instrument EVERYTHING and create a real-time analytical datastore that allows for real-time and predictive action on most anything…

…The technology exists and any good programmer can write analytical models to look for statistical anomalies and opportunities to exploit unstructured data in ways we haven’t thought of.  A little history…


Picture

…Model Output Statistics (MOS) The weather agencies have been modeling weather patterns for decades using super-computers to raise the accuracy of their predictions.  BIG DATA techniques have been used to match models for weather forecasting and we can learn a lot from the best-practices and analytical problems that have already been solved.  Computer models are great, but when you add “sky-watchers” and “storm-chasers”, the models get smarter because humans are (still) better at complex problem-solving, so when you build your ‘big data’, there’s that one important element that you need to consider…

Picture

…Brainpower.  Instrumenting data-capture is fun.  Storing it is easy and cheap.  Analyzing it is complex but we keep getting better at it.  Analyzing the data-sets, understanding the output and making sense of it all is hard work.  And visualizing it in a way that humans can easily understand and act upon is impossible.  Well, it pretty much is with current visualization technologies.  So, you see, the opportunities are humongous!..

…but I haven’t gotten to my thought-experiment yet.  What are some of the best use-cases that an entrepreneur can exploit?  How about completely instrumenting the human-body and analyzing health in real-time across the entire global population?  I will explore this in an upcoming thought-experiment but use this as thought provocation for other ginormous data uses.

P-Cz