Thursday, January 31, 2013

Big Data: Yes we can! Should we?

Martin Fowler has a gift for giving brilliantly simple explanations of complex topics. His info deck on Big Data is a must-read for anyone in the software industry. As he says:
Big data is a term that's generated a lot of hype. But I think it's important to resist our usual aversion to hype in this case - there is a significant change in thinking that's happening. 
This shift forces us to change many long-held assumptions about data. It opens up new opportunities, but also calls for new thinking and new skills.
(from "Thinking about Big Data" by Martin Fowler
I think it's also important to consider the implications of this technology. The mining and correlation of big data may have consequences for our privacy and freedom that we may not like. In this early part of the 21st century, software developers are actually shaping society, for example with social networking technologies. The technology itself may be neutral, simply a fact, but its application has consequences in the wider world. As one of those society-shapers, where do you stand on this?
Facts are simple and facts are straight 
Facts are lazy and facts are late
Facts all come with points of view
Facts don't do what I want them to 
(from "Crosseyed and Painless" by Talking Heads) 
Nicholas Carr is a particularly insightful writer on how software technology is changing the world - I recommend his recent article on Big Data as a digestif after you've devoured Fowler's info deck:
This is the nightmare world of Big Data, where the moment-by-moment behavior of human beings — analog resources — is tracked by sensors and engineered by central authorities to create optimal statistical outcomes.
(from "Max Levchin's dream" by Nicholas Carr
Is that hyperbole? Well this is what Google's Executive Chairman Eric Schmidt says:
Technology is not really about hardware and software any more. It’s really about the mining and use of this enormous [volume of] data [in order to] make the world a better place.
(from "Google’s Schmidt: ‘Global mind’ offers new opportunities" at MITnews
Now I kind of like Google. But who put them in charge - and can I vote them out if I don't like their vision of "a better place"? And you, skilled developer of software, how will you use your talent?

1 comment:

  1. For a counter-point to Fowler's positive assessment of Big Data technologies, see the article by the always interesting Julian Browne, "The Big Data Deception"