Thursday, January 31, 2013

Big Data: Yes we can! Should we?

Martin Fowler has a gift for giving brilliantly simple explanations of complex topics. His info deck on Big Data is a must-read for anyone in the software industry. As he says:
Big data is a term that's generated a lot of hype. But I think it's important to resist our usual aversion to hype in this case - there is a significant change in thinking that's happening. 
This shift forces us to change many long-held assumptions about data. It opens up new opportunities, but also calls for new thinking and new skills.
(from "Thinking about Big Data" by Martin Fowler
I think it's also important to consider the implications of this technology. The mining and correlation of big data may have consequences for our privacy and freedom that we may not like. In this early part of the 21st century, software developers are actually shaping society, for example with social networking technologies. The technology itself may be neutral, simply a fact, but its application has consequences in the wider world. As one of those society-shapers, where do you stand on this?
Facts are simple and facts are straight 
Facts are lazy and facts are late
Facts all come with points of view
Facts don't do what I want them to 
(from "Crosseyed and Painless" by Talking Heads) 
Nicholas Carr is a particularly insightful writer on how software technology is changing the world - I recommend his recent article on Big Data as a digestif after you've devoured Fowler's info deck:
This is the nightmare world of Big Data, where the moment-by-moment behavior of human beings — analog resources — is tracked by sensors and engineered by central authorities to create optimal statistical outcomes.
(from "Max Levchin's dream" by Nicholas Carr
Is that hyperbole? Well this is what Google's Executive Chairman Eric Schmidt says:
Technology is not really about hardware and software any more. It’s really about the mining and use of this enormous [volume of] data [in order to] make the world a better place.
(from "Google’s Schmidt: ‘Global mind’ offers new opportunities" at MITnews
Now I kind of like Google. But who put them in charge - and can I vote them out if I don't like their vision of "a better place"? And you, skilled developer of software, how will you use your talent?



Tuesday, January 29, 2013

Software Development and the English Language

My eleven-year-old son has started programming. An avid user, gamer and PowerPoint wiz for years now, he's been increasingly curious about "how computers work". After looking at a few programming environments for beginners we've found that Microsoft's Small Basic is an excellent introduction for the budding computer scientist.  For each small step he takes there's the reward of being able to program something interesting and most of the time he can make progress by himself.

Like most beginners, the first lesson he learned was that computers demand very precise instructions. The programmer must be clear about what he wants to achieve and express it precisely, so that the computer "understands". Computers are particularly fussy but any sort of writing can benefit from the same advice: be clear about what you want to say and then express it precisely.

I thought of this yesterday while reading through "Quality System" documentation. Now I'm not the greatest fan of documented quality systems, to put it politely; instead I try to live every day of my professional life according to the agile manifesto, valuing working software over documentation and processes. But sometimes I have to explain our work methods to customers and partners and it helps to have some high-level documentation. Unfortunately, in my experience, most quality documents are poorly written: they use a special insiders' language with terms I find vague like "quality policy", "quality assurance", "change management", "traceability" and on and on. In short, they would be a lot more useful if the writer was clear about what he wanted to say, why he wanted to say it, and then expressed it precisely.

George Orwell's essay "Politics and the English Language" is wonderfully clear and precise in describing the verbosity of politicians and how it might be corrected. As an essay it gives a great example to follow and its recommendations are valuable for any kind of technical writing. It concludes as follows:
  1. Never use a metaphor, simile, or other figure of speech which you are used to seeing in print.
  2. Never use a long word where a short one will do.
  3. If it is possible to cut a word out, always cut it out.
  4. Never use the passive where you can use the active.
  5. Never use a foreign phrase, a scientific word, or a jargon word if you can think of an everyday English equivalent.
  6. Break any of these rules sooner than say anything outright barbarous.
This is another manifesto I try to follow every day of my professional life. You can tell from this blog how well or how badly I'm doing.