16 Saturday May 2015
Posted in Big Data, Consider this, Good Strat, Good Strategy, Martyn Richard Jones, Strategy
Tags
Listen up Big Data playmates! The ubiquitous Big Data gurus, tied up in their regular chores of astroturfing mega-volumes, velocities and varieties of superficial flim flam, may not have noticed this, but, Hadoop is getting set up for one mighty fall – or a fast-tracked and vertiginous black run descent. Why do I say that? Well, let’s check the market. Continue reading
12 Tuesday May 2015
Posted in Big Data, Consider this, good start, goodstart, goostart, Martyn Jones
Bernice and the Martians, BATM for short, were an incredibly popular progressive-rock band.
Their first big commercial success came with the release of their first album and their planned promotional tour, which took in all continents.
The manager of the band was none other than effable polymath, Renaissance man and good all-round rogue, Ricky Jonesy – an obsessive control freak, lover of fine wines and darling of predictive analytics. He really loved his numbers, his social media and his sentiment analysis.
In fact, much of the early success BATM came about due to Ricky’s unparalleled passion for the ‘Big Data’.
Ricky was the band’s architect. He had major input into their material: what they composed; how they composed; their stage sets and lighting; where they performed; the way they played; how they dressed; were photographed; spoke; walked; and, ate and drank. In short, he controlled the whole BATM enchilada. It was like being in data-driven heaven.
As I said, their first album, a progressive-rock masterpiece called ‘Your Hole’, achieved major critical acclaim even before it was bolting out of the stalls and across the interwebs. Overnight the band became big property, and their notional market value ran higher than Twitter on steroids.
The band members were really please. The presses interviewed Bernice right, left and centre and he made no bones about the fact that a major part of their success was due to Ricky and his Big Data mojo.
Articles about the phenomenon appeared in all the major social media sites. Facebook, LinkedIn and BubbaToons. Ricky was named Supreme Data Scientist of the year by the Gardener Group, hailed as a messiah by the Big Data Front and lauded by all and sundry.
Then the band went on tour. Blazing a trail of ones and zeros across the face of the planet.
They were 5 gigs into their tour and Ricky decided to call a band meeting.
“Hi, guys” said Ricky “I’ve been analysing the stats, and I see that those yokes Big Blokes in Tights are trending strongly on the social media, coinciding with the release of their new single Never Stick A Banger In Your Ear”.
“Oh, whoa” chimes in Bernice, “tell us what we gotta do then, Ricky”.
Back comes Ricky. “Well, this is what I thought we might do”
“We take the old Fester and Ailin song Tropical Diseases, we practice it as much as can, and then we play it at the next gig in Birmingham, this weekend”
“But, Ricky!” pipes up Marty Smarty, “it’s an Irish country and western song. It doesn’t fit in with what we do, does it? And, anyway, we only have three days to get it prepared.”
Ricky responds. “Ah, you don’t want to be worrying your little head over that. Trust me. Learn the song. It’ll be great. The public will love it.”
So, BATM learn the song. It’s perfect. At the Saturday gig, they play it as the encore. The fans love it to bits and there’s not a cold cigarette lighter in the place.
Then they fly off to Palma de Mallorca for a bit of a rest before their next gig in Madrid.
The guys and gals are lounging at the poolside at the legendary Don Pimpón Espinete Plaza complex. The weather is glorious, the food is glorious, the scenery is glorious, and even the orchestra is glorious.
Then along comes Ricky, calling yet another band meeting.
“Hi, guys” said Ricky “I’ve been analysing the stats again, and I see that those yokes Spanky’s Magic Piano are trending strongly on the social media with their cover version of Engel Humpadink’s The Monkey Song”
“Oh yeah, what’s that mean for us, Ricky” chimes in Halo Popette, the bands keyboardist.
Back comes Ricky. “Well, this is what I thought we might do”
“We take the old Fester and Ailin song There’s A Dead Man Up The Chimney, and we rewrite it in the style of Tom Jones when he made that album of his, Little Fockers, was it? Then we practice it as much as can, until it’s perfect, and then we play it at the next gig in Madrid, this weekend”
“But, Ricky!” pipes up Brian McGarsical, “It’s a bit of an odd one isn’t it? I mean to say, it doesn’t fit in with what we do, does it? And, anyway, we only have four days to get it prepared.”
Ricky responds as fast as a chalked-up cat going down a drainpipe. “Ah, you don’t want to be worrying your little head over that. Trust me. Learn the song. It’ll be great. The public will love it. And anyways, it will fit nicely on the playlist, up there with Tropical Diseases.”
The band rewrite the song, and practice the Bedejaysus out of it. Ricky likes it so much that he gets the stats to confirm that this has to be number one on the next gig playlist.
Come the day of the gig, and BATM kick off, not with a progressive-rock anthem, but with There’s A Dead Man Up The Chimney. A group of young people at the front clearly are loving this new sound, but quite a few people are starting at the stage in fright, and it’s not from skunk induced paranoia either.
Two guys are having a conversation at the back of the hall.
“Yo, lunchbox, hurry this gig up, I thought this band was all progressive-rock and stuff, not this wiener schnitzel stuff.”
“No comment.”
Having divided the crowd with their first song, they play songs from their album. Again, they encore with Tropical Diseases. The crowd at the front go wild. The progressive rockers look on, bemused.
“Well, that was a mixed bag” says Bernice.
“Take it from your man Ricky. It all went fine lads. Just needs some fine-tuning of the songs and the analytics need to be a bit more real time. Take me word for it.”
Back comes a unison of “Okay, Ricky. We believe yas!”
So, off they go to Bonn, to prepare for the following weeks gig at the Live Music Hall in Cologne.
The band goes out visiting the museums, they have lunch at Brauhaus Bonnsch, and after a leisurely walk along the banks of Rhine they are taking a beer or three in a lovely little beer garden close to the United Nations campus.
Then out of the blue, a familiar voice can be heard.
“Hi, guys! We’re all goin’ on a summer ‘oliday”. It’s the voice of Ricky. “Anyway, Good news guys. I’ve been analysing the amazin’ Big Data stats again, and I see that those mensch Die Zahnarzt are trending strongly on the social media, especially on Swotter and Titter, with their amazon’ cover version of Podge and Rodge’s chillout mix of Currywurst and Microchips.”
Silence. No one says a word for the best part of infinity.
Ricky continues… “As you’re not going to ask, lads, I’ll tell you. We take the old song A Great Day for the Washing, and we rewrite it in the style of techno-Buddah-bar-chill. Then we practice it as much as can, until it’s perfect, and then we bang it out at the next gig in Cologne, this Friday. Innit. Come on lads, it’s 20 minutes of stage magic, and it’s a breeze.”
Come the day of the gig, and the band arrive early at the hall. Ricky is already there. He’s changed the stage set completely and has a new wardrobe for the lads – Bavarian romantic. They’ll soon be all Princed and Smiley Virused up to the eyeballs, wrecking ball included.
and BATM kick off, not with a progressive-rock anthem or chill, but again with There’s A Dead Man Up The Chimney. Again, a group of young people at the front clearly are loving this new sound, but quite a few people are starting at the stage in drug induced awe. Then they follow that up with A Great Day for Washing. By the time they get to the encore of There’s A Dead Man Up The Chimney, boisterous arguments are breaking out everywhere and empty crisp packets and used sticks of chalk are being thrown at the stage. It’s a disaster.
Four guys are having a conversation at the back of the hall.
“I liked the first song”
“No! The first was terrible. Minging! I want my prog rock back.”
“It’s like the choice of leprosy or the plague.”
“Down with this sort of thing.”
Next day Bernice calls an urgent meeting of the band.
Ricky kicks off.
“Well, lads bit of mid-week game yesterday wasn’t it?”
Bernice comes back with a “You can say that again, Rick”
“Don’t worry, I have analysed the social-media Big Data from all of the concerts, and we’re doing good guys. It’s in the analytics”
“We have to go back to our roots and drop all the changes we made”
A stranger in the lounge where they are having the meeting walks up to them and in simple language explains to them what has happened.
“You created a great product, a great brand, with some interesting progressive music”
“Your music was acclaimed and your world tour was eagerly anticipated by all your fans”
“But then you went wrong”
“You became data driven, dopey and data driven”
“You chased fads, tendencies and styles, and it became a mish-mash”
“People don’t want mish-mashes. Not your base. They wanted good progressive music”
“You’ve lost all credibility. No, you’re just an eccentric band of brothers and sisters that no one will really want to see more than once, if at all”
“Your former fan base is acutely embarrassed by you. That’s your bread, butter, vodka and caviar… in your terms”
“Data drive, Big Data, Big Data analytics in real time?”
“You people have no idea the damage you can do, and so easily”
To be continued…
Many thanks for reading.
01 Friday May 2015
Posted in Big Data, Consider this, good start, goodstart
Tags
Big Data, Consider this, data architecture, data management, good start, goodstart, Martyn Richard Jones
Miss Piggy said, “Never eat more than you can lift”. That statement is no less true today, especially when it comes to Big Data. Continue reading
20 Friday Mar 2015
Posted in Big Data, Consider this, good start, Good Strat, goodstart, Martyn Jones
Most of us would probably like to work in a profession recognised for its legality, decency and honesty. At least I hope so. In my line of work, what we have right now is palpable evidence that the IT industry lacks a moral compass.
Imagine this. A major sensationalist tabloid pulls together a team of diverse journalists who are set to work on a national campaign to promote very high usage of sunbeds as a cure for cancer. Why? The newspaper owner’s son owns the sunbed franchise.
The health experts criticise the publisher for being irresponsible, unprofessional and lacking in scruples.
The public is mainly undecided, but many take the story on face value and adopt the fad. The intensive use of sunbeds sharply increases. Elsewhere, in unrelated news, the cases of skin cancer show a marked increase. Some blame it on EU legislation for bangers and bananas.
In spite of protests, the press campaign continues over many months.
Eventually, and based on the evidence of recognised health experts and bodies, the press regulatory association tries to get the offending publisher to temper their claims, but without any success. It is only when the government’s lawyers step in and threaten the newspaper owners with legal proceedings, do they freeze their campaign. Much later, the editor resigns and the board of directors issue a short apology on the back pages of their much vaunted organ.
We have that in IT. Our current sunbed cure for cancer, if you believe those who are ‘bigging it up’, is undoubtedly Big Data.
I occasionally post content to Linkedin, some of it (maybe even this piece) gets promoted through the Pulse Big Data channel. There are some reasonable pieces pinned to that channel, but unfortunately, for much of the time what we get is total and moronic Big Data astroturfing. Tantamount to the equivalent of Big Data’s very own Big Lie campaign.
The Linkedin Big Data channel reflects life, and it is full of self-aggrandising and shameless marketing guff, shot-through with scandalously flimsy promotions of tendentious success stories, specious claims of value, half-truths about realisable benefits and embarrassing conjecture about the importance of social media and internet logs.
What I am referring to mainly are superficially neutral (yet virally toxic) pieces placed in the public domain in order to promote Big Data at any cost.
Now let’s step back a bit.
For over 125 years, the Financial Times (FT) has built up a solid professional reputation for accurate reporting, reliable journalism and informative editorials. The FT is a newspaper trusted by its discerning readership and admired everywhere. In fact, I could not imagine their journalists writing about markets, securities and financial houses the same way that pundits elsewhere write about Big Data, Dark Data and the Internet of Things. Because the FT knows, that maintaining the trust of their readership is far more important than winning the short-term favours of a few market players.
So consider this; if we in IT cannot bring our standards of communicating with the public up to the levels of the financial industry, at minimum, you know what that means don’t you?
Exactly. The IT industry will have a far worse public image problem than the bankers and the solicitors currently have, and we all understand the general public appreciation of those professions.
Now, call me old fashioned, but for me that possibility is worthy of serious consideration, and especially by those in IT who confuse no holds barred pimping of fads, trends and technology, in which truth, decency and honesty are optional, for ethical, candid and informative analysis and reporting of the industry.
How will the industry take these criticisms?
To go back to the sunbed analogy what we will most certainly get comments in this vein:
Whilst those who rail against ‘the cancer curing advantages of sunbed use’ may be right – or at least partially right – the sunbed revolution will continue, just as the IT revolution industry has done, and in spite of people saying that the age of computing would be a passing mania.
So, when someone tells you “intensive sunbed use is just a dangerous fad”, what they actually mean to say is that we don’t need the term any more, as intensive sunbed use is here to stay, as are those who are shrewd, unprincipled and cynical enough to cash-in on the public’s gullibility and wilful stupidity when it comes to fads.
Yes, it does get that bad.
We have people who seemingly spend all their waking lives working out not-so-original ways and means of riddling the IT industry with vacuous bullshit, and what Big Data promotion has shown us clearly is that what we have palpable and comprehensive evidence that the IT industry in general lacks a moral compass.
Is that a reflection of IT, of those who create and manipulate IT fads, or of society in general?
Many thanks for reading.
As always, please share your questions, views and criticisms on this piece using the comment box below. I frequently write about strategy, organisational leadership and information technology topics, trends and tendencies. You are more than welcome to keep up with my posts by clicking the ‘Follow’ link and perhaps even send me aLinkedIn invite. Also feel free to connect via Twitter, Facebook and the Cambriano Energy website.
20 Friday Mar 2015
Posted in Big Data, Consider this, good start, goodstart
Tags
Big Data, contradictions, data management, good start, Good Strategy, goodstart, Martyn Jones, Martyn Richard Jones
I have written at length about the fundamental contradictions of Big Data, but what I have omitted in the past is quite possibly the biggest contradiction of all. Probably because it has more to do with how Big Data is continually hyped, rather than having anything to do with Big Data as a bag of technologies – which has a whole assortment of problems in its own right.
Last time I spoke with you about the contradictions of the Big Data it was about the three Vs of volume, variety and velocity. In general, it was a view that was well received, even if not widely understood. Which of course is close enough for government work. But, get ready for “something completely different”.
If on the one hand some folk can claim that Big Data provides fact based insights and reliable forecasts of future habits, trends and preferences, then why is it so difficult to produce and socialise – yes, I like to use that term – Big Data success stories?
In short, I think we have arrived at the stage in Big Data’s cycle where it is reasonable to ask pundits to either put up or shut up.
So, why aren’t the current Big Data success fables accompanied by facts, such as names of those involved (at least businesses), the sponsors, the suppliers, the purpose of the exercise, the desired outcomes, the data used, how it is processed, what the results were, and what tangible benefits, if any, were accrued or are accruable. If that is not enough, then let people mention the technology used, the products purchased or licensed and the methodology followed.
In short, what I would like to know is why are the evangelists of Big Data telling us that bigger data is better, that more variety leads to greater insight, and that velocity is king. Why do we we told that Big Data almost assuredly results in better decisions, by people who are coy, shy or secretive about almost facts and data coming out of Big Data projects?
I have been reminded, time and time again, that there are Big Data success stories out there, and I have even been told that this information would be fully shared with me once it was agreed with the ‘clients’ that it was okay to do so. Okay, that’s fine, I know Big Data is a roaring success story (at least in people’s minds,) and I also know that it takes some time to make things up – some people are just not creative. Sure, I was told about these ‘successes’ some time ago, and you know, I’m not expecting anything that’s worth shaking a stick at, either now or later, but I’m still waiting, boys. Notwithstanding, you will still called you out as vacuous bullshitters when the time comes.
“But” I hear you cry “there is a wealth of success stories in the presses”.
Well, no, and you would wrong and gullible and foolish to think, but that is your problem, but unfortunately also mine, because this is my profession that you are playing fast and loose with.
The fact is that there is “wealth” of content that people try and pass off as legitimate Big Data success stories, but they aren’t in fact success stories, in any way, shape or form.
The thing is, people may read the blog title and even the stand-fast, but will be less inclined to actually read the article, so what remains is the impression that there are ‘loads of Big Data success stories’. But if people actually read the articles and were intelligent enough to understand them, then they would realise that inevitably there is a massive mismatch between the title of these pieces and the content. Indeed, if these pieces were actually pieces of advertising, rather than blog comments, they would be denounced in some jurisdictions for not fulfilling the advertising criteria of legal, decent and honest.
There is one more thing that Big Data evangelists (or any self-styled pundit, guru or expert for that matter) should understand, internalise and remember. If you say that you have a Big Data success story, with all the details, and that isn’t in fact the case, and it isn’t even remotely a success story or even true, then you are simply lying, and that’s deceit, it’s unprofessional and it’s unethical, and you are a scoundrel. So live with or fix it, the choice is yours.
Many thanks for reading.
20 Friday Mar 2015
Posted in Big Data, Consider this, data management, good start, goodstart
Tags
Big Data, data architecture, data management, good start, Good Strat, Good Strategy, goodstart, Martyn Jones, Martyn Richard Jones
Many people come up to me in the street and ask me what Big Data is all about. It has happened to me so many times in the past that I am convinced that it might just happen to you as well. I know sort of thing, I read the Big Data tealeaves. Nothing gets past me.
The first time a complete stranger came up to me in public and said “Hello, will you tell me what this Big Data lark is all about then?” I was lost for words, you just ask my Aunt Dolly, he can vouch for that, no problem. Later that day I read a book – it was my dad’s book – and I then decided to adopt a strategy.
Therefore, in the spirit of springtime goodwill to all men and women, I have put together this blog piece in that hope that it will enlighten, help and entertain.
What is big data?
Big Data can be characterised by the 10 Vs – yes, 10, not 4. Which, in my book, is more than enough to bring up-to-speed the average Big Data John or Jane that one meets on the street, and who naturally wish to be informed of such matters.
In layperson’s terms this a series of landmarks and pointers in the analytics space used to frame and guide the didactic aspects of Big Data.
The fundamental Vs of the Big Data canon are these:
So, let me now explain what each of these characteristics mean to those who might know and for those who might want to know.
Vagueness: This is perhaps the trickiest of questions to address, given the vast panorama that is cast before this incredibly complex yet easily graspable concept. But let me state this, and let there be no mistake about it. At this point in time, what makes Big Data vague is also what makes Big Data specific, explicit and certain. That is to say, in order to ‘come to an understanding’ of Big Data, it is necessary to completely embrace the dialectic of knowing the unknowable. So belief is an absolute essential element – belief and data, that is.
Volume – If there ever was a time to “pump up the volume”, we have it here with Big Data.
Big, voluminous, gorgeously rotund and infinite. Big Data is called Big Data because there is a lovely, roly-poly, likeable never-ending load of it. Its volumes can be measured in zeta-bytes, which you can be assured, is a helluva lot of data.
Variety – As they might say down my way, “variety is the spice of life, innit”. This is what makes Big Data so special. So appealing.
Because before Big Data there was absolutely no variety in anything, at all. We lived in a bland world, bereft of detail, nuance and diversity. Nothing could be measured, analysed or explained, because we lacked Big Data. We were ignorant. So ignorant and stupid that we couldn’t see the sense of putting the diapers next to the beer, or of offering three for the price of two.
Fortunately, today this is no longer the case if we don’t want it to be, and thanks to Big Data we have a veritable sensorial explosion. No longer is IT just a couple of symbols scribbled in crayon on someone’s school notebook.
Virility – Move over Smart Data, the new kid on the block is Big Data.
If Big Data were described in the manner of a religious text, it would be accompanied by a never ending narrative of begets.
So, what does that mean?
Simply stated, Big Data creates itself, in and of itself. The more Big Data you have, the more Big Data gets created. It’s like a self-fulfilling prophecy in 360 degree, high-definition, poly-faceted and all-encompassing knowing. The sort of thing that governments would pay an arm and a leg to get their mitts on.
Velocity – Velocity is of the essence. Velocity kills the competition. More velocity, less haste.
We demand that service is ‘velocious’. ‘Everything’ must be ‘now’, or it’s too late.
This means we need to be able to handle Big Data at velocity – at the speed of need.
Charles Babbage once stated (or maybe it was more than once) that “whenever the work is itself light, it becomes necessary, in order to economize time, to increase the velocity.”
But remember, we are dealing with mega-velocity here, so don’t drink and drive the Big Data Steamship, Star-ship or Mustang.
Vendible – If you can sell it, and sell it as Big Data, then it ‘is’ Big Data. If you can’t, then it’s not. The saleability of Big Data proves its existence.
So, what are the vendible aspects of Big Data?
Let’s leave that easy question for another day. But for now I can confidently state that it is used to mobilise armies of commentators, industry analysts, publicists, punters, writers, bloggers, gurus, futurologists, conference organisers, conference speakers, educators, customer relationship managers, salespeople, marketers and admen.
Vaticination – Edmund Burke is down on record as stating that “you can never plan the future by the past”. Now Burke may have been a clever person when it came to many things, but he wasn’t exactly a whiz when it came to Big Data.
There are people in the world who are in no doubt that Big Data provides the sort of visionary and predictive powers only previously obtainable through ritual sacrifice, magic potions and the casting of spells. Others are highly critical of the understatement implicit in this belief.
For many, Big Data will make the Oracle of Delphi look like a mere call centre.
This is why the power of vaticination plays a characteristically important role in the world of Big Data.
Voracity – This is based on the quasi-rationalist argument that Big Data is big and it has an omnipresent and insatiable self-fulfilling desire.
Big Data comes with an attendant requirement for hardware, even if it is a whole load of consumer hardware tacked together in a magnificent and miraculous mesh of magic.
Big Data can be characterised by voracity, but this comes hand in hand with the ‘ventripotent’ IT industry.
Veracity – The eminence of the data being captured for Big Data handling can vary significantly. The quality or lack of quality of the data naturally has the potential to impact the accuracy of analysis using that data.
Before Big Data arrived on the scene we knew nothing about Data Quality or data verification. This is why ETL and Data Cleansing tools lacked the power to effectively quality check and verify data, to ensure that any erroneous or anomalous data was rejected or flagged.
But now, with the sophistication of tools such as ‘grep’ and ‘awk’ at our disposal, we have the power in our hands to ensure nothing ‘dodgy’ gets into the analytical mix.
Vanity – In my opinion, to fully grasp the underlying and profound meaning of Big Data, it is essential for us to understand the difference between vanity and conceit. Max Counsell claimed that “Vanity is the flatterer of the soul”. Goethe characterised vanity as being “a desire for personal glory”. After an incident with an Anarchist (presumably a Big Data Anarchist), Blackadder remarked to Baldrick that “The criminal’s vanity always makes them make one tiny but fatal mistake. Theirs was to have their entire conspiracy printed and published in plain manuscript”.
So that ends the brief rundown of the defining characteristics of Big Data.
So, to summarise. That, which has passed before, necessarily divulges both the upside and downside of Big Data. By reaching out, opening up the kimono and relating the 10 Vs we are disclosing that which cannot be disclosed, exhibiting the absence of essential essence, and thereby opening up the entire field, discipline, profession, science and art to examination, questioning and ridicule.
Many thanks for reading.
14 Saturday Mar 2015
Posted in Big Data, Consider this, dark data, Martyn Jones
Dans ce pays-ci, il est bon de tuer de temps en temps un amiral pour encourager les autres – Voltair
My gran used to tell me that honesty pays. Of course, she never really understood banking or IT, probably because she didn’t want to know anything about them, and she never lived to witness the amazing hype circuses, the spin doctors spiel or the focus-group dog-and-pony show of the 21st century. Indeed, if honesty were a guaranteed payer my gran would have amassed more wealth than even Warren Buffet himself.
If my gran lived today, she might reflect on what Big Data might be about – maybe she would even consider it benignly, as a sort of shelter for fallen men of once uncertain virtue. We will never know. So onwards and upwards.
The Harvard Business Review contemplated honesty in somewhat different terms:
“Honesty is, in fact, primarily a moral choice. Businesspeople do tell themselves that, in the long run, they will do well by doing good. But there is little factual or logical basis for this conviction. Without values, without a basic preference for right over wrong, trust based on such self-delusion would crumble in the face of temptation.”
In a marvellous book, A few good from Univac, David E. Lundstrom narrates the story of Sperry Univac in the 1960s, one of the true great innovators in the first forty years of IT, and includes an allegory taken from the engineering front-line. I will recount it here, edited to highlight the zeitgeist, for your entertainment and as Voltaire put it, “to encourage the others”:
In the beginning was the Big Data Plan.
And then came the Big Data Assumptions.
And the Assumptions were without form.
And the Plan was without substance.
And darkness was upon the face of the Workers.
And they spoke amongst themselves, saying: “It is a crock of shit, and it stinketh.”
And the workers went unto their Supervisors and said: “It is a pail of dung, and none may abide the odor thereof.”
And the Supervisors went unto their Managers, saying: “It is a container of excrement, and it is very strong, such that none may abide by it.”
And the Managers went unto their Directors, saying: “It is a vessel of fertilizer, and none may abide its strength.”
And the Directors spoke amongst themselves, saying to one another: “It contains that which aids plant growth, and it is very powerful.”
And the Vice Presidents went unto the President, saying unto him: “This new plan will actively promote the growth and vigor of the company, with powerful effects.”
And the President looked upon the Big Data Plan, and saw that it was good.
“But?” I hear you say, “why fight it, why not take advantage of the Big Data zeitgeist?”, “Why not cash in on the grand bonanza Big Data bandwagon?” or “Monetise the 3 three famous Vs of Big Data?”
Well, it had crossed my mind, briefly, and (outside of the USA) we’ve all done stuff we have not entirely believed in, so the temptation to cash in is present, capisci? This paraphrasing of a piece from My Blue Heaven might give you a better idea:
One of my best friends makes his living as a completely phony Big Data Scientist. For two hundred bucks he can make you a Data Scientist or a Big Data guru. Some guys give you an education but this guy gives you immediate access to high paying jobs, sex that would make the 256 trillion Shades of Blah blush and a life in the City, the Big Apple or a small town in Germany.
Moreover, for an extra 250 bucks (limited time offer) you can also become a certified Big Data Neuro Trainer, which will allow you to do unto others what has been done unto you.
I also considered Big Data Brokerage, Big Data Certification and Big Data Independent Trading (New York – Paris – Peckham). The opportunities are immense.
However, what happens when the Big Data well runs dry, and I (and many others get tarnished with the mark of Big Data) become pariah by complicity, collusion or simple association?
That question I will leave for another day. But just consider the following.
All right, I admit, I am a big long-time fan of comic genius Mel Brooks, who has a knack of capturing deep insight from the human condition, especially when the human condition is off guard and shallow. In that vein, this is how I like to think the dialogue from the Dole Office scene from The History of the World Part Two would have gone, if he were to write that today:
Dole Office Clerk: Occupation?
Data Magnus Comicus: Stand-up Big Data scientist.
Dole Office Clerk: What?
Data Magnus Comicus: Stand-up Big Data scientist. I coalesce the vaporous datas of the human interaction with the social-media networking, Internet of Everything, and always-connected experience into a… viable, analytical and meaningful predictive-comprehension.
Dole Office Clerk: Oh, a Big Data bullshit artist!
Data Magnus Comicus: *Grumble*…
Dole Office Clerk: Did you bullshit Big Data last week?
Data Magnus Comicus: No.
Dole Office Clerk: Did you try to bullshit Big Data last week?
Data Magnus Comicus: Yes!
Finally, I leave you with some wise words from Israeli American professor of psychology and behavioural economics, Dan Ariely:
“Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it…”
Many thanks for reading.
13 Friday Mar 2015
Posted in agile, Big Data, Consider this, java
“Rats, rats for sale. Get your rats. Good for rat stew, rat soup, or the ever-popular ratatouille”. – Mel Brooks
Hold this thought: Everything that the Templars of Java touch turns to dreck.
In a small and timeless village in misty and mountainous Transylvania, the locals mourn the passing of yet another victim.
On the wind swept beaches of a wintry Costa Blanca, the reverberating voice of childish despair is barely perceptible through the crashing of the waves on the grey, cold and craggy rocks.
In Victorian London, a hobgoblin of indescribable and vacuous insanity stalks the silent and rain drizzled streets.
Cracking this curse will take more than the combined powers of Clint Eastwood, Mel Brooks and Homer Simpson.
A spectre haunts the face of Europe, the spectre of Big Data and the Curse of the Temple of Java.
Everything that the disciples of the Temple touch turns to blah. Everything that the disciples call their own has been blagged from elsewhere.
Take the very language of Java itself, an authentic eccentricity amongst computing languages. If Java code were real coffee grains, it would be used to make the shittiest coffee in the history of humankind.
Given the vast amounts of knowledge and experience that was washing around IT at the time of Java’s hatching, it must be considered to be the most demonic aberration of a programming language ever conceived by woman, man or beast.
“Cats have a scam going – you buy the food, they eat the food, they go away; that’s the deal.” – Eddie Izzard
If ever there was an excuse in IT for failing to deliver or for delivering badly and late, then Java is your friend.
In the hands of the right people, Java can turn a one year and $3M project into a five year and $300M project, and still not deliver anything of use.
Yet magically, and out of the people directly responsible for these debacles, no one is sacked, sued or busted as a result, the incumbent supplier either quietly leaves the scene or is rewarded for their gross incompetence and dishonesty, and in many cases a success is hailed, even if that success looks remarkably like abject failure. It is totally false, absolutely dishonest and thoroughly unprofessional. But that’s what we have, like it or not.
Java sucks, it is a horrid language, aesthetically and functionally, it’s a piecemeal pile of do-do, a dirty old ragbag of ‘object-oriented’ hacks, logical aberrations and lagoons of missing structure, dysfunctional rationality and discontinuity – and that that’s not just my opinion:
“I spent several months programming in Java. Contrary to its authors’ prediction, it did not grow on me. I did not find any new insights – for the first time in my life, programming in a new language did not bring me new insights. It keeps all the stuff that I never use in C++ – inheritance, virtuals – OO gook – and removes the stuff that I find useful.” – Alexander Stepanov
“Claiming Java is easier than C++ is like saying that K2 is shorter than Everest.” – Larry O’Brien
“I would rather use Java than Perl. And I’d rather be eaten by a crocodile than use Java.”
“If I wanted plastic scissors I’d use Java. Give me my scalpel back.”
And for the record, even Linus Torvald hates it.
But if you thought Java was a horrid, hype infested viper’s den of programming bad practice and hyper-hype, just wait until you see what’s behind Hadoop.
As long as the world is turning and spinning, we’re gonna be dizzy and we’re gonna make mistakes. – Mel Brooks
Hadoop must be the biggest piece of technical and rhetorical bullshit in the history of data management.
Repackage a series of Unix primitives (cat, grep, awk, cut, sed, wc) built on top of parallel Linux or Unix. Dress it up, take it out on the town, and call it the greatest thing since sliced bread. It is nothing less than a brazen and blatant con. Want to count words? Use wc (Unix wordcount).
Let me repeat that, using other words. If you made a compilation of extracts from the works of the world’s greatest thinkers and authors, randomised replacement of some of the words, and produced and published this compilation, as all your own work, what would you call that?
So back to when this happens, frequently, in IT.
This might fool the foolish who don’t have the first idea about anything technical, objective or rational beyond whatsapp, kiddy scripting and HTML, but if you have a clue, you know that this is a scam, a very big one. It is also dishonest.
So how do they (the scammers) get away with it?
Easy. You have bad apples everywhere. But there is another reason. For well over a decade the world of IT has become the dumping ground for the stupid, lazy and indolent kids of the comfortable middle-classes and also a hunting ground for unscrupulous wide-boys.
Listen up parents!
Do you think that your kid is way too thick to be a doctor, scientist, lawyer, researcher, professor, teacher, statistician, health worker, politician, bus driver, street cleaner, entrepreneur, sandwich maker or economist?
Your kid has no creativity beyond messing with their food?
Your kid has no sporting ability apart from skills at gaming?
The only academic ability your kid has is your money?
No worries!
IT for you, my son!
So if that’s you, then lap it up. Real knowledge and experience will not come your way, but you will learn the dogma of the Temple of Java, and you will be able to repeat it to perfection, just like Pavlov’s favourite dog.
You will learn to be be pliable, usable and even more gullible. You will know bugger all about practical IT or the architecture, evolution and application of information technology and data, and vendors will love you for it, for you will be just an extension of their idea of increasing the profit rate.
This is how IT business has become the refuge of liars, cheats, pimps and the chronically dopey, and this is why Java and Hadoop have become the ultimate expression in programming and data. It’s a geeky Greek tragedy being played out as we speak. O tempora, o morons.
But it isn’t just about Java and Hadoop. Everything the Templars of Java touch turns to dreck. Whether we are looking at aberrations and failures in rapid joint application development, end user computing, database design (refractor this, dimwits!), or solutions and domain architecture, and more, the dead cold hand of the Java Mafia is invariably behind it.
And now, to top it all off, the miserable Templars of Java want to take over and displace Bill’s Data Warehousing. You couldn’t make it up.
So, who will save the IT world from the evil doers?
To paraphrase Homer Simpson: I’m not normally a praying man, but if you’re up there, please save us, Wonderwoman.
Thank you so much for reading.
10 Tuesday Mar 2015
Posted in All Data, Big Data, Consider this, dark data, Good Strat
Tags
All Data, Big Data, dark data, data architecture, data management, Good Strat, Martyn Jones, Martyn Richard Jones
Dark data, what is it and why all the fuss?
First, I’ll give you the short answer. The right dark data, just like its brother right Big Data, can be monetised – honest, guv! There’s loadsa money to be made from dark data by ‘them that want to’, and as value propositions go, seriously, what could be more attractive?
Let’s take a look at the market.
Gartner defines dark data as “the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes” (IT Glossary – Gartner)
Techopedia describes dark data as being data that is “found in log files and data archives stored within large enterprise class data storage locations. It includes all data objects and types that have yet to be analyzed for any business or competitive intelligence or aid in business decision making.” (Techopedia – Cory Jannsen)
Cory also wrote that “IDC, a research firm, stated that up to 90 percent of big data is dark data.”
In an interesting whitepaper from C2C Systems it was noted that “PST files and ZIP files account for nearly 90% of dark data by IDC Estimates.” and that dark data is “Very simply, all those bits and pieces of data floating around in your environment that aren’t fully accounted for:” (Dark Data, Dark Email – C2C Systems)
Elsewhere, Charles Fiori defined dark data as “data whose existence is either unknown to a firm, known but inaccessible, too costly to access or inaccessible because of compliance concerns.” (Shedding Light on Dark Data – Michael Shashoua)
Not quite the last insight, but in a piece published by Datameer, John Nicholson wrote that “Research firm IDC estimates that 90 percent of digital data is dark.” And went on to state that “This dark data may come in the form of machine or sensor logs” (Shine Light on Dark Data – Joe Nicholson via Datameer)
Finally, Lug Bergman of NGDATA wrote this in a sponsored piece in Wired: “It” – dark data – “is different for each organization, but it is essentially data that is not being used to get a 360 degree view of a customer.
Okay, let’s see if we can be a bit more specific about the content of dark data?
Items on the dark data ticket include: Email; Instant messages; documents; Sharepoint content; content of collaboration databases; ZIP files; log files; archived sensor and signal data; archived web content; aged audit trails; operational database backups – full and incremental; roll-back, redo and spooled data files; sunsetted applications (code and documentation); partially developed and then abandoned applications; and, code snippets.
Most importantly, dark data is data that is not actively in use, is underutilised, or is something else. Seriously.
So, the conclusion that some have come to is this: there is a vast collection of data in various formats waiting to be monetised.
Personally, the idea that really grabs my attention is the potential ability to do novel forensic research on email. If only to find out what happened in the past.
For example, maybe it would be fascinating to see how significant challenges were identified, flagged and discussed; how strategic responses to those challenges were formulated, chosen and executed; and, how the outcomes of all of that process were reflected in email communications.
I think that this line of work can be very interesting for some people, and that interesting insights may be uncovered, but I would hate to have to put a tangible value on it, if only to avoid adding to the already galactic magnitudes of nonsense and hype surrounding certain data topics.
There are other more mundane uses of dark data.
Imagine that you are just about to embark on a Data Warehouse project (you really are a late adopter aren’t you), and you want establish a base collection of historical data. Where do you get that historical data from?
Right! Operational databases are not characteristically used to store significant amounts of historical reference data and historical transactions beyond a certain time window; there are performance and other reasons for keeping OLTP systems as lean as possible, so, initial loads of historical data is typically recreated in the Data Warehouse from backups, audit trails or logs.
You don’t need a Chief Data Officer in order to be able to catalogue all your data assets. However, it is still good idea to have a reliable inventory of all your business data, including the euphemistically termed Big Data and dark data.
If you have such an inventory, you will know:
What you have, where it is, where it came from, what it is used in, what qualitative or quantitative value it may have, and how it relates to other data (including metadata) and the business.
What needs to be kept, and for how long, and what can be safely discarded, and when.
The risks associated with the retention or loss of that data.
If you don’t have such a catalogue and have never done a data inventory then a full data inventory and audit seems to be your new best friend.
Simply stated, you may have dark data that has value, or it may be a simple collection of worthless digital nostalgia. But if you don’t know what you have, it may pay to find out what’s there, and if necessary, to let it go.
There is no point in hoarding unneeded and unwanted rubbish data. That is simply not good data management.
Finally a word on all the fuss surrounding dark data.
Failure to monetize when there is value to be obtained from dark data is one thing, claiming that value can be invariably obtained whilst actually not knowing what the data is, or how it could be monetised, is just adding to the mountain of data related ‘nonsense and hype’ doing the rounds these days. Please consider not adding to that mountain.
British Rail, the national UK rail Company, used to be notorious for the number of delays and cancellations to services, and their reasons for failing to meet their obligations became stranger and stranger.
In winter, it would snow and there would be problems. And people would ask ‘how come you couldn’t deal with the snow this year, we’ve had snow for centuries?’ And back came the answers ‘Yes, Sir, but this year it was the wrong type of snow’. In autumn (the fall), it was ‘the wrong types of leaves, and ‘the wrong type of rain’, and in Summer, the ‘wrong type of sunshine’ and so on and so forth.
I hope this will not be the excuse from the Big Data and dark data pundits and punters when the much-vaunted and ‘almost’ guaranteed monetisation isn’t frequently realised.
‘Of course Big Data gives you big dollar benefits, it was just littered with the wrong type of data’ or ‘you just weren’t trying hard enough’.
Many thanks for reading.
06 Friday Mar 2015
Posted in Big Data, Consider this, Data Warehousing, Good Strat, hadoop, hdfs, Martyn Jones
Tags
Big Data, enterprise data warehousing, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Jones
Hold this thought: To paraphrase the great Bob Hoffman, just when you think that if the Big Data babblers were to generate one more ounce of bull**** the entire f****** solar system would explode, what do they do? Exceed expectations.
I am a mild mannered person, but if there is one thing that irks me, it is when I hear variations on the theme of “Data Warehousing is Big Data”, “Big data is in many ways an evolution of data warehousing” and “with Big Data you no longer need a Data Warehouse”.
Big Data is not Data Warehousing, it is not the evolution of Data Warehousing and it is not a sensible and coherent alternative to Data Warehousing. No matter what certain vendors will put in their marketing brochures or stick up their noses.
In spite of all of the high-visibility screw-ups that have carried the name of Data Warehousing, even when they were not Data Warehouse projects at all, the definition, strategy, benefits and success stories of data warehousing are known, they are in the public domain and they are tangible.
Data Warehousing is a practical, rational and coherent way of providing information needed for strategic and tactical option-formulation and decision-making.
Data Warehousing is a strategy driven, business oriented and technology based business process.
We stock Data Warehouses with data that, in one way or another, comes from internal and optional external sources, and from structured and optional unstructured data. The process of getting data from a data source to the target Data Warehouse, involves extraction, scrubbing, transformation and loading, ETL for short.
Data Warehousing’s defining characteristics are:
Subject Oriented: Operational databases, such as order processing and payroll databases and ERP databases, are organized around business processes or functional areas. These databases grew out of the applications they served. Thus, the data was relative to the order processing application or the payroll application. Data on a particular subject, such as products or employees, was maintained separately (and usually inconsistently) in a number of different databases. In contrast, a data warehouse is organized around subjects. This subject orientation presents the data in a much easier-to-understand format for end users and non-IT business analysts.
Integrated: Integration of data within a warehouse is accomplished by making the data consistent in format, naming and other aspects. Operational databases, for historic reasons, often have major inconsistencies in data representation. For example, a set of operational databases may represent “male” and “female” by using codes such as “m” and “f”, by “1” and “2”, or by “b” and “g”. Often, the inconsistencies are more complex and subtle. In a Data Warehouse, on the other hand, data is always maintained in a consistent fashion.
Time Variant: Data warehouses are time variant in the sense that they maintain both historical and (nearly) current data. Operational databases, in contrast, contain only the most current, up-to-date data values. Furthermore, they generally maintain this information for no more than a year (and often much less). In contrast, data warehouses contain data that is generally loaded from the operational databases daily, weekly, or monthly, which is then typically maintained for a period of 3 to 10 years. This is a major difference between the two types of environments.
Historical information is of high importance to decision makers, who often want to understand trends and relationships between data. For example, the product manager for a Liquefied Natural Gas soda drink may want to see the relationship between coupon promotions and sales. This is information that is almost impossible – and certainly in most cases not cost effective – to determine with an operational database.
Non-Volatile: Non-volatility means that after the data warehouse is loaded there are no changes, inserts, or deletes performed against the informational database. The Data Warehouse is, of course, first loaded with cleaned, integrated and transformed data that originated in the operational databases.
We build Data Warehouses iteratively, a piece or two at a time, and each iteration is primarily a result of business requirements, and not technological considerations.
Each iteration of a Data Warehouse is well bound and understood – small enough to be deliverable in a short iteration, and large enough to be significant.
Conversely, Big Data is characterised as being about:
Massive volumes: so great are they that mainstream relational products and technologies such as Oracle, DB2 and Teradata just can’t hack it, and
High variety: not only structured data, but also the whole range of digital data, and
High velocity: the speed at which data is generated, transmitted and received.
These are known as the three Vs of Big Data, and they are subject to significant and debilitating contradictions, even amongst the gurus of Big Data (as I have commented elsewhere: Contradictions of Big Data).
From time to time, Big Data pundits slam Data Warehousing for not being able to cope with the Big Data type hacking that they are apparently used to carrying out, but this is a mistake of those who fail to recognise a false Data Warehouse when they see one.
So let’s call these false flag Data Warehouse projects something else, such as Data Doghouses.
“Data Doghouse, meet Pig Data.”
Failed or failing Data Doghouses fail for the same reasons that Big Data projects will frequently fail. Both will almost invariably fail to deliver artefacts on time and to expectations; there will be failures to deliver value or even simply to return a break even in costs versus benefits; and of course, there will be failures to deliver any recognisable insight.
Failure happens in Data Doghousing (and quite possibly in Big Data as well) because there is a lack of coherent and cohesive arguments for embarking on such endeavours in the first place; a lack of real business drivers; and, a lack of sense and sensibility.
There is also a willing tendency to ignore the advice of people who warn against joining in the Big Data hubris. Why do some many ignore the ulterior motives of interested parties who are solely engaged in riding on the faddish Big Data bandwagon to maximise the revenue they can milk off punters? Why do we entertain pundits and charlatans who ‘big up’ Big Data whilst simultaneously cultivating an ignorance of data architecture, data management and business realities?
Some people say that the main difference between Big Data and Data Warehousing is that Big Data is technology, and Data Warehousing is architecture.
Now, whilst I totally respect the views of the father of Data Warehousing himself, I also think that he was being far too kind to the Big Data technology camp. However, of course, that is Bill’s choice.
Let me put it this way, if Oracle gave me the code for Oracle 3, I could add 256 bit support, parallel processing and give it an interface makeover, and it would be 1000 times better than any Big Data technology currently in the market (and that version of Oracle is from about 1983).
Therefore, Data Warehousing has no serious competing paragon. Data Warehousing is a real architecture, it has real process methodologies, it is tried and proven, it has success stories that are no secrets, and these stories include details of data, applications and the names of the companies and people involved, and we can point at tangible benefits realised. It’s clear, it’s simple and it’s transparent.
Just like Big Data, right?
Well, no.
See what I mean?
Therefore, the next time someone says to you that Big Data will replace Data Warehousing or that Data Warehousing is Big Data, or any variations on that sort of ‘stupidity’ theme, you can now tell them to take a hike, in the confidence that you are on the side of reason.
Many thanks for reading.
Aligning Big Data: http://www.linkedin.com/pulse/aligning-big-data-martyn-jones
Big Data and the Analytics Data Store: http://www.linkedin.com/pulse/big-data-analytics-store-martyn-jones
A Modern Manager’s Guide to Big Data:http://www.linkedin.com/pulse/managers-guide-big-data-context-martyn-jones
Accomodating Big Data
GOOD STRATEGY 