02 Sunday Jul 2017
Posted in 4th generation Data Warehousing, All Data, Big Data, Big Data Analytics, dark data, data architecture, Data governance, Data Lake, data management, Data Mart, data science, Data Supply Framework, Data Warehouse, Data Warehousing, pig data, The Amazing Big Data Challenge, The Big Data Contrarians
Afilonius Rex
Dunny on the Wold, 1st July 2017
The 4th big data bullshit is here, and it is completely reimagining the way we conceptualise, think and reason. This amazingly and imaginary Govesque world is driven by hyperbole and mendacity distributed via interconnected digital devices that are capable of amassing and fermenting ever-growing amounts of big data bullshit.
16 Sunday Aug 2015
Posted in Big Data, Consider this, good start, goodstart, Strategy
Tags
Big Data, cynicism, data management, fakes, good start, goodstart, gurus, Martyn Jones, Martyn Richard Jones, Strategy
Why so many ‘fake’ Big Data Gurus?
Where do you all come from?
Where do you all come from?
All your integrity’s gone
Now tell me, where do you all come from?
From ‘Where Do You All Come From‘ by Mott the Hoople
Readers should be well aware that the comedian who wrote this piece is the self-styled founder of The Big Data Contrarians, which is quite possibly the most belligerently intelligent Big Data group you will ever come across in your entire life. You have been warned. If you need to verify these facts for yourself, then take a look here at your own risk: https://www.linkedin.com/grp/home?gid=8338976
You may have noticed the massive relative-growth in the number of people who are describing themselves as Big Data gurus, data science Kaisers or analytics evangelists. Okay, I exaggerate to evidence the trend, but you’ll hopefully get the gist. Your fellow comrade on the picket line, that sweetie you met at the Pitt Club — even your darling masseuse has had their carte de visite transformed according to the prevailing désir de Jour.
Many people out there in the big data world suddenly call themselves ‘Big Data gurus’ simply because it is the latest vogue. The Caerfyrddin Good Pub Guide even went so far as to say that, “adding Big Data to your job title was the equivalent of sexing up a dodgy dossier”. They also later suggested that boosting your resume with the judicious incorporation of a titillating title, such as Big Data analytic pole-dancer, may get you a few chuckles, even if most people don’t understand and appreciate its broad multi-faceted and humoristic ramifications.
That stated, the cruel and harsh reality is that many who call themselves Big Data gurus appear to be lacking the full Big Data picnic by quite a few sandwiches when it comes down to looking at the nitty gritty of their visible resumes. Moreover, and to be Frank and Earnest – Frank in Zurich and Earnest in Pontypridd – if I was hiring a top notch Big Data guru I wouldn’t even know where to start.
What I see are quite a number of courageous fellows who don’t really have a Scooby[1] (that’s a ‘clue’ for readers overseas) about Big Data, whose are enriched and swelled by others of their mind, who know just enough to be dangerous.
What I also see are Big Data hacks who cannot bring themselves to articulate one coherent, cohesive and verifiable Big Data success story. They are calling themselves Big Data gurus, presumably because of the incredible value accruable from their unrevealed bullion-class information.
The best offenders amongst the Big Data gurus being incredibly precocious with the hype and amazingly prudish with the facts.
Why, only the other day, one of these Big Data hype chappies – whose name escapes me for the moment – was lamenting the dearth of true data scientists whilst simultaneously lambasting and misrepresenting the profession of the contemporary statistician.
Now, I I am not fundamentally averse to a bit of hype, so long as it’s in moderation, and it doesn’t frighten the horses. Take my Dad, for example, he never curses, and he never has. Me, I use it for dramatic effect and emphasis – occasionally. However, a lot of the Big Data ‘hackery’ that we are entertained with is like having Big Data Derek and Clive playing in your ear, 24x7x52. It’s just too much, and most of the time it should be toned down or turned off. You know “So, this bloke comes up to me and says ‘Hello!’ ”
Then there are the people from the consultancies, the IT vendors and the service providers (alright, not all, just a few) who have a good grasp of the superficialities, the business, analytics and Big Data terminology, even if they have at best a tenuous grasp of the underlying structures, concepts and relationships that these terms relate to. Of course, it’s all hooked on context.
It’s not important if it’s just about a bit of a chit-chat down ‘The Black’ or a bit of banter over Kaffee und Kuchen, or a passing comment from the umpire at the crease. As Afilonius Basto put it to me “in a seriously professional business setting, many of the predominant Big Data gurus who rock the Big Data bull**** babble, just lack a certain creativity, knowledge and experience in order to act as truly reliable informers and trusted Big Data advisers.” So I ask you, who am I to argue such matters with a highly intelligent Gos D’atura?
Part of the problem here is, to borrow the unwritten style guide of Big Data ‘hackdom’, simple supply and demand voodoo and superfluous use of important and amorphous terms tagged on to such examples. To wit, the incongruous use of terms such as ‘economics’, ‘performance’ and ‘science’. To say nothing of the irritatingly banal use of ‘amazing’, ‘amazing’ and ‘amazing’. In addition, one thing that has been puzzling me is this. Why do some Big Data pundits have to overegg their verbosely literal output with US slang that even professional bloggers in the USA would tend to use very sparingly? I know, I know what you’re thinking. “Gerroutta heres”, right? But for me it is like the IT equivalent of the mock cockney accents that are used by some celebrity chefs. I kid you not! It’s awful and it’s going viral, just like the bubonic plague and Greensleeves.
Now back to the main story thread. There simply are not enough true Big Data gurus out there to fill the demand, and so barely qualified (or ‘make it up as you go along’) aspirants make it into the higher stratified stratosphere of the Big Data saloon.
Second, just like many Big Data success stories themselves, the role of a Big Data guru is often poorly demarcated within the ambit of association and relation and even indeed within a solitary business. People bandy around terms such as Big Data guru, Big Data whizz and Big Data who’s your daddy, willy-nilly postman style, These are terms that can mean everything and anything. From “he’s hot on Big Data that chap is” to “there goes thick Jack the Spratt the densest Big Data guru in Christendom”, and upwards and onwards to “did you see Marty, the Big Data party? Got Carmen Miranda from Data Analytics a leave of absence, although rumour has it that the she is down with predictive impregnation after reading a particularly sordid Big Data hype-piece on LinkedIn”.
A true Big Data guru/expert is so much more, so much more than what we have now. In my opinion, a Big Data guru/expert is about:
Facts. The Big Data guru should stick to the facts. For example, if a Big Data guru does not understand the roles and responsibilities of a statistician then they should keep mum, and be admired for their discretion, rather than opening up a floodgates of mental garbage and thereby inviting questions regarding credibility, and introducing the risk of being viewed as a buffoon. Exempli gratia, if you do not know what a statistician does, then ask, rather than simply making things up. Also ensure that you never get into a position of belief and thought that you did not rationalise yourself into. Because you will be stuck there and no one will be able to reason you out of that belief, because it’s a position that is not in itself based on reason, truth and logic.
As Malcolm X put it “Despite my firm convictions, I have always been a man who tries to face facts, and to accept the reality of life as new experience and new knowledge unfolds. I have always kept an open mind, a flexibility that must go hand in hand with every form of the intelligent search for truth.”
So, aspiring Big Data boys and girls? Please stick to the facts! Your Big Data god, friends or family will thank you for it.
Integrity. What Big Data definitely does not need is more hype-schlepping hypocrites, bamboozling babblers and conniving charlatans. What is needed are people who exude virtuous truthfulness, candour and pedagogical ethics.
According to Integrity Action, integrity is “the set of characteristics that justify trustworthiness and generate trust among stakeholders. Integrity creates the conditions for organisations to intelligently resist corruption and to be more trusted and efficient.” More broadly, Wikipedia puts it this way: ” Integrity is the quality of being honest and having strong moral principles; moral uprightness. It is generally a personal choice to uphold oneself to consistently moral and ethical standards.”
Trust. A Big Data guru/expert should be trustworthy, and seen to be trusty. Just like Caesar’s wife or the quality of brunch bar at Tiffany’s, the Big Data guru must be beyond reproach. Not that they aren’t allowed a little journalistic license, simply that the gaping abyss that separates barefaced porkies[2] from simple embellishments, is frankly enormous.
Knowledge. A Big Data guru/expert should be knowledgeable in all things data, and not just Big Data. Knowledge means you know what Data Warehousing is and don’t fib about it or grossly misrepresent it in order to score ill-gotten brownie points for Big Data babble. Knowledge means you know, not that you have a bit of an idea, know a friend of friend of a friend, or can ask the audience, in cases where one is caught-out, ill-advisedly pretending to know something one doesn’t know. My advice is this. Temper knowledge with humility, honesty and decency, and you won’t go far wrong.
Experience. A Big Data guru/expert must have walked the talk. Knowledge must go hand in hand with experience. Clearly a few self-labelled Big Data gurus doing the rounds these days do not fit the bill in this respect (or for that matter in many of the other ‘respects’). Alas, not all is lost. You too can acquire both the data knowledge and data experience to become a Big Data guru. How? Try working at it for a while.
There is another way at looking at experience, in a half-comic and half-serious way. To paraphrase a popular joke “Do not argue with an idiotic Big Data guru. He will drag you down to his level and beat you with experience.” And it happens…
Vocation. No, this is no time for a holiday. The Big Data guru must assume the mantle of Big Data stardom as a vocation and not just as an early-adopting fashion follower. As Voltaire put it, speaking of Newton but also commenting more broadly on education and the Enlightenment: “I have seen a professor of mathematics only because he was great in his vocation, buried like a king who had done well by his subjects.”
Simplicity. A Big Data guru/expert must be able to explain complex Big Data ideas in a simple ways, but without losing the essence or the credibility of what requires conveying. Tesco had a slogan, “you cannot bullshit simplicity”, which tried to convey this essence, and a German retailer took this even further with “Every Lidl helps”. So remember, keep it simple. “Sophistication is the ultimate sophistication” – Leonardo da Vinci.
Journalism. Although not necessarily a master of joined up handwriting and maths 101, a Big Data guru must also write like a journalist. He or she does not have to be a great scribe, simply being competent with words, concepts and numbers is a high enough bar. Okay, there is a little more to it than that – or at least, there should be. So I will explain.
These are a few things (also mentioned elsewhere in this piece, and influenced by the World Journalism Institute), that a good Big-Data-guru journalist should possess or be aware of:
A good story. According to the Writers Store there are seven elements that make a god story: ” the change of fortune, the problem of the story, the complications, crisis, climax and resolution of the classical structure, and the threat, which is by far the most important.” A truly great Big Data guru will be able to take this advice and apply it to the field of Big Data with little difficulty. It would make a great change from wading through Data Lakes of ‘bleh’ in search of the Holy Big Data grail. So, having set the scene, let’s take a closer look.
A real Big Data guru must be able to tell a compelling and credible story. However, they must also show as well as tell. Telling alone is fiction, which is fine, but a Big Data needs to back great fiction up with fact. A Big Data guru worth their salt must be able to tell and show. Nothing less than a verifiable story is acceptable. Don’t be promiscuous with the facts in a story, especially when one heralds a Big Data success, without providing any hard evidence to back it up. Remember this, just because some people are incredibly gullible when it comes to Big Data, it doesn’t mean you should lead them in ignorance down the garden path, like so many innocent lambs to the slaughter. That sort of thing is quite despicable even by our regular standards. So don’t go out of your way to prove Dan Ariely right yet again, especially with regards to his very accurate comment that “Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it…”
Technology. A Big Data guru should know the technology. They should also know the origins of the technology and its influences. For example, a Big Data guru should have a sound grasp of the principles of the following:
This is far from being an exhaustive list, but it should give you a flavour for what is required.
Also, take heed of the words of Pablo Picasso, who stated, “Computers are useless. They can only give you answers.”
Agnostic. An ethical and professional Big Data pundit that really rocks the Kasbah, must also be as agnostic as it is possible to be, yet without letting an insistence on ‘fair’ and ‘balance’ upset the civic order of things. This simply means that you do not have to give equal merit to all and sundry, especially when equal merit is palpably not inherently the case in the broad range of technical, project and business offerings that wash over the decks of the SS Big Data. At the end of the day, in all cases the postmodern interpretation of fair and balanced is also a massive contradiction in terms.
The Swiss philosopher, poet and critic Henri Frédéric Amiel once wrote that “A belief is not true because it is useful”, and this is we should take special care in taking an agnostic stance with regards to Big Data and its technologies. There is a tendency for Big Data gurus to big up Hadoop and ignore the rest of the field. Not only is this narrow-minded view an injustice, but it is ultimately detrimental to those who seek to understand and obtain benefit from deploying Big Data solutions.
Versatile. I have seen many people try to accommodate the meagre competence of some self-anointed Big Data gurus within a far too wide an area of acceptance. This is not an issue if people are aware of the rampant bias, babble and boloney factors, but nonetheless extreme caution needs to be exercised if awkward unintended consequences are to be avoided. Remember a good Big Data guru can come from a variety of backgrounds — and not all of them will necessarily require a degree from a prestigious centre of educational excellence such as Oxford, the LSE or Prifysgol Cymru, Y Drindod Dewi Sant. In fact, one could say that attendance at St Trinian’s School for Young Ladies, Hogwarts School of Witchcraft and Wizardry, or Cambridge Finishing School could quite possibly be a negative factor, although I would not associate a certainty factor with any of these bets. Although to be fair, I am reliably informed by the Headmistress that Saint Trinian has an excellent Data Science faculty.
Business understanding. The ideal Big Data guru must be business aware, savvy to the point of shrewdness, and cunning, preferably “as cunning as a fox who’s just been appointed Professor of Cunning at Oxford University”[3]. They must understand business process, the commonalities and differences of business sectors and players, and the motivations, competitive forces and key influencers in business. They must also understand the meaning of business-strategy, how it is developed, chosen and executed. Finally, the Big Data guru must have a good handle on irrationality and its degrees of predictability. So, know your business, and understand the business of others. So when someone gives you a bucket and tells you to go down to Tesco’s to buy a petabyte of data and a Euro Millions Lottery ticket, you’ll know what it’s all about.
Analytical. A good Big Data guru must be naturally analytical, but not to the point of being anally analytical, and should possess an ability to spot patterns in behaviour as well as in data. CIA veteran Dick Heuer put it this way: “Thinking analytically is a skill like carpentry or driving a car. It can be taught, it can be learned, and it can improve with practice. But like many other skills, such as riding a bike, it is not learned by sitting in a classroom and being told how to do it. Analysts learn by doing.”
If you encounter a candidate Big Data guru with all of these traits — or have a candidate who ticks most of the boxes but is willing to acquire more ticks — then you’ve found someone who might deliver unbelievable value to the cause of Big Data, your struggle, your reason, your content, and your living-room. So, delight in your find.
However, be sparing with candidates on any of these inherent individualities, and you run the risk of acquiring a graded and coarse-grained pretender, someone just hoping to travel the Big Data bullshit babbling bubble until it bursts in their brazen time-pieces[4].
I know, I know, I hear you say “but does this all really matter?” Probably not. Probably in the grand scheme of things this is yet another boom and bust fad destined to become matter of fact in some reduced circles and a plain waste of time, money and patience, in others. No doubt we shall see, as the story unfolds and the ‘sublimely absurd’ metamorphoses into the ‘I don’t bloody believe it’, or not.
So, now over to you. What would you add to this compendium of convenient characteristics? I would really love to receive your views, opinions and perspectives in the comments section that follows on from this piece.
Many thanks for reading.
A note from the Prime Minister:
Data is only as good as its time and place utility. If it has none, it has no present value, unless of course, someone wants to pay something for nothing, but that is constructing a con not an economy, an aberration destined to be hated and then forgotten. Don’t only think about how to use the data you have, but also about what data should be captured and how it should be used. By the way, join the Big Data contrarians here on LinkedIn: https://www.linkedin.com/grp/home?gid=8338976
Vote for the Big Data Contrarians. Vote early! Vote often!
[1] Scooby, Scooby doo, rhyming slang for ‘clue’
[2] Porkies, pork pies, lies
[3] Blackadder Goes Forth, Richard Curtis and Ben Elton
[4] Clock, face.
01 Friday May 2015
Posted in Big Data, Consider this, good start, goodstart
Tags
Big Data, Consider this, data architecture, data management, good start, goodstart, Martyn Richard Jones
Miss Piggy said, “Never eat more than you can lift”. That statement is no less true today, especially when it comes to Big Data.
The biggest disadvantage of Big Data is that there is so much of it, and one of the biggest problems with Big Data is that few people can agree on what it is. Overcoming the disadvantage of size is possible; overcoming the problem of understanding may take some time.
As I mentioned in my piece Taming Big Data, “the best application of Big Data is in systems and methods that will significantly reduce the data footprint.” In that piece I also outlined three conclusions:
“Data Governors”, I hear you ask, “What are Data Governors?”
Let me address that question.
Simply stated, the Data Governor approach to Big Data obtuseness is this:
In Short, it is a comprehensive approach to reducing the Big Data footprint whilst simultaneously maintaining data fidelity.
Here are some examples:
What’s this all about? Here’s an answer the good folk at Wikipedia cooked up earlier:
“Wafer testing is a step performed during semiconductor device fabrication. During this step, performed before a wafer is sent to die preparation, all individual integrated circuits that are present on the wafer are tested for functional defects by applying special test patterns to them. The wafer testing is performed by a piece of test equipment called a wafer prober. The process of wafer testing can be referred to in several ways: Wafer Final Test (WFT), Electronic Die Sort (EDS) and Circuit Probe (CP) are probably the most common.” (Link / Wikipedia)
Fig.1 – IC Fab Testing and the CE Data Governor
This exhibit shows where the Data Governor is placed in the Integration Circuit fabrication and testing/probing chain.
In large plants, the IC probing process generates very large volumes of data at high velocity rates.
Based on exception rules the Data Governor reduces the flow of data to the centralised data store.
It also speeds up velocity and time to analysis.
Greater speed and less volume mean that production showstoppers are spotted earlier, thereby potentially leading to significant savings in production and recuperation costs.
Let’s look at some of the technical details:
Intrinsically linked to Big Data and Big Data Analytics, the Internet of Things (IoT) is described as follows:
“The Internet of Things (IoT) is the network of physical objects or “things” embedded with electronics, software, sensors and connectivity to enable it to achieve greater value and service by exchanging data with the manufacturer, operator and/or other connected devices. Each thing is uniquely identifiable through its embedded computing system but is able to interoperate within the existing Internet infrastructure.” (Link / Wikipedia)
Fig.2 – The Internet of Things and the CE Data Governor
This exhibit shows where the Data Governor is placed in the Internet of Things data flow.
The Data Governor is embedded into an IoT device, and functions as a data exception engine.
Based on exception rules and triggers the Data Governor reduces the flow of data to the centralised / regionalised data store.
It also speeds up velocity and time to analysis.
Greater speed and less volume means that important signals are spotted earlier, thereby quite possibly leading to more effective analysis and quicker time to action.
Much play is made of the possibility that we will all be extracting golden nuggets from web server logs sometime in the near future. I don’t want to get into the business value argument here, but would like to describe a way of getting Big Data to shed the excess web-server-log bloat.
Fig.3 – Web Server Activity Logging and the CE Data Governor
This exhibit shows where the Data Governor is placed in the capture and logging of interactive internet activity.
The Data Governor acts as a virtual device written to by standard and customised log writers, and functions as a data exception engine.
Based on exception rules and triggers the Data Governor reduces the flow of data from internet activity logging.
It also speeds up velocity and time to analysis.
Greater speed and significantly reduced data volumes may lead to more effective and focused analysis and quicker time to action.
Signal data can be a continuous stream of data originating from devices such as temperature and proximity sensors, by its nature, it can generate high-volumes of data and at high velocity – it can add lots of data, and very quickly.
Fig.4 – Signal Data and the CE Data Governor
This exhibit shows where the Data Governor is placed in the stream of continuous signal data.
The Data Governor acts as an in-line data-exception engine.
Based on exception rules and triggers the Data Governor reduces the flow of signal data.
It also speeds up velocity and time to analysis.
Greater speed and significantly reduced data volumes may lead to more effective and focused analysis and quicker time to action.
“Machine-generated data is information which was automatically created from a computer process, application, or other machine without the intervention of a human.” (Link / Wikipedia)
Fig.5 – Machine Data and the CE Data Governor
This exhibit shows where the Data Governor is placed in the stream of continuous machine generated data.
The Data Governor acts as an in-line data analysis and exception engine.
Exception data is stored locally and periodically transferred to an analysis centre.
Analysis of the totality of the same class and origins of data can be used to drive ANN* and statistical analysis which can be used to support (for example) the automatic and semi-automatic generation of preventive maintenance rules.
Greater speed and significantly reduced data volumes may lead to more effective and focused analysis and quicker time to proactivity.
The options are not endless and the prizes are not rich beyond the dreams of avarice, but there are some exciting possibilities out there. Including applications in the trading; plant monitoring; sport; and, climate change ‘spaces’.
Fig.6 – Other Applications in the Big Data ‘space’ and the CE Data Governor
To wrap up, this is what the CE Data Governor approach looks like at a high level of abstraction:
To summarise the drivers:
Moreover, we have a set of clear and justifiable objectives:
Many thanks for reading.
Can we help? Leave a comment below. contact me via LinkedIn or write to martyn.jones@cambriano.es
20 Friday Mar 2015
Posted in Big Data, Consider this, good start, goodstart
Tags
Big Data, contradictions, data management, good start, Good Strategy, goodstart, Martyn Jones, Martyn Richard Jones
I have written at length about the fundamental contradictions of Big Data, but what I have omitted in the past is quite possibly the biggest contradiction of all. Probably because it has more to do with how Big Data is continually hyped, rather than having anything to do with Big Data as a bag of technologies – which has a whole assortment of problems in its own right.
Last time I spoke with you about the contradictions of the Big Data it was about the three Vs of volume, variety and velocity. In general, it was a view that was well received, even if not widely understood. Which of course is close enough for government work. But, get ready for “something completely different”.
If on the one hand some folk can claim that Big Data provides fact based insights and reliable forecasts of future habits, trends and preferences, then why is it so difficult to produce and socialise – yes, I like to use that term – Big Data success stories?
In short, I think we have arrived at the stage in Big Data’s cycle where it is reasonable to ask pundits to either put up or shut up.
So, why aren’t the current Big Data success fables accompanied by facts, such as names of those involved (at least businesses), the sponsors, the suppliers, the purpose of the exercise, the desired outcomes, the data used, how it is processed, what the results were, and what tangible benefits, if any, were accrued or are accruable. If that is not enough, then let people mention the technology used, the products purchased or licensed and the methodology followed.
In short, what I would like to know is why are the evangelists of Big Data telling us that bigger data is better, that more variety leads to greater insight, and that velocity is king. Why do we we told that Big Data almost assuredly results in better decisions, by people who are coy, shy or secretive about almost facts and data coming out of Big Data projects?
I have been reminded, time and time again, that there are Big Data success stories out there, and I have even been told that this information would be fully shared with me once it was agreed with the ‘clients’ that it was okay to do so. Okay, that’s fine, I know Big Data is a roaring success story (at least in people’s minds,) and I also know that it takes some time to make things up – some people are just not creative. Sure, I was told about these ‘successes’ some time ago, and you know, I’m not expecting anything that’s worth shaking a stick at, either now or later, but I’m still waiting, boys. Notwithstanding, you will still called you out as vacuous bullshitters when the time comes.
“But” I hear you cry “there is a wealth of success stories in the presses”.
Well, no, and you would wrong and gullible and foolish to think, but that is your problem, but unfortunately also mine, because this is my profession that you are playing fast and loose with.
The fact is that there is “wealth” of content that people try and pass off as legitimate Big Data success stories, but they aren’t in fact success stories, in any way, shape or form.
The thing is, people may read the blog title and even the stand-fast, but will be less inclined to actually read the article, so what remains is the impression that there are ‘loads of Big Data success stories’. But if people actually read the articles and were intelligent enough to understand them, then they would realise that inevitably there is a massive mismatch between the title of these pieces and the content. Indeed, if these pieces were actually pieces of advertising, rather than blog comments, they would be denounced in some jurisdictions for not fulfilling the advertising criteria of legal, decent and honest.
There is one more thing that Big Data evangelists (or any self-styled pundit, guru or expert for that matter) should understand, internalise and remember. If you say that you have a Big Data success story, with all the details, and that isn’t in fact the case, and it isn’t even remotely a success story or even true, then you are simply lying, and that’s deceit, it’s unprofessional and it’s unethical, and you are a scoundrel. So live with or fix it, the choice is yours.
Many thanks for reading.
20 Friday Mar 2015
Posted in Big Data, Consider this, data management, good start, goodstart
Tags
Big Data, data architecture, data management, good start, Good Strat, Good Strategy, goodstart, Martyn Jones, Martyn Richard Jones
Many people come up to me in the street and ask me what Big Data is all about. It has happened to me so many times in the past that I am convinced that it might just happen to you as well. I know sort of thing, I read the Big Data tealeaves. Nothing gets past me.
The first time a complete stranger came up to me in public and said “Hello, will you tell me what this Big Data lark is all about then?” I was lost for words, you just ask my Aunt Dolly, he can vouch for that, no problem. Later that day I read a book – it was my dad’s book – and I then decided to adopt a strategy.
Therefore, in the spirit of springtime goodwill to all men and women, I have put together this blog piece in that hope that it will enlighten, help and entertain.
What is big data?
Big Data can be characterised by the 10 Vs – yes, 10, not 4. Which, in my book, is more than enough to bring up-to-speed the average Big Data John or Jane that one meets on the street, and who naturally wish to be informed of such matters.
In layperson’s terms this a series of landmarks and pointers in the analytics space used to frame and guide the didactic aspects of Big Data.
The fundamental Vs of the Big Data canon are these:
So, let me now explain what each of these characteristics mean to those who might know and for those who might want to know.
Vagueness: This is perhaps the trickiest of questions to address, given the vast panorama that is cast before this incredibly complex yet easily graspable concept. But let me state this, and let there be no mistake about it. At this point in time, what makes Big Data vague is also what makes Big Data specific, explicit and certain. That is to say, in order to ‘come to an understanding’ of Big Data, it is necessary to completely embrace the dialectic of knowing the unknowable. So belief is an absolute essential element – belief and data, that is.
Volume – If there ever was a time to “pump up the volume”, we have it here with Big Data.
Big, voluminous, gorgeously rotund and infinite. Big Data is called Big Data because there is a lovely, roly-poly, likeable never-ending load of it. Its volumes can be measured in zeta-bytes, which you can be assured, is a helluva lot of data.
Variety – As they might say down my way, “variety is the spice of life, innit”. This is what makes Big Data so special. So appealing.
Because before Big Data there was absolutely no variety in anything, at all. We lived in a bland world, bereft of detail, nuance and diversity. Nothing could be measured, analysed or explained, because we lacked Big Data. We were ignorant. So ignorant and stupid that we couldn’t see the sense of putting the diapers next to the beer, or of offering three for the price of two.
Fortunately, today this is no longer the case if we don’t want it to be, and thanks to Big Data we have a veritable sensorial explosion. No longer is IT just a couple of symbols scribbled in crayon on someone’s school notebook.
Virility – Move over Smart Data, the new kid on the block is Big Data.
If Big Data were described in the manner of a religious text, it would be accompanied by a never ending narrative of begets.
So, what does that mean?
Simply stated, Big Data creates itself, in and of itself. The more Big Data you have, the more Big Data gets created. It’s like a self-fulfilling prophecy in 360 degree, high-definition, poly-faceted and all-encompassing knowing. The sort of thing that governments would pay an arm and a leg to get their mitts on.
Velocity – Velocity is of the essence. Velocity kills the competition. More velocity, less haste.
We demand that service is ‘velocious’. ‘Everything’ must be ‘now’, or it’s too late.
This means we need to be able to handle Big Data at velocity – at the speed of need.
Charles Babbage once stated (or maybe it was more than once) that “whenever the work is itself light, it becomes necessary, in order to economize time, to increase the velocity.”
But remember, we are dealing with mega-velocity here, so don’t drink and drive the Big Data Steamship, Star-ship or Mustang.
Vendible – If you can sell it, and sell it as Big Data, then it ‘is’ Big Data. If you can’t, then it’s not. The saleability of Big Data proves its existence.
So, what are the vendible aspects of Big Data?
Let’s leave that easy question for another day. But for now I can confidently state that it is used to mobilise armies of commentators, industry analysts, publicists, punters, writers, bloggers, gurus, futurologists, conference organisers, conference speakers, educators, customer relationship managers, salespeople, marketers and admen.
Vaticination – Edmund Burke is down on record as stating that “you can never plan the future by the past”. Now Burke may have been a clever person when it came to many things, but he wasn’t exactly a whiz when it came to Big Data.
There are people in the world who are in no doubt that Big Data provides the sort of visionary and predictive powers only previously obtainable through ritual sacrifice, magic potions and the casting of spells. Others are highly critical of the understatement implicit in this belief.
For many, Big Data will make the Oracle of Delphi look like a mere call centre.
This is why the power of vaticination plays a characteristically important role in the world of Big Data.
Voracity – This is based on the quasi-rationalist argument that Big Data is big and it has an omnipresent and insatiable self-fulfilling desire.
Big Data comes with an attendant requirement for hardware, even if it is a whole load of consumer hardware tacked together in a magnificent and miraculous mesh of magic.
Big Data can be characterised by voracity, but this comes hand in hand with the ‘ventripotent’ IT industry.
Veracity – The eminence of the data being captured for Big Data handling can vary significantly. The quality or lack of quality of the data naturally has the potential to impact the accuracy of analysis using that data.
Before Big Data arrived on the scene we knew nothing about Data Quality or data verification. This is why ETL and Data Cleansing tools lacked the power to effectively quality check and verify data, to ensure that any erroneous or anomalous data was rejected or flagged.
But now, with the sophistication of tools such as ‘grep’ and ‘awk’ at our disposal, we have the power in our hands to ensure nothing ‘dodgy’ gets into the analytical mix.
Vanity – In my opinion, to fully grasp the underlying and profound meaning of Big Data, it is essential for us to understand the difference between vanity and conceit. Max Counsell claimed that “Vanity is the flatterer of the soul”. Goethe characterised vanity as being “a desire for personal glory”. After an incident with an Anarchist (presumably a Big Data Anarchist), Blackadder remarked to Baldrick that “The criminal’s vanity always makes them make one tiny but fatal mistake. Theirs was to have their entire conspiracy printed and published in plain manuscript”.
So that ends the brief rundown of the defining characteristics of Big Data.
So, to summarise. That, which has passed before, necessarily divulges both the upside and downside of Big Data. By reaching out, opening up the kimono and relating the 10 Vs we are disclosing that which cannot be disclosed, exhibiting the absence of essential essence, and thereby opening up the entire field, discipline, profession, science and art to examination, questioning and ridicule.
Many thanks for reading.
10 Tuesday Mar 2015
Posted in All Data, Big Data, Consider this, dark data, Good Strat
Tags
All Data, Big Data, dark data, data architecture, data management, Good Strat, Martyn Jones, Martyn Richard Jones
Dark data, what is it and why all the fuss?
First, I’ll give you the short answer. The right dark data, just like its brother right Big Data, can be monetised – honest, guv! There’s loadsa money to be made from dark data by ‘them that want to’, and as value propositions go, seriously, what could be more attractive?
Let’s take a look at the market.
Gartner defines dark data as “the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes” (IT Glossary – Gartner)
Techopedia describes dark data as being data that is “found in log files and data archives stored within large enterprise class data storage locations. It includes all data objects and types that have yet to be analyzed for any business or competitive intelligence or aid in business decision making.” (Techopedia – Cory Jannsen)
Cory also wrote that “IDC, a research firm, stated that up to 90 percent of big data is dark data.”
In an interesting whitepaper from C2C Systems it was noted that “PST files and ZIP files account for nearly 90% of dark data by IDC Estimates.” and that dark data is “Very simply, all those bits and pieces of data floating around in your environment that aren’t fully accounted for:” (Dark Data, Dark Email – C2C Systems)
Elsewhere, Charles Fiori defined dark data as “data whose existence is either unknown to a firm, known but inaccessible, too costly to access or inaccessible because of compliance concerns.” (Shedding Light on Dark Data – Michael Shashoua)
Not quite the last insight, but in a piece published by Datameer, John Nicholson wrote that “Research firm IDC estimates that 90 percent of digital data is dark.” And went on to state that “This dark data may come in the form of machine or sensor logs” (Shine Light on Dark Data – Joe Nicholson via Datameer)
Finally, Lug Bergman of NGDATA wrote this in a sponsored piece in Wired: “It” – dark data – “is different for each organization, but it is essentially data that is not being used to get a 360 degree view of a customer.
Okay, let’s see if we can be a bit more specific about the content of dark data?
Items on the dark data ticket include: Email; Instant messages; documents; Sharepoint content; content of collaboration databases; ZIP files; log files; archived sensor and signal data; archived web content; aged audit trails; operational database backups – full and incremental; roll-back, redo and spooled data files; sunsetted applications (code and documentation); partially developed and then abandoned applications; and, code snippets.
Most importantly, dark data is data that is not actively in use, is underutilised, or is something else. Seriously.
So, the conclusion that some have come to is this: there is a vast collection of data in various formats waiting to be monetised.
Personally, the idea that really grabs my attention is the potential ability to do novel forensic research on email. If only to find out what happened in the past.
For example, maybe it would be fascinating to see how significant challenges were identified, flagged and discussed; how strategic responses to those challenges were formulated, chosen and executed; and, how the outcomes of all of that process were reflected in email communications.
I think that this line of work can be very interesting for some people, and that interesting insights may be uncovered, but I would hate to have to put a tangible value on it, if only to avoid adding to the already galactic magnitudes of nonsense and hype surrounding certain data topics.
There are other more mundane uses of dark data.
Imagine that you are just about to embark on a Data Warehouse project (you really are a late adopter aren’t you), and you want establish a base collection of historical data. Where do you get that historical data from?
Right! Operational databases are not characteristically used to store significant amounts of historical reference data and historical transactions beyond a certain time window; there are performance and other reasons for keeping OLTP systems as lean as possible, so, initial loads of historical data is typically recreated in the Data Warehouse from backups, audit trails or logs.
You don’t need a Chief Data Officer in order to be able to catalogue all your data assets. However, it is still good idea to have a reliable inventory of all your business data, including the euphemistically termed Big Data and dark data.
If you have such an inventory, you will know:
What you have, where it is, where it came from, what it is used in, what qualitative or quantitative value it may have, and how it relates to other data (including metadata) and the business.
What needs to be kept, and for how long, and what can be safely discarded, and when.
The risks associated with the retention or loss of that data.
If you don’t have such a catalogue and have never done a data inventory then a full data inventory and audit seems to be your new best friend.
Simply stated, you may have dark data that has value, or it may be a simple collection of worthless digital nostalgia. But if you don’t know what you have, it may pay to find out what’s there, and if necessary, to let it go.
There is no point in hoarding unneeded and unwanted rubbish data. That is simply not good data management.
Finally a word on all the fuss surrounding dark data.
Failure to monetize when there is value to be obtained from dark data is one thing, claiming that value can be invariably obtained whilst actually not knowing what the data is, or how it could be monetised, is just adding to the mountain of data related ‘nonsense and hype’ doing the rounds these days. Please consider not adding to that mountain.
British Rail, the national UK rail Company, used to be notorious for the number of delays and cancellations to services, and their reasons for failing to meet their obligations became stranger and stranger.
In winter, it would snow and there would be problems. And people would ask ‘how come you couldn’t deal with the snow this year, we’ve had snow for centuries?’ And back came the answers ‘Yes, Sir, but this year it was the wrong type of snow’. In autumn (the fall), it was ‘the wrong types of leaves, and ‘the wrong type of rain’, and in Summer, the ‘wrong type of sunshine’ and so on and so forth.
I hope this will not be the excuse from the Big Data and dark data pundits and punters when the much-vaunted and ‘almost’ guaranteed monetisation isn’t frequently realised.
‘Of course Big Data gives you big dollar benefits, it was just littered with the wrong type of data’ or ‘you just weren’t trying hard enough’.
Many thanks for reading.
01 Sunday Mar 2015
Posted in Ask Martyn, Big Data, Consider this
We’ve been told that Big Data is the greatest thing since sliced bread, and that its major characteristics are massive volumes (so great are they that mainstream relational products and technologies such as Oracle, DB2 and Teradata just can’t hack it), high variety (not only structured data, but also the whole range of digital data), and high velocity (the speed at which data is generated and transmitted). Also, from time to time, much to the chagrin of some Big Data disciples, a whole slew of new identifying Vs are produced, touted and then dismissed (check out my LinkedIn Pulse article on Big Data and the Vs).
So, beware. Things in Big Data may not be as they may seem.
It’s not about bigI have been waging an uphill battle against the nonsensical and unsubstantiated idea that more data is better data, but now this view is getting some additional support, and from some surprising corners.
In a recent blog piece on IBM’s Big Data and Analytics Hub (Big data: Think Smarter, not bigger), Bernard Marr wrote that “the truth is, it isn’t how big your data is, it’s what you do with it that matters!”
Elsewhere, SAS echoed similar sentiments on their web site: “The real issue is not that you are acquiring large amounts of data. It’s what you do with the data that counts.”
Can we call that ‘strike one’ for Big Data Vs?
It’s not about varietyIt is claimed that 20% of digital data is structured, it is based on the problematic suggestion that structured data is uniquely relational. It is also claimed that unstructured data includes CSV files and XML data, and this makes up far more than the 20% of the data generated. But this definition is simply wrong.
If anything, CSV data is structured, and XML data is highly structured, and it’s typically regular ASCII data. So it does not add variety, even though it is not structured in the ways that some people might expect, especially if that someone lacks the required knowledge and experience. Simply stated, CSV data is structured, it’s just that it lacks rich metadata, but that doesn’t make it unstructured.
“But”, I hear you say “what about all the non-textual data such as multi-media, and what about the masses of unstructured textual data?”
Take it from me, most businesses will not be basing their business strategies on the analysis of a glut of selfies, home videos of cute kittens, or the complete works of William Shakespeare or Dan Brown. Almost all business analysis will continue to be carried out on structured data obtained primarily from internal operational systems and external structured data providers.
Strike two! Third time lucky?
It’s not even about velocitySo, if we accept that Big Data isn’t really about the data volumes or data variety that leaves us with velocity, right? Well no, because if it isn’t about record breaking VLDBor significant data variety, then for most commercial businesses the management of data velocity becomes either less of an issue or just is no issue. The fact that some software vendors and IT service suppliers set up this ‘straw man’ argument and then knock it down with the ‘amazing powers’ of their products and services, is quite another matter.
Strike three, and counting.
It’s not about the manageability of Big DataWe have been told and time again that the major difference between a data scientist and professional statistician is that the ‘scientists’ know how to cope very well with massive volumes, varieties and velocities of data. Now it turns out that this is also questionable.
According to Bob Violino writing in Information Management (Messy Big Data Overwhelms Data Scientists – 20 February 2015) “Data scientists see messy, disorganized data as a major hurdle preventing them from doing what they find most interesting in their jobs”. So, when it comes to data quality and structure the ‘scientists’ don’t really have an advantage over professional statisticians.
Last year Thomas C. Redman writing in the Harvard Business Review (Data’s Credibility Problem) noted that when Big Data is unreliable “managers quickly lose faith” and “and fall back on their intuition to make decisions, steer their companies, and implement strategy” and when this happens there is a propensity to reject potentially “important, counterintuitive implications that emerge from big data analyses.”
Strike four?
The new analytics aren’t newData science and Big Data analytics are the new kids on the block, aren’t they?
Well, here are some real life scenarios.
A major banking equipment supplier: A lot of banking equipment is hybrid analogic-digital, a simple example of this would be a photo copier or a physical document processing device. One major supplier decided to incorporate the capture of sensor data produced by their devices to predict failure and problems. Predictive preventive maintenance rules are created and corroborated using the data generated by sensors on each customer device, and these rules then get incorporated into the devices logic.
A major IT vendor: What happens when you create an intersection and convergence between technologies, techniques and method from areas of mainstream IT, data architecture and management, statistics (quantitative and qualitative analytics) and data visualisation, artificial intelligence/machine learning and knowledge management? This is precisely what one of the main European IT vendors did, and the idea proved to be quite attractive to customers, prospects and investors.
A major integrated circuit supplier: The testing of ICs at the ‘fabs’ (manufacturing plants) generates serious amount of data. This data is used to detect errors in the IC manufacturing process, it is captured and analysed in as near real-time as possible, which is necessary due to the costly nature of over-running the production of faulty ICs. To get around this problem the company uses a combination of fast data capture, transformation and loading of data into a data analytics area to ensure early and precise problem detection.
All Big Data Analytics success stories?
The first happened in 1989, the second in 1993 and the third in 2001. Yes, Big Data and Big Data analytics are sort of newish.
Strike five.
What is science?
According to Vasant Dhar of the Stern School of Business (Data Science and Prediction), Jeff Leek (The key word in “Data Science” is not Data, it is Science), and repeated on Wikipedia, “In general terms, data science is the extraction of knowledge from data”. Well, excuse me if I beg to differ. I have seen data scientists at work, and the word science doesn’t actually jump out and grab you. It’s difficult to make the connection, just as it is to accurately connect some popular science magazines with fundamental scientific research.
If a professional and qualified statistician wants to label themselves a data scientist then I have no issue with that, it’s their problem, but I am not willing to lend credibility to the term ‘data scientist’ when it is merely an interesting job title, with at most a tenuous connection to the actual role, and one that is liberally applied, with the almost customary largesse of IT, to creative code hackers and business-averse dabblers in data.
As Hazelcast VP Miko Matsumura suggested in Data Science is Dead “… put “Data Scientist” on your resume. It may get you additional calls from recruiters, and maybe even a spiffy new job, where you’ll be the King or Queen of a rotting whale-carcass of data” and ” Don’t be the data scientist tasked with the crime-scene cleanup of most companies’ “Big Data”—be the developer, programmer, or entrepreneur who can think, code, and create the future.”
Strike six.
And the value is questionableDATA: “Data is a super-class of a modern representation of an arcane symbology.” – Anon
If I had a dollar for every time I heard someone claim that data has intrinsic positive value then I would be as wealthy as Warren Buffet.
If I have said it once, I have said it a hundred time. In order for data to be more than an operational necessity it requires context.
Providing valid data with valid context turns that data into information.
Data can be relevant and data can be irrelevant. That relevance or irrelevance of data may be permanent or temporary, continuous or episodic, qualitative or quantitative.
Some data is meaningless, and there are cases whereby nobody can remember why it was collected or what purpose it serves.
Taking all this into account we can ask the deadly pragmatic question: what value does this data have? Which is sometimes answered with a pertinent ‘no value whatsoever’.
Strike seven.
It is said that Big Data is changing the world, but for all intents and purposes, and shamed by previous Big Data excesses, some people are rapidly changing the definitions and parameters of Big Data, and to position it as being more tangible and down-to-earth, whilst moving it away from its position as an overhyped and dead-ended liability.
Big Data is a dopey term, applied necessarily ambiguously to a surfeit of tenuously connected vagaries, and its time has come and gone. So, let’s drop the Big Data moniker, and embrace the fact that data is data, and long live ‘All Data’, yes, all digital data. Let’s consider all data and for what it’s worth to the business, and not for what some chatterers reckon its value is – having as they do, little or no insight into the businesses to which they refer, or of the data in that these businesses possess.
So, when push comes to shove, is Big Data really about high volumes, high velocity and high variety, or is it in fact about much noise, too much pomposity and abundant similarity leading to unnecessary high anxiety?
Thanks very much for reading.
01 Sunday Mar 2015
Posted in All Data, Big Data, Consider this
Tags
All Data, Big Data, data management, Good Strat, good strat blog, Good Strategy, Martyn Jones, Martyn Richard Jones
Big Data is now an inhospitable and unhealthy land inhabited by those who, through accident or design, deceive naïve and sentimental bystanders and those who are willingly mislead.
When all of this Big Data malarkey started it was sort of funny, humorous and occasional witty, especially in the affected, bizarre and the frequently uninhibited ways that freshly-minted self-appointed gurus and experts would “big it up”
Doctor Freud would have had a field day with all of that, being as it was, and for that matter still is, a postmodern mishmash of Riefenstahl, Freddy Mercury and Monty Python on steroids. However, after that extended, operatic and high-camp hiatus it all went downhill.
The Big Data scene is fast becoming an outrageous and brash festival of deception, disinformation and obliviousness. Which is a pity, because it does the industry no good whatsoever.
It is telling that Big Data evangelists, gurus and assorted sycophants cannot even define Big Data adequately, never mind discuss (or for that matter, point at) tangible success stories, without falling into contradictions on all of the key defining characteristics of volume, variety and velocity, and resorting to crude debating devices to avoid or finesse the concerns and the questions.
Almost every morning I check out the industry news, and almost invariably, it comes with new mind-boggling examples of Big Data nonsense.
However, it isn’t always nonsense for nonsense’s sake, there are agendas, there are rational explanations why Big Data has become at the same time, one of the most hyped up fads in the history of IT, and one that its supporters find so difficult to actually explain and justify, in any reasonable sort of way.
Therefore, when it comes to Big Data, beyond the surfeit of platitudes, clichés, bluff and bluster, the only thing in play are the interests of industry, the patrons, the courtesans and their entourage of the innocent and the beguiled.
One of the biggest deceptions in Big Data is in the misleadingly named ‘success stories’. The thing is that most of these success stories that I have ever read have been:
However, it doesn’t stop there.
One of the clearest examples of the questionable nature of Big Data evangelism is when it is used to piggyback Big Data hype on simple, tangible and immediately recognisable artefacts or applications that have little in common with Big Data.
This is an extreme illustration, but it works like this: “iPhones are commercially successful, iPhones are part of Big Data, and therefore Big Data is commercially successful.”
As if the mere conjuring up of association, affinity and proximity will convince people of the great and growing value of Big Data.
What I am also referring to are publicity pieces that may as well have been titled:
Do you recognise similarities?
It’s no big deal, just the use of unreliable, misleading and inappropriate fallacies, dressed up as cute, plausible and accessible collateral. People may think that such things are clever and witty, but they aren’t, it’s just misleading.
Let’s continue with something simple.
Evasion is, in ethics, an act that deceives by stating a true statement that is immaterial or leads to a false deduction. For example, citing events, persons or anecdotes from the history of IT to justify the supposed or imaginary value of Big Data. This is close to the notion of a non sequitur, which of course is an argument, the conclusions from which do not follow from its premise. It falls short of being full-on sophistry, purely because the simplistic, puerile and superficial arguments put forward in favour of Big Data do not match those of the true sophist who seeks to reason with clever but fallacious and deceptive arguments. Too many of the Big Data arguments are fallacious and deceptive, but no one, equipped with a reasonable capacity for critical thinking, should take such ‘arguments’ as valid.
Hold this thought: Big Data hype is a viper’s nest of logical fallacies, white lies and disinformation.
Just when I think things could not get any weirder, they do, and Big Data ceiling of hyperbole rises even higher, up to the rarer atmosphere of extreme tendentiousness.
There is a growing mass of Big Data hoop-la, hyperbole and flim flam that exceeds all previously bounds of overstatement, solecism and confabulation. This is where the real volumes, varieties and velocities are in Big Data; in hokie.
We live, as Oscar Wilde said in his day, in and age of surfaces. Yes, superficiality, puerility and short-termism are the competing orders of the day. However, I am still amazed – and maybe wrongly so – by what ostensibly professional, experienced and knowledgeable people are willing, able and prepared to accept, especially when it comes to Big Data flim flam sauce.
Here are some examples of the nonsense about Big Data that is taken as gospel by ‘adults’:
Data Warehousing is part of Big Data: No comment.
Big Data will replace Enterprise Data Warehousing: People can’t even explain the features and benefits of Big Data. I try it make it as easy as possible, ‘if you can’t say it, point to it’. But, seriously, people can’t even relate tangible and credible Big Data success stories, never mind show how it will replace Enterprise Data Warehousing, whether that’s the Inmon or Kimball flavour, take your pick.
Everyone and every organisation can benefit from Big Data: If people can’t explain this, and they don’t in terms of tangible benefits, then the claim should remain questionable.
Data Scientists will replace Statisticians: Why is that so? It is claimed that Data Scientists are uniquely equipped to handle massive volumes, varieties and velocities of data – well, as it turns out, this isn’t certain either.
Big Data is in its infancy: I think we may be confusing infancy with lack of real traction, and of time and place utility.
You cannot be serious: Just what are people talking about here? I have read vague, naïve and ill-informed pieces about data management, data architecture, data warehousing, reporting, business intelligence and a plethora of etcetera that have been passed off as observations and commentary on Big Data. So, what makes people recycle hackneyed, misleading and badly conceptualised ‘content’?
In the commentary on one of Bernard Marr’s pieces on LinkedIn (a professional networking site) I observed that no one can adequately explain what Big Data is without falling into contradictions and fancies, and no one seems to be capable or willing to provide tangible success stories.
Bernard responded to this comment by pointing out “the reason for that is that Big Data means different things to different people.”
Fair enough. It’s an explanation.
That said, I have always had more than a tenuous dislike of postmodern thinking, in fact most things ‘postmodern’. Call me old fashioned, jaded or cynical, but to me, the idea that everything can mean anything is an aberration that I prefer to leave to others.
I am at a loss to explain why so many reasonable people are willing to embrace the hype surrounding Big Data and Big Data Analytics, including the attendant surfeit of nonsense, incongruences and contradictions, and from my perspective, it defies reason and good sense.
Therefore, I will just end again with a fabulous quote from Ben Goldacre:
“You cannot reason people out of a position that they did not reason themselves into”.
Many thanks for reading.
01 Sunday Mar 2015
Posted in Big Data, Consider this, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Jones
Please note: This is an edited version of a previous piece with a similar name, but focusing solely on the three main Vs of Big Data.
What we’ve been toldWe’ve been told that business Big Data is the greatest thing since sliced bread, and that its major characteristics are:
Which is a simple and straightforward means of classification. Big Data is about massive volumes, high variety and high velocity. Right?
It’s not about bigI have never bought into the idea that more data is necessarily better data, or that it provides better focus or leads to increased insight, in fact I have been quite vocal with my contrarian opinion, but now this view is getting some additional support, and from some surprising corners.
In a recent blog piece on IBM’s Big Data and Analytics Hub (Big data: Think Smarter, not bigger), Bernard Marr wrote that “the truth is, it isn’t how big your data is, it’s what you do with it that matters!”
Over at Fierce Big Data it was Pam Baker who stated that “the term big data is unfortunate because it’s really not about the size of the data”. (Big data is not about petabytes, but complex computing).
Elsewhere, SAS echoed similar sentiments on their web site: “The real issue is not that you are acquiring large amounts of data. It’s what you do with the data that counts.”
Well, apparently Big Data isn’t about “massive volumes” of data.
Strike 1!
It is claimed that 20% of digital data is structured, it is based on the problematic suggestion that structured data is uniquely relational.
It is also said that unstructured data includes CSV files and XML data, and this makes up far more than the 20% of the data generated. But this definition is wrong.
If anything, CSV data is structured, and XML data is highly structured, and it’s typically regular ASCII data. So there it does not add variety, even though it is not structured in the ways that some someone might expect, especially if that someone lacks the required knowledge and experience. Simply stated, CSV data is structured, it’s just that it lacks rich metadata, but that doesn’t make it unstructured.
“But”, I hear you say “what about all the non-textual data such as multi-media, and what about the masses of unstructured textual data?”
Take it from me, most businesses will not be basing their business strategies on the analysis of a glut of selfies, juvenile twittering, home videos of cute kittens, or the complete works of William Shakespeare. Almost all business analysis (whether done by a professional statistician or a data scientist) will continue to be carried out using structured data obtained primarily from internal operational systems and external structured data providers.
Variety, Sir? No problem.
Strike two!
So, if we accept that Big Data isn’t really about the massive data volumes or high data variety then that leaves us with velocity. Because if it isn’t about record breaking VLDB or significant data variety, then for most commercial businesses the management of data velocity becomes either less of an issue or just is no issue.
Even in some extreme circumstances, one can explore the suggestion that data sampling can remove issues with data volume as well as velocity.
However, the fact that some software vendors and IT service suppliers set up this‘straw man’ velocity argument and then knock it down with the ‘amazing powers’ of their products and services, is quite another matter.
So, is it really about velocity?
Strike three!
Big Data is a dopey term, applied necessarily ambiguously to a surfeit of tenuously connected vagaries, and its time has come and gone. Let’s dump the Big Data moniker, and the 3 Vs along with it, and embrace the fact that data is data, there will always be more of it.
So, let’s consider ‘all data’ and principally for its time and place utility.
If there is something that you are not sure about or have questions with then please leave a comment below or email me.
Thanks very much for reading.
17 Tuesday Feb 2015
Posted in Big Data, Consider this, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Jones, Strategy
Tags
Analytics, Big Data, data management, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Jones
Hold this thought: Big Data is King.
Is there just nothing that Big Data isn’t capable of fixing? From terrorism, world hunger, Ebola, HIV, fraud, money laundering and hiring the ‘right’ people through to winning the lottery, curing hangovers, arranging entrapment and finding the love of your life. Big Data is King.
I have it on good authority that by 2016, Big Data will be at the forefront of everything we do, think and wish for. In the era of unending benefits, options and possibilities, what more can one say other than ‘All hail Big Data! Our spiritual leader and mystical guiding light’.
Which brings me to the subject of Harry Potter.
There’s a series of Harry Potter books written for children, yes, children’s books written for children. Not written for adults, but for children. Anyway, there are a lot of Harry Potter books, with an amazing range of popular titles, such as Harry Potter and the Mitten of Mirth (apologies to Stewart Lee), Harry Potter and the Piece of Wood and Harry Potter and the Crock of Crap, and so on and so forth. All of which reminded me of the fantastic Big Data promotional material that can be found on scientific, technical and management blogs, and also on the social media sites, such as LinkedIn, Facebook and Twitter.
As a tribute to Harry Potter, Big Data and those who vaingloriously and thanklessly schlepp the hype, I have rewritten some of the amazing and almost unbelievable Big Data success stories in the style of postmodern children’s literature.
Enjoy.
Now who doesn’t love a lovely cuppa? From Indore to Idaho, Dubai to Dortmund and Beijing to Birmingham, it’s the breakfast drink of champions, the soother of pains and the bringer of temporal peace.
But did you know that if it weren’t for Big Data, we wouldn’t know how to brew a proper pot of tea. You see it’s all in the value chain, the data and the fresh boiling water, naturally.
If it weren’t for Big Data, then tea growers wouldn’t know what they were doing, they wouldn’t know when to plant, when to nurture and when to harvest, they wouldn’t even know what to plant, or even why – without Big Data they would be completely stupid. Likewise, the processing of tea without Big Data would lose all sense of being and purpose, which leads me to logistics. Without Big Data it would be impossible to move the exabytes of tea from one side of the world to the other. Fact!
But the most important role of Big Data in all of this is simple. If we hadn’t been able to use Big Data to analyse the results of how everyone makes a pot of tea, and the feelings of satisfaction or disappointment they gain from that process, then no one could even make a decent pot of tea, anywhere, and they wouldn’t even be motivated to engage with the process never mind the tea brands.
Now that really is food for thought.
Big Data and the Piece of WorkIt is official, the fact that dared not speak its name has now been revealed: Big Data is helping to take the conjecture out of hiring and firing.
We can now leverage the amazing power of Big Data in the cradle-to-grave Human Resource process. This is the true democratisation of the world of work.
Because of the amazing correlations made by data scientists across massive HR datasets and golden nuggets from social media, we now know how to place people exactly in the organisation, in the approptiate roles and aligned to the appropriate responsibilities.
In the past we used to hire on the basis of personality, values, recommendations, knowledge and experience, certifications, qualifications, cost, and so on and so forth. All that has changed.
Big Data has determined that the most important asset that a job candidate can possess is aggregated mediocrity. No more heroes, no more geniuses, no more hard workers and thinkers, and certainly no innovators, because what most established organisations need, according to in-depth Big Data research, are workers who are dull, boring, servile and above all, mediocre.
How can we ensure that an employee sticks to the game plan? Now here’s a thought: just wire them up with wearable Internet of Things probes, and if they start to think too much or do too much or just exude an air of knowing too much, the system just automatically issues them with their dismissal notice, alternatively, the more humane organisations may consider the use of incentivised mild electric shocks administered via those very same IoT devices.
No longer will HR departments and line managers take the rap for hiring the wrong suits. Now we can blame it all on Big Data, or even better, the belligerent, alienated and disgruntled ex-employee.
Thanks to Big Data we now know that people (that’s human adults of a certain age or maturity) give birth approximately 9 months from the process point of conception, the people are invariably female, and the offspring are invariably of one sex or the other, in no special order of appearance.
Thanks to Big Data we now know that most people live from a ripe young age of approximately nothing (years of age) to a grand old age (of something), or not, and that this applies to various levels of demographic extrapolation, and is generally accompanied by an attendant aging process.
Whichever way you slice and dice it, the great Universal Heater in the sky is warming up the planet, and probably this is happening because of something we did way back when, or conversely, something that we persist in doing in the here and now, or maybe a bit of both.
Climate Change is reality. Heuristics together with dialectic and historical materialism showed Climate Change to be a fact, then science proved it to be true, now Big Data substantiates conclusively that true is frequently correct.
So, either you think that Big Data will save the world, as it most evidently will, or you might just dismiss it, along with Global Warming, as yet another scary Marxist-Leninist inspired technology plot and fad, which you most evidently can do.
Personally I think that Big Data is generating far more tonnes of equivalent Greenhouse Gas emissions than some people will openly admit to, and that the charges of Big Data collectivism are somewhat premature, foolish and misguided, even if they are suspiciously, precociously and predictably present, or hopelessly, recklessly and witheringly absent. Which either way, represents a dilemma within an enigma inside a hypothesis wrapped in swaddling clothes – to various degrees of certainty, of course.
Maybe we need to consider hedging our bets a little with the ‘angels of our better nature’. Either way, Big Data will be inextricably linked to Climate Change, Global Warming, Immigration, Emigration, Inhalation and Global Governance, which will keep a lot of curtain twitchers busy for decades to come.
In the Oscar winning film The Wrong Trousers, the accidental hero, Wallace, thinking that it will make his life easier, becomes the innocent victim to a pair of automated ‘Techno Trousers’, that take control and carry him off against his will in directions and to places that he does not wish to go.
Wallace uses the ‘Techno Trousers’ thinking that it will increase his power and stride when he is out walking the dog – that’s Gromit, if you remember – but he is gravely mistaken.
Things get worse when the felonious penguin Feathers McGraw attempts to make Wallace, now locked into Techno Trouser mode, an unwilling accomplice to the theft of a diamond. Thanks largely to Gromit, the plan does not succeed, and Feathers ends up in jail. After his traumatic experiences, Wallace realises that the Trousers are not the valuable addition to his lifestyle that he once thought they were.
Which is pretty much sums up what can be said about some of the vast quantities and qualities of ‘valuable’ Big Data flowing through the data lakes and digital drains of the corporate underground. Some of it is as potentially useful and risk free as ‘Techno Trousers’, and may not be the valuable addition to your corporate lifestyle that you might have initially been told to expect.
But your idol may be Feathers McGraw, in which case the only sensible recommendation for you may be: do your own thing, Big Data baby! Word!
So if you are a CIO, CTO or even CEO who feels an extremely strong affinity to Harry Potter, a series of children’s books, then you might also be fascinated by the whole Big Data hoopla as well.
But before I leave you, back to the thought… what about Big Data being King? What’s all that about then?
Well, every courtesan needs his monarch and court, and this is no different with the Emperor’s Big Data. All those pushing the Big Data showboat by dodgy means or foul, need a royal reference point, a shining brilliant light of data aristocracy and patronage in which they can bathe and frolic, oblivious to the trials and tribulations of the proletariat, and the organisations they seek to fleece. In the old days the monarchs would have names such as Friedrich II der Grosse, Aethelred II the Unready or Alfonso X el Sabio. Now they are named after fruits, brown sauces and Greek and geek myths, and others with even less imagination than sense, simply go for the old three letter acronym solutions, such as BOB, SUE or HAL.
Am I bothered by this? Should you or your organisation be bothered by this? Does this affect the price of bread or the quality of beer? Will this be the mommy and daddy of brazen hyperbole, manipulation and fraud to end all credibility, sanity and reason anyone has ever had, everywhere and anywhere, the whole world over?
No, not at all. This is a minor blip, blown up out of all proportions by the media foghorn of published (not public) opinion. So, down with the data Aristos. Vive la République! Vive la Résistance!
Many thanks for reading.
