Tags
Behavioural Economics, Big Data, Business Enablement, business intelligence, Business Management, business strategy, Challenges, Creativity, Data Warehouse, Organisational Autism
We analysed all the big data and discovered that the biggest reason for IT project failure is people – Big Data Informs…
We had failed at Data Warehousing, Business Intelligence, Core Competence, and quite a few other things, so some bright spark decided to give Big Data a shot.
The first task was to identify the reasons for IT project failure, globally.
According to the techies, Big Data was helping to move things on quite a bit, especially considering a previous attempt to analyse IT project ended tragically when the Data Warehouse coal-face caved in.
Before we gathered together all of the data in the Good Big Data project, we didn’t have a clue as to what was causing so many frequent, costly and dramatic failures.
We have an idea of where the biggest problems may be, but the Big Data team are afraid to pony up.
So, instead of boiling the ocean of data again, we decide to narrow the scope to Data Warehousing and Business Intelligence projects.
We were three months into this project and we’d still not achieved anything to brag about. So I put on my Project Manager’s hat and diplomatically engage up with the Big Data team.
“What the feck are you guys playing at?” I ask “You’ve had three months to come up with findings, and you have found nothing”
So, one by one, out come all the perfectly reasonable excuses and justifications.
“We didn’t know”, “this is very complicated”, “you don’t understand”, “I have the flu”. It all came out. We dance around the issues for a while, and then I set some tasks.
“I want you to find out what the prime motivators are for working on Data Warehouse and Business Intelligence projects”
“Is it for the cache of working on such projects?”
“Is it to bring real technical knowledge and experience to the party?”
“Is it to learn a technology, new product or technique?”
“Is it solely for the money?”
“Is it to ensure that the project lasts for as long as it can?”
“Is it to milk the budget for all its worth’”
“Is it to achieve the business objectives?”
“Is it to create inertia?”
“Is it to be on the inside, to ensure that the project fails?”
“Go and find out just what motivates people to work on these projects”
“Do it now and report back to me this time next week”.
So, I set and assign the tasks, clarify and address every current doubt, and leave.
Next week I go back. The team has a delegated spokesperson.
He says “we have addressed the questions you posed, and the answer is yes”
“Go on” I reply.
“It seems that to a greater of lesser extent, the questions you posed last week are all relevant”
“Fine, now tell me more”
“Well, there is not much to say, apart from the fact that what motivate many people isn’t exactly in the best interests of the projects in question”.
What am I listening to? No shit Sherlock!
“Can you expand on that?” I ask “Let’s open this up to everyone”.
So we have another three hours of discussion.
In the end what emerges is a classic set of metaphors and analogies that clearly identify why so many Data Warehouse and Business Intelligence projects go wrong, and indeed why this particular project cannot really deliver.
So, I wind things up.
“This is how I see it”
“3rd party suppliers and vendors want to see these projects last for as long as possible”
“The more licences, consulting days and bodies they can bill for, the better for them”
“The longer they take our money, and the more of it they take, the better it is for them”
“The more that innocent glitches, hiccups, procrastination and prevarication can be fabricated, forced and imposed, the longer everything takes, and the more that is billed for”
“So, better to over-promise, over-reach and under-deliver, than do things on time and to spec”
“What’s more, many people working on such projects will take the sides of the supplier, to the detriment of the client’s interests”
“Money is being leeched from healthy corporations to pay for bullshit death-march projects that deliver no value, bring no insight and can actually be a risk to corporate health”
“Projects are being financed by us, and used by others, as training”
“Corporations are being used as reference sites, even though the fundamental premise is nonsense”
“We are paying to teach people, bad-practice, worst-practice and no-practice”
“We are creating private armies of artful mediocrity, banality and imbecility”
“And we are proclaiming it as the way that business should be done in the future”
“Well, feck that! I don’t need big data to inform me that we are being taken for a ride”.
So, after ten days of contemplation, I formally close the project.
It had become a meta-example of what we were ostensibly investigating, analysing.and reporting on.
Yours strategically, Martyn
GOOD STRATEGY 
It looks like “business intelligence” has as much to do with intelligence as military music with Music, isn’t it?
LikeLiked by 1 person
Indeed. It’s fast becoming one of the great oxymoron, at least in IT.
LikeLike
Drastic step You took Martyn. But for good reasons it seems.
However I wonder: Why was “Big Data” ever involved in a Data Warehousing and Business Intelligence project that had been narrowed?
Big data (volatile, unstructured and/or huge amounts of data) only makes sense when You can relate them to highly credible and well structured data. So why would You want to involve Big data in a “narrow” DWH-project?
If someone needs to add Big Data to an analysis foundation, then why not start up with a prototype or two to illustrate how Big data can be used in collaboration with the structured DWH data?
That way You should relatively quickly be able to determine, how valid and significant the Big data is, compared to other more readily available and easily sourced data.
My personal observation from Copenhagen:
I keep hearing: Big Data is comming! It cannot be avoided. You need to addapt to Big data NOW!
On the other hand, Most companies do not deal through social media, market through socail media or complie petabytes of data.
Meanwhile Hardware is constantly pushing the limmits of the amounts of structured data that can be handled.
My conclusion: Most companies can afford to wait a few years more, before they plunge into Big data. And maybe (just maybe) tools by then will have matured enough to make handling of Big data simple.
LikeLiked by 1 person
Hi Bjorne,
Many thanks for your comments.
IMHO Data Warehousing as defined by Inmon, is a practical IT engineering concept for creating an architecture and process for collecting, processing and delivering data for strategic and tactical support. It is intrinsically tied to iterative and agile approaches to requirements gathering, analysis, build, test and delivery. In its infancy many in the industry sought to redefine DW in order to be able to better position their own products, therefore people like Sybase, backed by the dimensional model advocates, preached that a Data Warehouse was not a necessary component of Data Warehousing, and that the same could be achieved by building lots and lots of independent data marts.
Nowadays, Kimball has shifted his position on the subject so much that he is to all extents and purposes agreeing with Inmon on 95% of what Data Warehousing should be. There is only minor divergence in what data model the Data Warehouse itself should use.
Many organisations still avoid Inmon and Kimball when it comes to DW and especially EDW. Much like programmers have traditionally ignored Donald Knuth when they need a basic algorithm.
Business Intelligence and Big Data have become catch all marketing terms.
Business Intelligence is old fashioned Management Information Systems with reporting, flashier visualisations and interface and larger menus.
Big data has been injected with so many marketing steroids that it has become a massive catch all, which can mean something or nothing, or something else, depending on the time of day and the proclivity of the snake oil merchant.
Big data seems to have displaced the more scientific and technical aspects of VLDB (with its constantly changing definition of what is a VLDB – but only a question of size).
When I read descriptions of what Big Data is supposed to be, it’s like a potpourri of things that have been done before.
For starters, it is claimed that Big Data will now give us insight into new patterns in data. i.e. it will produce actionable information from data. Yet, we know from decades of research into parallel distributed processing and data mining (the real one based on statistical techniques and adaptive neural nets, and not the navigation tools marketing nonsense) that having more data may actually reduce the chances of being able to extract hidden patterns to zero.
Also, many large data sets, for example in the financial industry you can recover years of trading patterns and price curves, will contain biases so massive that it isn’t of much use in presenting an accurate picture of what actually happened and what would happen in similar circumstances.
Also, in some organisations (Telcos, major semiconductor manufacturers, intelligence agencies, etc.) people such as SAS analysts and programmers have been doing Big Data for years, and no one noticed or even called it Big Data.
In my opinion, the essential elements of Data Warehousing, Business Intelligence and Big Data aren’t really that new. Even Dimensional Modelling is too old for the army.
Data Warehousing evolved out of Information Centres. Because of the time and place issue, very large Information Centres required hugely expensive proprietary hardware. By the time Bill published Building the Data Warehouse, hardware/OS had become open, more powerful for basic data manipulation and far cheaper. It’s not that it wasn’t done before, it was that it was horrendously costly to do so. Nonetheless, people did it.
Same for Business Intelligence (Reporting and MIS) and VLDB (Big Data).
Also the issue of managing unstructured data is also not new. Serious work in this area was carried out under the pretence of working on the Strategic Defense Initiative, it formed a part of earlier work into Knowledge Management, and popular ideas such as Web Harvesting. One of the ideas that I worked on at HP in the mid to late nineties was how to link Data Warehousing and KM.
BTW I actually don’t like the term unstructured data, because mostly we seem to be referring to highly-structured and highly variable structured data/information. But I guess there are some cute marketing reasons for calling it unstructured.
I.e. Unstructured = not so difficult; complex-structures = we’ll never be able to do that.
What I see most of the time is this:
Data Warehouse projects that are not Data Warehouse projects, but very bad re-enactments of the worst fails in Information Centre history.
Business Intelligence projects that fail to deliver intelligence or any business value or meaning, many are worse than the worst Management Information Centre projects of all times. I tend to agree with Antonio Olmedo (previous comment), and that Business Intelligence has become an oxymoron, much like Military Music and ‘British Intelligence’ (i.e. the intelligence agencies, not the general population).
Big Data projects that will produce far less insight than the approaches of the judicial application of data sampling techniques and data analytical approaches. I’m still waiting for a Big Data pundit to claim that Big Data helped Wal-Mart to place the beer next to the diapers, therefore selling more beer.
Anyway, those are my thoughts on the subject this AM.
Would like to expand on and clarify some of these points, but I have a meeting in just under an hour.
Cheers,
Martyn
LikeLike