30 Tuesday Dec 2014
Posted in Big Data
I have looked into the immediate twelve-month future of Big Data and the immediate future looked back and said “Be more goat”.
27 Saturday Dec 2014
Posted in Consider this
Tags
In the modern world if you question the lack of ethics, absence of morality or the scarcity of good sense of something, some claim, some recommendation or some piece of advice, especially if confronts or compromises the status quo, the wilfully ignorant and content and the obtuse conformists, then more often than not you will be labelled as a negative person, a mischief maker and a member of the awkward squad. Continue reading
27 Saturday Dec 2014
Posted in Big Data, Consider this
Tags
Plus ça change, plus c’est la même chose.
Jean-Baptiste Alphonse Karr
I have been involved (all afternoon as a matter of fact) in an in-depth study of the changing face of IT, data architecture and data management, and the challenges that the profession faces.
In particular I have tried to focus on emerging and evolving roles and responsibilities, and in their significance, synergies and collaborative potential in a predictably high-speed, volatile and exotic future.
I know that many people will question the need to create new roles in statistical analysis, qualitative analysis, and data architecture and management, and I must admit that I also shy away from the invention of new terms, especially when they may seem to be superfluous and misleading, but I feel that the spirit of the times is calling out for a revolution in how we view and appreciate the world of data professionals.
Some of the new roles detailed here may not be immediately familiar or intuitive, and some of the responsibilities may seem to be somewhat onerous or even trivial. But this is not accidental. As what has lead me here is the desire to formulate a coherent and cohesive response to the IT industries sea-change with respect to disruptive and game-changing innovations such as Cloud data centres, the Internet of Things and Big Data.
So here is my take on what I see as being the new roles – 7+3 in all – and responsibilities within many if not all of the Next Generation Mega-Mega Data projects coming our way. The roles that will be discussed here, are as follows:
Data Trader – The Data Trader is the high flying market maker of the alternative data universe. They are essentially the gears and oil of the data market, introducing market providers of data to market consumers of data. The Data Trader identifies potentially undervalued data, and price and quality discrepancies in alternative data sources, and then seeks to leverage these discrepancies in order to ‘monetise’ their valuable role in keeping the data market healthy. Data Traders also seek out data instruments on the instructions of a client. They may also issue and buy options and futures contracts on commoditised data, to be optionally executed and delivered at a later date. Although it is technically feasible, Data Traders will rarely trade on their own account.
Data Hound –Although the Data Hound is a special pedigree breed of data management role, the job of the Data Hound is essential to the work of the Data Trader.
When the Data Trader gets a new requirement for novel, fresh or new data, the Data Hound is charged with searching out the best, cheapest and most reliable sources for that data, and to identify the owners and vendors of that data. Essentially, they assist in the data market-making responsibilities of the Data Trader. But there is more to the role than that. Only a Data Hound can bring infectious enthusiasm to a long walk on the data landscape. Only a Data Hound can be such a perfect, patient distraction for the knowledge workers. And only a Data Hound can dispel all gloom, tension and work-stress with a single explosion of excitement every time you walk through the data portal.
Not for nothing shall the motto of the Data Hounds be Ad grandior data, Winalot!
Data Plumber – The Data Plumber designs, builds and maintains the infrastructure to ensure that any validly supplied data reaches the data preparation stage prior to its selection, analysis and consumption. The Data Plumber is charged with ensuring that the required data correctly gets from the data provider to the data consumer, first time, every time.
Typical responsibilities of the Data Plumber may include:
Data Butcher – The Data Butcher works in conjunction with the Data Chef. The Data Butcher selects and prepares the desired parts of the supplied data which they then pass on to the Data Chef for data mining, ad-hoc predictive analysis and visualisation. The Data Butcher removes the fat data from the lean data, and provides quality data that can then be subsequently ‘sliced, diced and spiced’ in downstream analytics applications.
In years from now, IT archaeologists will marvel at the Tau influences inherent in the role of the Data Butcher in particular, and data architecture and management, in general. By way of evidence, the following is a philosophical anecdote from the future:
There was once a Data Butcher who was preparing a piece of Big Data for a customer who had been coming for many years.
“Pardon me, Sir” the customer asked, “But isn’t that the same ETL you used last year? Don’t you ever need to upgrade it or maybe go for a more sophisticated and sharper solution?”
“No…” replied the kindly Data Butcher “It’s the same ETL I’ve been using for the last 17 years”. He stared wistfully into the distance for a few moments, looking for inspiration, and then continued. “And I haven’t had to upgrade it, sunset it or change it even once. For, when I select, transform and integrate raw data, I allow the trusty ETL to find its own way through it without effort or stress. Just like Bill told me. And when I come to a tricky bit with lots of disconnected, superfluous and erroneous data, I just slow down and allow the mystery to solve itself and in no time the good data comes right through the process.”
Adapted from Chuang Tzu: The Basic Writings, 1964
Data Chef – If you’ve ever seen a great Chef working, up close and intimate (to use a west coast expression), then you will appreciate the need for the role of Data Chef.
First and foremost, the Data Chef is curator of all the organisations data analytics ‘recipes’. They have the data analytics ‘knowledge’. Ideally, the Data Chef has a solid grounding in formal statistical methods and a solid appreciation of data architecture. This may also be augmented by a wide range of other skills, such as in Nouvelle analyse des données. The Data Chef also works in conjunction with the Data Trader and the Data Butcher to determine and identify prime data material in the data markets. Based on the available of prime data the Data Chef is able to determine a menu of data analysis approaches that will dynamically change depending on what data is in season and available on the data market.
Data Taster – A Data Taster is a person that takes data (or information) to be provided to a person or entity to confirm that it is safe to issue. This is perhaps one of the oldest professions in data, coming, as it does, from the ancient Roman role of praegustator or data unini. The person or entity to whom the data is going to be issued is usually an important person or body (for example, a regulatory reporting body or an organisational strategy group) or any person or body that could possibly be placed at risk if the data (or information) is erroneous, misleading or compromised. For example, the Data Taster verifies the outcomes of Big Data Analytics and confirms that the data is plausible and that the models used are valid so that they do not permit either the accidental or intentional introduction of data contamination. The Data Taster may also be accountable for the preparation and provision of data. The hope is that the Data Taster will be conscientious and meticulous in preventing contamination from being introduced into data, in order to safeguard their own reputation and that of their organisation.
Data Server – The Data Server is a role that is closely tied to the roles of Data Whisperer and Data Czar.
At a superficial level, the Data Server presents the data menu and takes the data orders, then serves what has been ordered. The Data Server may also advise data clients on the optimal choices of data, based on the data that is available and the data preferences of other clients.
Because the role of the Data Server requires that they know a little about everything and a lot about something, the two most popular career progression paths for Data Servers are moves into the role of Data Whisperer or Data Shouterer.
Data Whisperer – Is integrally associated with the roles of Data Server and Data Czar. This role is an extremely important key-stone position within an organisation.
The Data Whisperer is an explainer, a storyteller and a stand-up philosopher. The primary responsibility of the Data Whisperer is to avoid that any senior executive or regulatory body throws a ‘wobbly’ when they fail to correctly interpret the data that they are provided. Therefore, the responsibility of the Data Whisperer is to correctly socialise data analysis outcomes with the intended audiences for those outcomes, and to jointly present and explain those outcomes in plain and simple language. Thea are required to have courage, strength and a high degree of empathy both with the data and also with the consumers of that data.
Data Czar – Typically this is a senior Board role, comparable to that of CFO. Indeed, the role of Data Czar (or Data Tsar, for the British) may also be held by the CFO. The role itself is that of visible figurehead of all data architecture and management activities within an organisation. Although it bears some striking resemblance with the now widely discredited role of bygone days, which I will delve into in subsequent articles, its remit goes much further. The Data Czar has the ability and the empowerment to break down barriers, cut through red tape and knock down walls that create organisational silos. They can free and easily engage with and involve senior organisation players in their data campaigns and battles, gaining commitment, trust and willing complicity along the way. Naturally, the Data Czar can also call on the skills, talents, knowledge and experience of the other 9 roles identified here.
Data Shouterer (aka Data Shouter) – Finally, we come to the last of the roles. The Data Shouterer is primarily the role of the data evangelist, the extoller of grand data ‘truths’, the purveyor of serendipitous comfort, and the herald of a brave new data world.
When things go well the Data Shouterer is called upon to holler out the successes of data analysis from the rooftops.
When success is reluctant to come forth, they must be there to ‘big up’ the inherent potential for success, with brave tales of data buccaneering, ace information pilots and glorious exponents of the Art of Data.
So there you have a brief explanation of the 7(+3) new big data roles for 2015. So, to conclude…
Necessity is the mother of invention and mother is the invention of necessity, and just as the judicious use of parallel grep, awk and bash could have been reinvented, rebadged and released as perhaps justifiably the next best thing in data, so too must there be concomitant and relevant roles to suit the revolutionary data spirit of the times. No?
But, to paraphrase John McEnroe, “surely this piece cannot be serious?” To which I might reply, maybe yes, or maybe no, it simply ‘depends’. But depend on what?
The English writer George Orwell once mused that “The most effective way to destroy people is to deny and obliterate their own understanding of their history”, to which I could add, “This may occur whether the act is intentional, accidental or systemic”. I think it is important that when we look at any new IT industry trend or fad, that we do so with a reasonable knowledge of IT history and the evolution of IT technology, and with a good understanding of contemporary and legacy technologies and architectures. This, to my mind, is how we respect both the IT/Information Architecture and Management profession and those whom we seek to help.
Finally, dear reader, although as I state my intention is quite serious, please do take this piece with a modicum of sodium chloride.
In subsequent blog pieces I will be sharing my views on the evolution of information management in general, and the incorporation novel and innovative techniques, technologies and methods into well architected mainstream information supply frameworks, for primarily strategic and tactical objectives.
File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro
25 Thursday Dec 2014
Posted in Big Data, Business Intelligence, Consider this, Data Warehousing
Tags
I have worked in data architecture and management for three decades, I have become a recognised expert in my field, and as a result I have become almost oblivious to the fads, fancies and fashions that pass through IT. However, being an expert in a field also means that from time to time we are oblivious to the difficulties that some people may have when trying to understand issues and concepts that we simply take for granted – because, one simply knows. This is the case with data.
With the polemic and resourcefulness surrounding buzz-words such as Big Data, Cloud and the Internet of Things, one may be forgiven for assuming that there has been a massive inflection point in the generation, variety, understanding and use of data. Now, this isn’t strictly accurate, as, outside of the handful of speculative and high-visibility projects of social media and networking, data collators and indexers, search engines and online volume ad-sellers, there has been scant publicised evidence of significant ‘data revolutions’ elsewhere.
Things haven’t changed significantly simply because a handful of companies are making money with other people’s data, rather than in the more traditional organisation, where their own data is one of the most important assets. So what’s really out there in the world of data? First, let’s look at some broad-brush classes of data, namely:
Enterprise Operational Data – This is data that is used in applications that support the day to day running of an organisation’s operations. Typical data items in this space are sales transactions, purchase transactions, product information, client and contact information. Enterprise Operational Data may also include complexly structured data, such as contracts and other business documents. Applications in this space may include production control, logistics and stock control, as well as purchase order, supply chain management, management accounting and human resource modules.
Enterprise Process Data – This is measurement and management data collected to show how the operational systems are performing. In the past the recording of events went down to the level of a completed transaction – with a start and an end and nothing in between, and as transactions were kept as simple as possible, to maximize performance and throughput and minimise the risk of failure, very little process data was captured. Now, especially with the advent of Business Process Management and Web Logs, we collect a whole array of transaction and process performance data that was never previously captured.
Enterprise Information Data – This is primarily data which is collected from internal and external data sources, the most significant source being typically Enterprise Operational Data. Other sources for this aspect of data include Enterprise Process Data and data provided by 3rd party data providers. In this data space we find Enterprise Data Warehousing, Operational Data Stores, Data Marts and Special Project Data Stores. Applications in this space include support for strategic and tactical decision making, formal statistical analysis, speculative ad-hoc analysis, data mining, business intelligence and reporting (also called Management Information reporting), and qualitative as well as quantitative analysis.
As we can see, there are interdependencies, synergies between each of these broad areas of data generation and use. Of course, data in each of these areas can be augmented and enriched by new sources of data, whether that data is richer market data, competitive data or weather data. Now, this is a very simplistic view of data, but for the purposes here it is both coherent and cohesive. 
Fig. 1 – Data Made SImple
In the above diagram I have identified an area labelled as ‘Data Transitioning’. This is usually referred to as ETL (Extract, Transform and Load), although in more and more cases, instead of extracting data directly from source systems (Enterprise Operational Data Management) we are capturing data sent via enterprise message queueing, this drip feed approach in most cases allows us to maximise the time available for data loading and analysis.
Another important point to note is that although Enterprise Information Data Management has been associated with Relational Database platforms, such as Oracle, Teradata and DB/2, in this IT domain we are also using a wide range of ‘databases’, from the humble ‘flat file’ to powerful column oriented database engines, such as EXASOL, Teradata and Vertica to provide information and analysis to business stakeholders. As more and more ‘exotic’ data formats and sources are incorporated into our Enterprise Information Data platforms, we will also witness the evolution of tools, technologies and techniques to meet those new requirements.
Last of all, there will be no revolution in data management, because in most cases data architectures are built around sound data engineering principles, constrained and governed by the limitations of mainstream hardware, the operating systems that bring them to life and the competencies of those charged with designing, building or using them.
In subsequent blog pieces I will be sharing my views on the evolution of information management in general, and the incorporation of ‘Ad hoc Speculative-Predictive Analytics’ into well architected mainstream information supply frameworks for primarily strategic and tactical objectives.
File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro
25 Thursday Dec 2014
Posted in Books with influence, Creativity
Dialectic of Enlightenment is undoubtedly the most influential publication of the Frankfurt School of Critical Theory. Written during the Second World War and circulated privately, it appeared in a printed edition in Amsterdam in 1947. “What we had set out to do,” the authors write in the Preface, “was nothing less than to explain why humanity, instead of entering a truly human state, is sinking into a new kind of barbarism.”
From the 2007 Edition.
As a collection of philosophical and social musings, reflections and ideas planted firmly in the spirit and events and flux of the 1930s and 40s, Dialectic of Enlightenment, with its broad range of sketches, provides an accessible route to some of the thought of the times, and in a form that is easy to comprehend, think about and delve deeper into. I think that the problems associated with the reading of Dialectic of Enlightenment tend to be related to the fact that it is either treated as a coherent and cohesive master work, which it is not, that it is treated as highly accessible and therefore highly trivial, which it is not, and that frequently the subtle nuances of the works of thoughts and fragmentary ideas of the continental authors, especially when not originally written in English, tend to get lost in the translation.
What I also like about Dialectic of Enlightenment is that it is a subtle guide for the baffled by the perplexed, a handbook of philosophical survival, of political, social and mental health, a compendium of reassurance, and a book of ‘yes, we also think that our present paths will lead us to nowhere good’.
But the book is not without its detractors. There are some who have trouble with Dialectic of Enlightenment, for what it is, for what they want it to represent – but doesn’t, for what it tells us, for what it leaves for us to contemplate. That is, what it represents, what it represented and the uncomfortable messages it still sends no longer pleases so many.
The comfortably content have no such problems; they can just hate it, despise it and rubbish it, without actually understanding it or reading it. Even shallow thinkers are subject to dilemmas, but the ‘no thought zone’ of the ‘wilfully ignorant’ apart from perhaps being the most comfortable place on earth, is also one of its more sinister aspects.
Today, everyone’s opinion is treated equally, with the same validity, with the same value, with the same level of indifference or respectful disrespect, but at the end of the day, it is not the powerful, reasonable, logical and humane argument that wins, but the arguments that most closely flow with the system, for the benefit of the system and without upsetting the system to any large degree.
In this way we are suckered into accepting the status quo.
Just look at how we cannot think ourselves out of a paper bag. This is problem brought about in part by having no alternative to the current parties and their policies and leaders and interests, and that the only alternative would be for violent revolt, which most of us are strongly against. So we just accept what there is, without seeing the elephant in the room is saying “get off your arse and use the system to deinstall itself, and to install a more humane and reasonable system in its place”… too much like hard thinking and hard work I suppose, and no one is going to ‘provide’ that service over the internet or at the hyper-hyper store or mall.
The POMO nonsense of ‘everything goes’, ‘do your own thing, baby’ and ‘everything can mean anything’ is just that, self-entrapping crap that will ensure that quite a few people will rage against the effects of the central problems and causes of those problems by first exaggerating peripheral, marginal and frequently superfluous issues.
As Isaac Asimov so succinctly put it “Anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that ‘my ignorance is just as good as your knowledge.’”
In summary, these are some of the things I get from reading Horkheimer and Adorno’s sketches, a sense of ‘we are not alone’ when we think this way, but nonetheless, aspirations aside, this is what we have.
File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro
20 Saturday Dec 2014
Posted in Big Data, Data governance, Data Warehousing
Tags
Alas, poor Yorick! I knew him, Horatio; a fellow of infinite jest, of most excellent fancy; he hath borne me on his back a thousand times; and now, how abhorred in my imagination it is!
From the play Hamlet by William Shakespeare
Big Data is dead! Long live Information Management.
Of course, I am overstating. It isn’t tangibly dead, because outside of PR and Marketing, and the gullible imaginations of a few punters with more dollars than sense, it was never actually alive.
I don’t blame the industry for ‘bigging up’ Big Data, it was a ‘common sense’ continuation of the hubris and nonsense that began with total hegemony and total information awareness. It was a ‘natural’ response to a set of grandiose, flawed and quite unrealisable strategic objectives.
This is why Creative Directors in Advertising have such a hard time with it because it’s a dopey ‘brand’ of sorts that cannot even be pinned down. It’s vague, amorphous and brittle, that tries to be all things to all people, and trying to turn that into attention-grabbing advertising is like trying to nail Jell-O to an elephant.
So, Big Data will be no more. It will be ‘deceased’. As deceased as a Norwegian Blue, as dead as a Dodo, and as stiff as a cardboard box on the steppes of Russia in winter. But its demise is not going to substantially change a thing.
Why?
Because before Big Data there was everything that Big Data is supposed to be uniquely capable of doing, and much more, and this will be the case when the faddish use of the Big Data label is dropped in favour of other (possible equally fuzzy or meaningless terms).
Before Big Data came along we could do distributed and parallel processing, and performed database loading and querying, handling larger and larger volumes of data, from an ever-growing variety of sources.
I know younger readers may be shocked to discover this, but before Big Data came along we could also perform database restrictions; projections; joins; and, relational operations.
Finally, before Big Data came along we could perform statistical analysis on data.
Remarkable, huh?
You see, Big Data has been a faddish industry label, and for some, as confusing as it has been misleading.
I only have five issues with Big Data.
The first is the hype and all that serves to perpetuate it.
The second issue lies in the pervasive idea that more and more data is necessarily better.
The third issue is with the notion that extending information architecture and management will fall short when dealing with growing volumes of data, greater velocity, greater variety and increased needs for data veracity, especially in connection with the euphemistically termed unstructured data – which is in fact typically complexly structured information.
The fourth issue is the incessant claim that every organisation will need to be doing Big Data in order to survive and thrive.
The fifth issue concerns the widespread notion that Big Data has necessarily disregarded any ethical considerations in the use of data or any legal requirements in terms of data privacy and data protection.
I have an issue with the hype[1] because on so many occasions it’s utterly preposterous and thoroughly misleading. One may also be led to wonder if, in fact, the famous Big Data Vs of volume, velocity, variety and veracity refers to the data or to the hype. So, why should I care about that? Because, for better or worse, Information Management is my chosen profession, and as an ethical professional I want to see higher standards of professionalism. Nothing exceptional. It’s quite straightforward.
Just consider this dialogue from the absolutely fabulous British comedy show, Absolutely Fabulous:
Saffie: I’m sorry, mum, but I’ve never seen what it is that you actually do.
Eddie: PRrr.
Saffie: Yes, but…
Eddie: PR. I PR things. People. Places. Concepts…
Patsy: …Lulu.
Eddie: Lulu… I make the fabulous… I make the crap into credible. I make the dull into…
Patsy: …Delicious.
The second notion is that more data will always lead to better analysis[2]. This is certainly not something that has universal applicability. Random sampling has provided quite a reasonable and sufficiently sound basis in the past, so what changed? Just take the old saying of opinion poll analysts “If you don’t believe in random sampling, the next time you have a blood test, tell the doctor to take it all”.
Added to that, I am always a little wary of people claim that they can help people do a ‘better job’, and not only for the degree of arrogance that it typically conveys.
Thirdly, there is a marked tendency to dismiss evolving best principles in information architecture and management as being unfit for the Big Data revolution[3], and incapable of adapting to increasing volumes, velocity and variety of data. A claim that is both appealing and absurd. Not for nothing did Bill Inmon take people to task[4] for even suggesting that Big Data should somehow replace Data Warehousing as the preferred means to support strategic and tactical information needs.
Fourthly, do all organisations need to maximise their use of all of their data? There are parts of the Big Data campaign that attempts to bludgeon all and sundry into submitting to the Big Data mantra[5]. Entire industrial sectors are railed against for not embracing the faith. Take the energy industry, for example, parts of which have been analysing very large data sets for decades. Did these oil companies and other energy companies pre-empt the Big Data revolution or did Big Data hype find something tangible to attach itself to?
As Dan Ariely so succinctly put it: “Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it…”
So, in fifth place in my list of Big Data issues[6] is the notion that we must accept what is for all intents and purposes a conscious disregard for ‘some’ of the legal requirements pertaining to data protection and data privacy[7], and that ethical considerations regarding the use of data are somewhat irrelevant.
For these five issues alone we should welcome with open arms the overdue demise of the Big Data campaign. With this, we can forge ahead and continually seek to incorporate new data requirements and information supply frameworks into a well-architected, well-engineered and evolutionary approach to the timely, adequate and appropriate supply of data and information.
So, just to wrap up, let’s revisit some old chestnuts from the Big Data circus:
Big Data is dead. Long live DW 3.0 and comprehensive Information Architecture and Management.
In subsequent blog pieces I will be sharing my views on the evolution of information management in general, and the incorporation of ‘Ad hoc Speculative-Predictive Analytics’ into well architected mainstream information supply frameworks for primarily strategic and tactical objectives.
As always, please share your questions, views and criticisms of this piece using the comment box below. I frequently write about strategy, organisational, leadership and information technology topics, trends and tendencies. You are more than welcome to keep up with my posts by clicking the ‘Follow’ link and perhaps even send me a LinkedIn invite. Also feel free to connect via Twitter, Facebook and the Martyn Systems website.
For more on the topic, check out my other recent posts:
[1] Buytendijk, Frank. Gartner. Hype Cycle for Big Data, 2014. https://www.gartner.com/doc/2814517/hype-cycle-big-data-
[2] Big Data and competition – data-rich does not mean dominant. Global Data Hub. http://www.taylorwessing.com/globaldatahub/article_big_data_dominant.html
[3] Eckerson, Wayne. Big data vendors should stop dissing data warehouse systems. http://searchbusinessanalytics.techtarget.com/feature/Big-data-vendors-should-stop-dissing-data-warehouse-systems
[4] Inmon, W. H. Big Data or Data Warehouse? Turbocharge Your Porsche – Buy An Elephant. http://www.forestrimtech.com/big-data-vs-data-warehouse
[5] Gault, Sebastian. How oil and gas firms are failing to grasp the necessity of Big Data analytics. Alberta Oil. http://www.albertaoilmagazine.com/2014/10/big-data-blowout/
[6] Marr, Bernard. The Big Data Economy: Here’s What You Must Know. https://www.linkedin.com/pulse/big-data-economy-heres-what-you-bernard-marr
[7] Does Big Data Mean the End of Privacy? – http://www.dataversity.net/big-data-mean-end-privacy/
File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro
19 Friday Dec 2014
Posted in Data Warehouse, Data Warehousing, EDW
“A thing is not necessarily true because a man dies for it.”
Oscar Wilde
For well over two decades one of the most talked about benefits of Enterprise Data Warehousing (EDW) has been that it gives us a single source of business truth.
In the past, people working in the field of Information and Communications Technologies, in common with other industrial and technological sectors, have striven to maximise what could perhaps be described as a relatively flexible ‘one method fits all’ approach to providing services and products.
In the field of Information Management the search for a harmonised, sanitised and superficially conformed view of business data has achieved its highest form of expression in Enterprise Data Warehousing.
So what led us to view EDW as being a valuable asset in the corporate toolbox?
In short, EDW met a real and present need in the marketplace and in the workplace. It achieved success precisely because it wasn’t a technology looking for a solution but a technology based solution driven fundamentally by business demand and market needs.
The business imperative was simple; businesses wanted a single accurate, reliable and trustworthy view of corporate data, a true reflection of the state of the enterprise. EDW arrived on the scene to meet that demand.
An opportunity arises, the business moves, the market reacts, and the data warehouse captures the events. With the solid foundations of a Data Warehouse, we can report the past, monitor the present and try to predict the future.
Now, the suggestion that a database can contain ‘truth’ is a bit of an contentious claim, no matter how factual the data in the database might be, or how ‘open to interpretation’ the term “truth” is.
We should take arbitrary claims of truth, which form part of a wider social phenomenon, with a pillar of salt. It’s not our fault if we are suckered by words; after all, we are the day to day consumers of spin, a product of the all-pervasive and fashion conscious media industry. Term abuse is no less prominent in my own field of Information Technology (IT), where we have a proven tendency to openly embrace and extend the fast and loose, the tendentious and illusory, the marginal and transient, of old worlds and new.
However, if we stick to the basics of this debate, what is significant about the ” single source of truth” claim is that all too often reality does not live up to the promise. In some instances, business people disagree that there are automatic benefits for a corporation that implements a Data Warehouse, and often the critics have very good reasons for taking the positions they do. However, this should not detract from the proven fact that Enterprise Data Warehousing, when done right and for the right reasons, has the potential to deliver real business benefits.
One of the most popular storylines used to highlight the self-inflicted pain of not having a “single version of the truth” when it comes to corporate data, involves a hypothetical business meeting and two possible scenarios – one terribly bad of course, and the second a thinly veiled representation of business data nirvana, if one can imagine such a thing.
In the first of the two scenarios, the nasty scenario, an executive management meeting is called, and like so many management meetings, it triggers a demand for departmental statistics, forecasts, tactical reviews and position reports; a set of requirements that, by twists and turns, evolves into everyone’s favourite nightmare.
“My sources are unreliable, but their information is fascinating”
Ashleigh Brilliant
I am at the head offices of McWidget Associates, a regular corporation in a regular corporate climate. McWidget Associates holds a regular executive meeting, in which heads of marketing, finance and production, and so on, present and discuss business performance and profitability statistics. As I mentioned, in this, the first of our scenarios, we see the redeployment of whole teams of people in their respective departments, frantically working away preparing the reports and the statistics for the executive meeting. The work is essentially long, arduous and tedious.
Whilst the traitorous clock on the wall marks internet time, a loyal band of employee’s battle with complexity, fatigue and monotony in order to produce their departments very own pixel-perfect executive reports.
As the time marches on, reports are constructed based on data painstakingly extracted from departmental computer systems, spread-sheets and personal files. This intellectually mind-numbing productive output is then checked, double-checked and checked again; the mantra is that if you must be wrong just make sure you are consistently wrong, and that your consistency has been validated.
The output of a small Amazonian rainforest is indelibly etched with data that has been massaged by many minds, cleaned for accuracy and processed by each department’s carefully developed statistical analysis programs.
Several minutes before the meeting is due to start, the finalised reports are reviewed, approved and then reprinted in colour for posterity. All of this output is the tangible result of immense effort, the absence of real asset management, a failure to adopt good business practices and a lack of foresight.
Nervous executives, armed with their data, rush off to the conference room. The scene is set, and the game begins. As is predictable, a short way into the meeting, an intense and vitriolic argument breaks out between some of the participants; accusations of cooking the books, sexing up statistics, overplaying the competition and downplaying success, get bandied about, as one report after the next is trashed, and the veracity of certain data is hotly disputed.
Now, there are various reasons why meetings end in verbal bust ups, but in this story, we will keep it simple. Anomalies between the sales and revenue data in the reports presented by each department head have caused the friction and fractiousness. No one agrees on the exact sales or revenue figures for the first quarter. It has become a meeting in which three distinct versions of the truth are discussed but never reconciled; of course, all of this is elegantly framed and presented using charts, more charts, animated graphics and other favourites.
The bottom line result of this “confusion of data” is that everything is a phenomenal mess. The department heads start getting overly snippy and overtly political, unwisely coffee is served, and a PowerPoint battle ensues, in which the CEO is left wondering “what on earth is going on”. In short, the meeting becomes an exercise in engineered confusion, worthy of Machiavelli. All because Marketing took its data from the marketing system, Finance from the finance system and Production from the production system, and so on and so forth. Therefore, the basic data, or better said, the differences in the basic data, from department to department, simply failed to pass the truth and reconciliation test.
Now, the story does not end there. The first scenario, the nasty scenario, leads, almost predictably, to a second scenario, the happy scenario, one that illustrates quite nicely that the smart organisation and its supply chain of information would sensibly and naturally leverage Enterprise Data Warehousing.
“Sameness is the mother of disgust, variety the cure”
Francesco Petrarch
In my second and shorter scenario, you see the director of the marketing department in her glass, steel, white and black minimalist office. An office that is spacious, cool and efficient. We have from the madness of incongruence to a sane and workable environment in a sane and workable world, courtesy of Knoll International, Bauhaus, B&O, Apple and Elle.
The director is discussing new and dynamic business opportunities with members of her team, who are as equally dynamic and full of ideas as the landscape and culture in which they work.
In that moment, an electronic message pops-up on the screen of a matt platinum tablet, a simple reminder that the interdepartmental meeting is due to start in fifteen minutes time. In a couple of virtual mouse clicks, we see a simple dialogue window, the director’s assistant selects a report type from a drop down menu, then a quarterly period, and then clicks on the “generate report” button. In an instant, the Business Intelligence application retrieves data from one of the data marts in the Enterprise Data Warehousing infrastructure, this then leads to very rapid in-memory calculations and formatting, and the final report directly appears on the tablet screen in spread-sheet format. The team quickly reviews the report, and adds any contextual information that is required; the report parameters are stored for use later. The director picks up her limited edition of Ive’s swish little gadget and heads for the meeting; all the key department heads have repeated this exercise.
People wander into the meeting room. Small talk ensues, pleasantries exchanged, and the mood is relaxed and professional.
The meeting develops in a perfectly acceptable manner, as we would expect; nobody disputes the veracity of the data, just the rationale and logic of some of the interpretations.
People ask questions about strategy and operational tactics; this is a forum to raise issues of concern, and immediate access to information ensured that they dealt with on a case by cases basis, effectively and efficiently.
In any case, in our second scenario the executives have access to the Enterprise Data Warehouse infrastructure directly from their laptops, tablets and assorted mobile computing devices. So even when questions of detail arise they can be answered there and then, either qualitatively, by interpretation of the facts, or quantitatively, by drilling down through the underlying data, or indeed, through the creation of interactive “what if” scenarios.
What is important in the second scenario is how it highlights the benefits of a single source of the truth. It’s all about knowledge and trust, knowing that you have the data and trusting the validity of that data that you have, across the enterprise.
The message is simple. Enterprise Data Warehousing can take you and your organisation, towards total flexibility, total information availability, and total coherence and cohesiveness. In short, the second scenario demonstrates the power of adequate, appropriate and timely information. Without this capability, an organisation is just not going to be as agile or as responsive as it might otherwise be.
“I never read a book before reviewing it – it prejudices a man so”
Sydney Smith
Of the two scenarios, the second is clearly preferable. Having the choir sing from the same music sheet is preferable to constant dissonance or unhelpful improvisation. But, one nagging question always remains, is it truly desirable to have the members of the choir always singing in perfect harmony?
There are a number of business people, such as accountants, administrators and generalists, who have the occasional tendency to dismiss the idea of “a single version of the truth”. Typically, their dismissal is based on unconvincing grounds – a misunderstanding of the data warehouse process, that are truly unimportant, are simply unsound – aversion to the questioning of financial accounting practices and certain data, or that are just objections that do not apply.
Constructive criticism can be helpful, but some criticisms of the benefits of Enterprise Data Warehousing are so facile as to be absurd. One example that illustrates the immaturity that sometimes erupts on the data-warehousing scene is an example of an accountant, who is supposed to have claimed that, “you can never have one report that will be everything to everybody”. This criticism of the “single source of the truth” just goes to exemplify the point, that both the exaltation of truth in Enterprise Data Warehousing and the subsequent criticism of the claim both miss the point of Enterprise Data Warehousing.
Maybe, instead of using the term “single version of the truth”, it would simplify matters if we used the term “trusted source of integrated enterprise data”. This does not add anything to or take away anything from the value propositions of the data warehouse paradigm, but it may serve to anchor sanity in a safer haven, especially in these curious times, where the usage of super-sized claims, of dubious validity, is imperceptibly going out of fashion.
Using an Enterprise Data Warehouse to impose a single clean source of integrated enterprise data, to be used for reporting, collaboration, planning, and data analytics and decision making, is very useful. Nevertheless, it is also possibly a mistake to believe that the story starts and ends there, and to assume that an Enterprise Data Warehouse constitutes only one version of the truth. For example, if an Enterprise Data Warehouse is used as the data integration hub for regulatory reporting, compliance reporting, taxation reporting, accounting reports and management reports, and so on and so forth, one may already start to see that a single version of the truth might not meet all of these requirements.
However, when critics of the “single source of the truth” catchphrase use it to leverage their criticism of the Enterprise Data Warehouse paradigm in general, they are probably being somewhat easy on the pocket in terms of their degrees of accuracy. An Enterprise Data Warehouse is the perfect hub for integrating enterprise information; and it is rarely deployed as the last repository in the reporting chain, as this is typically the role of a Data Mart. This is precisely where the arguments of certain critics of the Enterprise Data Warehouse paradigm also fall down, as a lot of financial reporting is performed using data stored in Data Marts or more frequently in Operational Systems.
“What sets worlds in motion is the interplay of differences”
Octavio Paz
What I want to achieve with the focus of this discussion is to help bring clarity to the muddied waters of the concept of having a single source of the truth, and having done that, to explain its relationship to contemporary Enterprise Data Warehousing.
In order for an enterprise to benefit from an Enterprise Data Warehouse, in ways that allow for divergences in reason, logic, interpretation as well as data, the organisation and the members in the organisation that use the data, must have reached a certain level of maturity with respect to divergences, what those divergences represent, and what they will effect.
That is, the organisation must have a clear and good understanding of its data. Not just in order to be able to cope with the associated pitfalls, but also to be able to grasp the benefits of the value obtained from certain divergences that may be present in differing viewpoints and interpretations of data. This is all very academic, because there is no single commercial enterprise that is close to achieving this level of maturity in its use of data and information. Expressed in a different way, the organisation that has reached this level of maturity has yet to become a reality.
For the moment, the Enterprise Data Warehouse model, together with its Data Mart component, is the most versatile and cost effective method we have for providing information for a range of reporting and analysis requirements. More to the point, it works. So, if it makes more sense to describe the Enterprise Data Warehouse as “single source of integrated enterprise data” rather than a “single source of truth”, then we should do that, and with the added assurance that, if we do data warehousing the right way, and for all the right reasons, then accruable value will be intrinsic.
So, at the end of the day we must cater for two requirements; a single version of the truth, and multiple versions of the truth. Technically this is not such a big issue, but conceptually it is likely to give quite a few technicians a headache, so, probably the best thing to do is to give the responsibility for this conundrum to someone who can comfortable manage and reconcile seemingly conflicting and disparate dialectics, such as someone with a solid grounding in classical philosophy.
At the end of the day, the occasional criticism that the Enterprise Data Warehouse model doesn’t provide what isn’t actually needed is pointless, sterile and quite irrelevant. Rather like complaining about not being able to make a decent cup of tea with a Porsche.
File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro
16 Tuesday Dec 2014
Posted in Consider this
Tags
Even before the Duke of Gloucester had berated Edward Gibbon for his “thick, square book”, the encouragement of ignorance was already a powerful force in the English speaking world.
In a young USA, both Thomas Jefferson and John Quincy Adams, were accused of being unsuitable for political office due to their intellectual pursuits.
Much later, Richard Hofstadter, in a classic study of the plight of intellectualism in the USA, noted that “It seemed to be the goal of the common man in America to build a society that would show how much could be done without literature and learning–or rather, a society whose literature and learning would be largely limited to such elementary things as the common man could grasp and use”.
Indeed, the widely popularised and populist Andrew Jackson, seventh President of the United States, delighted his followers with witty observations of the type “It’s a damn poor mind that can only think of one way to spell a word”.
In France, Sarkozy’s “Cultivated Anti-Intellectualism” used to be something of an embarrassment, given that the Gaullist right were never “quintessentially vulgar” or anti-intellectual. That was in sharp contrast to the situation in other G8 countries, where “bread and circuses” are served up as a daily substitute for political engagement.
Indeed, if people consider that a bit of rad chit-chat on the old iDog ‘n’ Bone is a form of political engagement, then where are we?
Which leads me to Isaac Asimov, as dead as he is, who thought that there was a ” false notion that democracy meant that ‘my ignorance is just as good as your knowledge'”.
Meanwhile, in the UK, a former Education Secretary told an audience, including “gifted and talented” children, that academic success is to be celebrated as much as sporting achievement, adding “being clever is sometimes seen as a term of abuse, for example: ‘Too clever for your own good’.”
Unfortunately this was the very same person who was also the cabinet colleague of a man who blamed the French for the Iraq War, because, according to this particular ‘brain box’ they had caused the war, because, they voted against it.
It didn’t stop there, because the justification for using the “blame the French” ruse was so mangled and indecipherable, that comedian Mark Steel was lead to remark that “the Government would have done better to have their policy explained by Po of Teletubbies”.
Anti-intellectualism extends to the political blogosphere, where the established principle seems to be that the way to attack the opposition is not to address it – because that would involve reading and comprehension and joined-up-writing. But, conversely, to mock, ridicule, trivialise, deny, invent, deflect, and misrepresent, is an acceptable opening gambit, which when all else fails, can be replaced by attack using blunt adjectives and deceitful indignation.
Of course, there is nothing new in that. Stewart Lee is on record as saying that “you can prove anything with facts, can’t you”, since, when an argument cannot be sustained by reason or evidence, then gut instinct and prejudice become your friends.
It seems that giving a reasoned and well placed opinion is actually a transgression. Something that decent chaps just don’t do. As if political life was a football match, between right and wrong.
The keeper in the red, effortlessly blocks a weak shot, boots the ball right up the field to his mate cruising in the opposition’s penalty area, who Gareth Bale like, sends the ball crashing into the back of the opponents net.
Of course, when things don’t go to plan, Team Right, in the blue strip for those watching in black and white, surround the man in black with appeals to reason.
“Eh, Referee! The other lot are cheating again. That argument was well offside”.
This cultivated anti-intellectualism is not confined to politics.
Just look at the manufactured hype and faddishness surrounding Big Data, and the wilful ignorance of those who truly want to believe.
Anti-Intellectualism is not clever or funny, and ultimately it is divisive and destructive.
Thanks for reading.
File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro
16 Tuesday Dec 2014
Tags
Apologies to the late Benny Hill
You could hear the dollar fall, then it crashed upon the ground
And the chatter from the White House as they spun, around and ’round
And he glided into Wall Street, his scam beneath his vest
His name was Bernie, and he sold the slickest secrets in the west
Now Bernie loved his Lipstick, a place where mom’s the word
He worked alone on Third Avenue, at 53rd at Third
They said that he was just da’bom[1]; they were greedy, vain and chic
But Bernie got his kickbacks[2] there, five days in every week
They called him Bernie, Bernieeeeeeeeeee!
And he finessed the slickest Hedge Fund in the west
She said she’d like to work in funds, he said, “All right, braveheart”
And when he’d finished fiddling, he loaded up his chart
He said, “D’you want to leverage? ‘Cause leverage is best”
She says, “Bernie, I’ll be happy if there’s money left to invest”
That tickled old Bernie, Bernieeeeeeeeeee!
As he pimped the slickest Hedge Fund in the west.
Now Bernie had a rival, a governmental man
Called quickstep Chris from Harvard Yard[3], and he drove S.E.C.’s[4] van.
He tempted her with his oversight, regulator’s feet of lead
And when she seen the size of his compliant eyes, Lipstick trader placed a spread[5]
She almost sold on his insider tips and he said, “If you put me right,
You’ll have issues every morning, dividends every night.”
He knew once she sampled his laissez faire, he’d have his hedging way,
And all Bernie had to offer was a NASDAQ loss a day.
Poor Bernie, Bernieeeeeeeeeee…
And he rode the slickest Hedge Fund in the west.
One bell time Chris copped Bernie, doing deals outside her floor,
It drove him mad to find them trading even after half past four.
And as he jumped up from his chair, hot issues through his veins did course,
And he went across to Bernie’s trades and didn’t half kick his Bourse[6].
[Of course, it was his horse]
Whose name was Ponzi, Ponziiii…
And he schlepped the slickest Hedge Fund in the west.
Now Bernie rushed out onto Wall Street, prospectus in his hand,
He said, “you wanna subscribe to Lipstick you’ll pay for her like a man.”
“Oh why don’t we place bets for her?” he derisively replied,
“And just to make it interesting we’ll do some shorting on the side.”
Now Bernie dragged him up from F Street and beneath the Times Square clock,
They stood there face to face, and Chris went for his stock.
But Bernie was too quick, things didn’t go the way Chris planned,
An unexpected corporate-action sent it spinning from his hand.
Then Lipstick rigged a Chinese Wall to keep them both onside
But Bernie, made a haircut deal and subprime caught him underneath his pride.
And as he looked up at BASEL II, an OFAC hardened trust
Of a next day trade, caught him in the PEP, and Bernie got ‘da bust’[7].
Poor Bernie, Bernieeeeeeeeeee…
And he blagged the slickest Hedge Fund in the west.
Bernie was only just 60, he didn’t wanna lie
But now he’s stopped making deliveries to that Hedge Fund in the sky.
Where the investors are quite stupid, and regulation banned,
And the trader’s life is full of dosh, in that alternative land.
Yet Lipstock’s needs are manifold, and soon she collared Chris
Strange things happened on their redemption day, as they talked a ‘loada pish’[8]
A KYC oversight? Or old secrets out the gate?
Ole Bernie’s market making? A rattled exchange rate?
They won’t forget Bernie, Bernieeeeeeeeee…
As he ‘vanished’ the slickest Hedge Fund in the west.
[1] Quite jolly good.
[2] A return of a part of a sum received often because of confidential agreement or coercion.
[3] Harvard Yard, in Cambridge, Massachusetts, is a grassy area of 22.4 acres enclosed by fences with twenty-seven gates. It is the oldest part of the Harvard University campus, its historic centre, and its modern crossroads.
[4] Securities And Exchange Commission
[5] An options position established by purchasing one option and selling another option of the same class but of a different series.
[6] A market organized for the purpose of buying and selling securities, commodities, options and other investments.
[7] Revealed
[8] Not of a superior quality
File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro
15 Monday Dec 2014
Posted in Consider this
Tags
For their feet run to evil, and make haste to shed blood.
Proverbs 1:16
Fools rush in where angels fear to tread.
Alexander Pope, An Essay on Criticism
I’m not a psychologist, and people who read my blog will know this for sure. But I would like to examine the notion of wanting something so much that at best we don’t enjoy it when we obtain it, or worst still, that our eagerness means that we will never obtain that which we truly want.
In his way, the great stoic Seneca of Cordoba warned us against what I would call “destructive eagerness” and its influence. Oscar Wilde also lamented that “each man kills the thing he loves”. Whilst Charles Caleb Colton commented that “patience is the support of weakness; impatience the ruin of strength”.
I have been in the field of Data Warehousing ever since Bill Inmon typed the last word of his ground-breaking and defining tome, Building the Data Warehouse, and I am proud to say that I have been on the same page as Bill with respect to Data Warehousing ever since.
Many Data Warehouse projects have not been the resounding successes that they could have been. Many people give reasons why these projects fail. Most of the reasons given are really peripheral excuses. But that’s human nature.
In the early days of Data Warehousing I was involved in implementing the Inmon approach. An approach that worked like a dream. We weren’t dealing with high volumes of data, but the business benefits didn’t come from volumes of data, but form the quality, relevance, timeliness and usability of the information supplied. The computer industry didn’t take much interest in us because we didn’t need massive big iron, and we were at the time (1993/1994) in an oddball sort of project for Europe.
As use of Data Warehousing expanded at Compaq, iteration by iteration, we were on track.
Then things started to heat up when we began to think about data volumes approaching 1 terabyte.
All of a sudden – okay, not so sudden – the traditional hardware vendors saw an opportunity, and the less traditional relational DB vendors as well had mega-buck dollar signs in their eyes.
Data Warehousing might eventually need a lot of hardware and quite a few software licences.
So far, so good.
Then came an inflexion point.
Organisations started to be aware that a small band of corporations were having success with their Data Warehouse projects, and, well, were understandably curious.
At least two new expectations had been set.
For IT vendors, especially the big guys and gals, it was an opportunity to sell serious infrastructure. Especially as Telco’s had started to follow the previous experiences of big retailers in the use of dimensional model based analysis and reporting.
For organisational IT departments it spelled out the need to react, to respond to a mounting interest amongst senior management with this new trend called Data Warehousing.
There was only one problem.
Both had forgotten about the fundamental imperatives that should drive useful, sustainable and justifiable Data Warehousing.
For IT departments it spelled the need to do something. For IT vendors it became the need to maximise sales; that is, also to do something.
From there on in things went awry.
It didn’t matter that the defining characteristics of Data Warehousing were ignored, it was enough to be doing something, to be building something, to be selling a lot of something, anything. Activity, buying and selling was of the essence.
Of course, IT bought, built and presented (sometimes), and in many cases were told by business stakeholders that their gifts were useless, or worse than useless.
How tragic is that?
The IT vendors were so eager to sell, and the IT departments so eager to please, that their joint eagerness became counter-intuitive and destructive.
Of course, in most cases they weren’t doing Data Warehousing, they were simply ‘experimenting’, and calling those ‘experiments’ Data Warehousing.
This was how Data Warehousing was fragged by the friendly fire known as “destructive eagerness”.
Both the IT vendors and the corporate IT departments desperately wanted to do the right thing. But there were hidden risks, false friends and other complications.
If I look back into the history of IT I see so many examples of “destructive eagerness”.
Maybe you can also recognise this phenomenon in your line of business.
For I would love it – and I’m sure others would as well – if people would ‘pony up’ their own experiences in this respect. Because as George Santayana stated “Those who cannot remember the past are condemned to repeat it”.
In future blogs, and with your help and guidance and input, I will expand on this theme.
Thanks for reading.
As always, please share your questions, views and criticisms on this piece using the comment box below. I frequently write about strategy, organisational, leadership and information technology topics, trends and tendencies. You are more than welcome to keep up with my posts by clicking the ‘Follow’ link and perhaps even send me a LinkedIn invite. Also feel free to connect via Twitter, Facebook and the Cambriano Energy website.
For more on the topic, check out my other recent posts:
File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro
