If you enjoy this piece or find it useful then please consider joining The Big Data Contrarians: https://www.linkedin.com/grp/home?gid=8338976

Many thanks, Martyn.

Big Data, together with Cloud computing, the Internet of Things and Machine Learning, are topics that are very much to the fore in contemporary trends in Information Management. But is Big Data really the revolution that people have been waiting for or is it simply about the next steps in the evolution of business data architecture and management? 

Whilst it is true that we are increasingly generating more and more data, do we really need to rethink our whole approach to data demand and data supply, or can we build on the best of what we have already accomplished in order to move with intelligence, rigour and stability to the next level?

We are told that all organisations, small, medium, large and gigantic, will be overwhelmed with an existential threat if they don’t recognise and act on the fundamental need to leverage all data – massive data, in all formats, all volumes and all qualities. From wherever it may come from, legally or legally.

But is this really the case? Are we being urged to action by caring, principled and disinterested good data Samaritans? Or are we being railroaded into a shallow, anti-intellectual and mass-reactionary fervour that will run ultimately against the grain of our own best interests? Whether as individuals or as stakeholders.

I am not entirely convinced by the breadth or depth or intellectual veracity of the claims for Big Data, which frequently border the hyperbolic and the absurd, but I also think it is wise to be forearmed as well as forewarned.

That is why I am encouraging a discussion about where Big Data and Data Science fit into a much larger picture of business information supply and demand.

To begin at the beginning

There are many dimensions to business data and information, and here I will touch on some of those dimensions here, albeit with a broad brush.

There are also many potential sources of data; these can be divided into two broad dimensions: analogue data and digital data.

Analogue data – Any type of data not stored in a digital format. This covers the data found in analogue form.  For the purpose of this piece this dimension is overlooked.

Digital data – Any type of data stored in a digital format. The source may be a conventional database, a log file, and audit trail, a spreadsheet, document, presentation, project plan or other similar source.

What is being contemplated is data that could help us to interpret history; manage the present; and, plan for the future.

Past – What happened and what can we learn from it?

Present – What is happening and what, if anything, should we do differently?

Future – What might or might not happen and what might we do or not do if this happens, or to prevent this from happening?

If the data in focus cannot help us with any of these dimensions does that data really matter?

The third dimension of data that I want to identify concerns place (internal and external):

Internal data – This is the information that belongs to the organisation and is obtained from internal sources, such as operational systems.

External data – This is the data from external sources, such as market data providers. that an organisation can legally and legitimately acquire and use.

And finally, for the purpose of brevity, the dimensions of demand and supply:

Demand driven data – This is when business is the driver for the flow of information. Data is delivered because there is a business demand. In simple terms this is about giving people what they want, continuously.

Supply driven data – This is about delivering data and information based upon what is available. If done on the right scale and with the aim of creating new internal markets for data, then it can be beneficial. This is where IT needs to be far more sales, marketing and Ad-land oriented.

But what do all of these considerations mean to a managers need for adequate, appropriate and timely information? Let’s take a look.

The following diagram illustrates the overall drivers for the Information Supply Framework.

From the right, the consumers and prospective consumers of data and information create the demand for information and data.

The middle process of ‘data processing, enrichment and information creation’ strives to meet the business demands and also ‘provokes’ business demands for data and information.

From the left the data sources provide data to meet real and secondary demands.

So how do we ‘make it happen’?

How do we try and obtain the business advantages supposedly accruable to Big Data without succumbing to yet another anarchic information management aberration that will take us one step forward, and two steps back?

Let me explain.

Is this the sort of diagram that gives you a warm and fuzzy feeling?

Of course not, it’s a mess. Print it out on a plotter and place it on the office wall, by all means. You can even call it art, but it’s also a recipe for data bedlam hell.

Now contrast that colourful fun-loving anarchical approach to something I prepared earlier.

There, isn’t that better?

It’s clean, cohesive and coherent. It’s a rational, structurally sound and practical model to support the business demands for data and for the speculative supply of data. The model is based on unifying strands of business data requirements, (with specific regard to operational, tactical and strategic aspects,) into to an integrated, agile and flexible whole.

In short, it is a coalescing model for business data and information.

That’s all folks

Is it absolutely unnecessary in this day and age to take a maverick, piecemeal and DIY approach to the provision of non-traditional business data and information?

No, evidently it’s not.

There is no rational business reason to use speculative, quick and dirty – and characteristically ‘throw away’ approaches to data supply, when quick, clean, useful, maintainable, cost effective and usable alternatives are available.

The Information Supply Framework provides a strategic approach to meeting the data and information needs of an organisation. It accurately reflects business process, it can change in step with the changing needs of the organisation, and it is capable of handling the emerging requirements associated with ever increasing data volumes, data variety and data velocity.

Simply stated, the DW 3.0 Information Supply Framework provides a nascent opportunity for businesses to easily transition to the inclusion of more formal and rigorous methods of statistical analysis, analysis that can now be powered by the inclusion of new, alternative and complementary sources of data.

Naturally the choice is yours. But ensure that you weigh up your options with your ‘eyes wide open’, or it may well all end in tears.

Many thanks for reading.

If you enjoyed this piece or found it useful then please consider joining The Big Data Contrarians: https://www.linkedin.com/grp/home?gid=8338976

Many thanks, Martyn.