Tag Archives: Data Monetisation

What’s all the fuss about Dark Data? Big Data’s New Best Friend

10 Tuesday Mar 2015

Posted by in All Data, Big Data, Consider this, dark data, Good Strat

≈ Leave a comment

Tags

, , , , , , , , ,

What is Dark Data?

Dark data, what is it and why all the fuss?

First, I’ll give you the short answer. The right dark data, just like its brother right Big Data, can be monetised – honest, guv! There’s loadsa money to be made from dark data by ‘them that want to’, and as value propositions go, seriously, what could be more attractive?

Let’s take a look at the market.

Gartner defines dark data as “the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes” (IT Glossary – Gartner)

Techopedia describes dark data as being data that is “found in log files and data archives stored within large enterprise class data storage locations. It includes all data objects and types that have yet to be analyzed for any business or competitive intelligence or aid in business decision making.” (Techopedia – Cory Jannsen)

Cory also wrote that “IDC, a research firm, stated that up to 90 percent of big data is dark data.”

In an interesting whitepaper from C2C Systems it was noted that “PST files and ZIP files account for nearly 90% of dark data by IDC Estimates.” and that dark data is “Very simply, all those bits and pieces of data floating around in your environment that aren’t fully accounted for:” (Dark Data, Dark Email – C2C Systems)

Elsewhere, Charles Fiori defined dark data as “data whose existence is either unknown to a firm, known but inaccessible, too costly to access or inaccessible because of compliance concerns.” (Shedding Light on Dark Data – Michael Shashoua)

Not quite the last insight, but in a piece published by Datameer, John Nicholson wrote that “Research firm IDC estimates that 90 percent of digital data is dark.” And went on to state that “This dark data may come in the form of machine or sensor logs” (Shine Light on Dark Data – Joe Nicholson via Datameer)

Finally, Lug Bergman of NGDATA wrote this in a sponsored piece in Wired: “It” – dark data – “is different for each organization, but it is essentially data that is not being used to get a 360 degree view of a customer.

Say what?

Okay, let’s see if we can be a bit more specific about the content of dark data?

Items on the dark data ticket include: Email; Instant messages; documents; Sharepoint content; content of collaboration databases; ZIP files; log files; archived sensor and signal data; archived web content; aged audit trails; operational database backups – full and incremental; roll-back, redo and spooled data files; sunsetted applications (code and documentation); partially developed and then abandoned applications; and, code snippets.

Most importantly, dark data is data that is not actively in use, is underutilised, or is something else. Seriously.

What can you do with it?

So, the conclusion that some have come to is this: there is a vast collection of data in various formats waiting to be monetised.

Personally, the idea that really grabs my attention is the potential ability to do novel forensic research on email. If only to find out what happened in the past.

For example, maybe it would be fascinating to see how significant challenges were identified, flagged and discussed; how strategic responses to those challenges were formulated, chosen and executed; and, how the outcomes of all of that process were reflected in email communications.

I think that this line of work can be very interesting for some people, and that interesting insights may be uncovered, but I would hate to have to put a tangible value on it, if only to avoid adding to the already galactic magnitudes of nonsense and hype surrounding certain data topics.

There are other more mundane uses of dark data.

Imagine that you are just about to embark on a Data Warehouse project (you really are a late adopter aren’t you), and you want establish a base collection of historical data. Where do you get that historical data from?

Right! Operational databases are not characteristically used to store significant amounts of historical reference data and historical transactions beyond a certain time window; there are performance and other reasons for keeping OLTP systems as lean as possible, so, initial loads of historical data is typically recreated in the Data Warehouse from backups, audit trails or logs.

Dark data and data governance

You don’t need a Chief Data Officer in order to be able to catalogue all your data assets. However, it is still good idea to have a reliable inventory of all your business data, including the euphemistically termed Big Data and dark data.

If you have such an inventory, you will know:

What you have, where it is, where it came from, what it is used in, what qualitative or quantitative value it may have, and how it relates to other data (including metadata) and the business.

What needs to be kept, and for how long, and what can be safely discarded, and when.

The risks associated with the retention or loss of that data.

If you don’t have such a catalogue and have never done a data inventory then a full data inventory and audit seems to be your new best friend.

What does it mean?

Simply stated, you may have dark data that has value, or it may be a simple collection of worthless digital nostalgia. But if you don’t know what you have, it may pay to find out what’s there, and if necessary, to let it go.

There is no point in hoarding unneeded and unwanted rubbish data. That is simply not good data management.

Finally a word on all the fuss surrounding dark data.

Failure to monetize when there is value to be obtained from dark data is one thing, claiming that value can be invariably obtained whilst actually not knowing what the data is, or how it could be monetised, is just adding to the mountain of data related ‘nonsense and hype’ doing the rounds these days. Please consider not adding to that mountain.

That’s all folks

British Rail, the national UK rail Company, used to be notorious for the number of delays and cancellations to services, and their reasons for failing to meet their obligations became stranger and stranger.

In winter, it would snow and there would be problems. And people would ask ‘how come you couldn’t deal with the snow this year, we’ve had snow for centuries?’ And back came the answers ‘Yes, Sir, but this year it was the wrong type of snow’. In autumn (the fall), it was ‘the wrong types of leaves, and ‘the wrong type of rain’, and in Summer, the ‘wrong type of sunshine’ and so on and so forth.

I hope this will not be the excuse from the Big Data and dark data pundits and punters when the much-vaunted and ‘almost’ guaranteed monetisation isn’t frequently realised.

‘Of course Big Data gives you big dollar benefits, it was just littered with the wrong type of data’ or ‘you just weren’t trying hard enough’.

Many thanks for reading.

Consider this: Does all data have value?

30 Friday Jan 2015

Posted by in Assets, Consider this, Data governance, data science, Good Strat, Martyn Jones

≈ 1 Comment

Tags

, , , , , , , , ,

Header1

To begin at the beginning

You can use all the quantitative data you can get,

but you still have to distrust it and use your own intelligence and judgment.

Alvin Toffler

There is a touching belief that all markets are rational and that a company’s value is accurately reflected in the current share price multiplied by the total number of shares in circulation.

The thing is, this doesn’t square up with formal accounting practices and financial reporting, as they now stand. So, there is a sentiment that the gap between capitalisation and accounting valuation must be due to some less tangible factors, which is true, and for many years it was accepted that aspects such as good will, were not in fact measurable or even easily manageable.

In the nineties, Leif Edvinsson and a team of accounting and finance specialists at Skandia pioneered the accounting and reporting of non-tangible assets, which they called the Skandia Navigator, the focus was clearly on reporting the progress of the creation and use of Intellectual Capital (IC). What the Skandia IC team did, how they went about it, and how they presented their results, (as a supplement to their annual report,) made a lot of good sense.

Now there is a movement to measure, value and manage all data as if it were a highly tangible asset, because, according to some, IT has been incapable of managing data because they don’t know how to measure it and they aren’t particularly well equipped to do so, a claim that is paradoxically made by people who are very much in the IT camp.

But this is not about treating data as an asset but focuses on potential risks due to data loss, data corruption, quality issues, misinterpretation, misuse or whatever. Simply stated, a risk prevented from arising or a risk that is mitigated is not an asset, and risk management is not primarily about rigidly theoretical asset management and vice verse.

Now, I won’t argue that it’s not a good idea to have an idea of the value of business assets, from either a quantitative or qualitative perspective, but I think there are more moderate, coherent and less fundamentalist approaches to the understanding and valuation of data, approaches that are more aligned to contemporary views of the understanding of the value of intellectual capital assets – unstructured and structured. What follows is a brief and high-level view of some potential options.

Data Assets in MOSCOW

What follows is a simple explanatory technique that is part of the DW 3.0 Information Supply Framework approach. It focuses on qualitative aspects of data as an asset. The following diagram provides an overview of foundations of the first phase of this simplified risk, classification and prioritisation approach:

On the horizontal access we have used the MOSCOW classification to break the problem down into simple terms. Here is an example of its configurable application:

MUST – Must treat the data as an asset: If this data is compromised in anyway then it would possibly result in serious negative implications for all (or most) customers and severe financial, contractual and reputation impacts on the business

SHOULD – Should treat the data as an asset: If this data is compromised in significant ways it would result in intolerable negative financial implications for a number of customers and would also lead to negative financial impacts on business

COULD – Could treat the data as an asset: If this data is compromised in significant ways then it could result in financial consequences for the business, but without impacting customer churn

WON’T – Won’t treat the data as an asset: Data that if compromised would not result in financial consequences wither for customers or for the business

It is important to understand that these definitions are examples and that in practice the descriptions and parameters must be determined and aligned in cooperation between the distinct business stakeholders, including business IT.

Data Value Chains

Another aspect that is important to the understanding of ‘data as an asset’ is to be found in data value chains. The following diagram illustrates the DW 3.0 technique used to simplify the attainment of a consensual view of the understanding of this value.

DATA: “Data is a super-class of a modern representation of an arcane symbology.” –Anon

In order for data to be more than an operational necessity it requires context.

Providing valid data with valid context turns that data into information.

Data can be relevant and data can be irrelevant. That relevance or irrelevance of data may be permanent or temporary, continuous or episodic, qualitative or quantitative.

Some data is meaningless, and there are cases whereby nobody can remember why it was collected or what purpose it has.

Taking all this into account we can now ask the pragmatic question: what value does this data have? Which is sometimes answered with a: ‘no value whatsoever’.

INFORMATION: “My sources are unreliable, but their information is fascinating.” –Ashleigh Ellwood Brilliant

In order for information to be created to drive organizational operations and tactics we must have valid interpretations of data.

Providing valid information with valid interpretations potentially turns that information into knowledge.

Information can be correct, partially correct or incorrect; it may be relevant or irrelevant. The validity of information may also be permanent or temporary, continuous or episodic, and qualitative, quantitative or both.

Some information may be relevant, irrelevant or misleading. Information derived from erroneous and incomplete data may be usable, but the outcomes of using that information may be unpredictable.

Taking all this into account we can now ask the pragmatic question: what value does this information have? A question which can sometimes be answered with an ‘it is probably too early to say’.

KNOWLEDGE: This necessarily refers to a subset of ‘business knowledge’ known as structured intellectual capital.

The adequate, appropriate and timely application of knowledge (structured intellectual) requires wisdom; knowledge alone will no longer be good enough.

Providing valid knowledge with valid interpretations potentially turns that knowledge into valid and executable strategies.

Knowledge may be useless or useless; it may be relevant or irrelevant; it may even be wrong, even if we have named it ‘knowledge’. The usefulness of knowledge may also be permanent or temporary, continuous or episodic, qualitative or quantitative.

Some knowledge may be relevant, irrelevant or misleading. Knowledge derived from erroneous and incomplete information may be usable, but the outcomes may be unpredictable. As they might say in Córdoba: ‘Give knowledge to a donkey and it will still remain a donkey’.

Taking all of this into account we can now ask the pragmatic question: what value does this knowledge have? A question which can sometimes be answered with: ‘Isn’t it getting rather hot in here’.

What does it all mean?

To paraphrase George S. Patton, for the purposes of business data, an imperfectly good pragmatic valuation of data executed today is better than a perfect theoretic valuation of data made in the next life.

The valuation of data, information and knowledge is complex and involves many intangibles, and although some may view the data, information and knowledge chain as a closed process, this is not in fact the case, as each step of the process is influenced by a number of factors that fall outside of a simplified view of any theoretical or practical progress from data to wisdom.

When I started in IT over 30 years ago I worked on some of the first large-scale OLTP projects in Europe. On one such project, the development, test and production the databases were continuously replicated. Every day, not one, but ten backups of the data were made, and then shipped to different physical locations.

Thirty years on and some people are saying that IT has never treated data as an asset?

The fact of the matter is that data is managed as an asset. In fact, some IT organizations may be taking exaggerated measures in order to protect the data and information in their care. That some people have issues with the contemporary management of data does not change the facts on the ground, and the perceived shortfall in the commercial exploitation of data is frequently and erroneously interpreted as the absence of asset appreciation and management.

Moreover, even with generally accepted accounting principles there are tangible assets that are either of dubious value or are cost-absorbing liabilities.

Finally, there may also be unintended consequences of an overzealous approach to the financial reporting of intangible assets such as data, and just as the value of tangible assets can be accidentally or deliberately overstated, so too could the value of data, which may well lead to a significant overvaluation of a business. Moreover, it’s not beyond the realms of possibility that big data lakes, data universes and dark data dungeons – the ones that apparently will drive trillion dollar economies – are in fact somewhat worthless.

In this respect I think there is in some quarters an unhealthy fixation on the supposed (and as yet mainly unproven) value accruable from masses of data.

That’s all folks

Steve Jobs didn’t turn Apple around by relying solely on data and information, it was knowledge, and more especially the wisdom of knowing where, when, why and how to apply that knowledge that made the significant difference.

So, until the next time, hold this thought: ‘Google market cap slashed and Facebook in freefall as accountants ask “where’s the big data beef?”… Breaking news!’

Many thanks for reading.