Category: data science

  • Big Data, a promised land where the Big Bucks grow

    Big Data, a promised land where the Big Bucks grow

    B

    Consider this: Many people come up to me in the street, and, apropos of nothing, they ask me how they can make money from Big Data.

    Normally I would send such people to see a specialist – no, not a guru, but a sort of health specialist, but because this has happened to me so many times now, I eventually decided to put pen to paper, push the envelope, open up the kimono, and to record my advice for posterity and the great grandchildren.

    So, here are my top seven tips for cashing in quick on the new big thing on the block.

    1 – A business opportunity for faith

    Like every new religion, trend or fad, Big Data has its own founding myths, theology and liturgy, and there is money to be made in it; loadsa lovely jubbly money. By predicating and evangelising Big Data you will be welcomed with open arms into the Big Data faith, and will receive all the attendant benefits that will miraculously and mysteriously fall upon you and your devout friends. Go on, I dare you. Be a Big Data guru, a shepherd to a flock of sheep, and enjoy the wealth, health and happiness that most surely will come your way. You too can look cool in red Prada slippers, a flattering and flowing gown and matching accessories.

    2 – Acquire it, multiply it, weigh it, mark it up and sell it on

    Simply stated, this is about acquiring other people’s data, by sacred means or profane, marking it up and then selling it on. The value you add is that you act as a trusted conduit, a conduit for good. You may care to enrich the data, swop the order of data, replicate and embellish data, make stuff up, etc. which all serves to ‘add value’ to the data. You may even consider adding nuggets of value to the data, just for kicks and giggles. My best friend’s favourite is injecting the good old ‘diaper and beer’ and ‘friends and family’ clichés into every Big Data collection, as it never fails to thrill, please and delight.

    3 – Anything can be anything

    The good thing about making money from Big Data is that it doesn’t need to be anything to do with Big Data. Make a 20GB Enterprise Data Warehouse? Call it a Big Data success. Sell 20 boxes of dodgy doughnuts down the alternative market? Proclaim a Big Data triumph. Sell your digital porn stash to your best mate? Point to the incredible invisible hand of the Big Data market at work. See what I’m doing there. Anything can be anything, and you too can cash in on that opportunity, big time.

    4 – Big Data Patronage

    Tense, nervous headaches? Do you like making up stories about Big Data, or for that matter anything else? Are you a natural born fibber but are strapped for cash? Then worry no longer. If you get a Big Data patron you will be sorted for ‘life’; get two and you’ll be sorted for the afterlife as well. With a Big Data patron you can get the most tenuous, crappiest and superficial of pieces published, promoted and vaunted – globally. Can’t make it up yourself, then outsource and offshore it, after all, just get the keywords right for SEO ranking and the gullible will flock to you in droves. The down side of this profession is that you will be targeted for writing half-truths, quarter-truths and downright lies, and you will be pilloried as a purveyor of rank hyperbole. But don’t worry, take heart and never lose the faith, you will be in good company. As one Big Data guru was want to say ” If you repeat a lie often enough, people will believe it, and you will even come to believe it yourself.” Amen! brother.

    5 – Big Data Certification

    By 2016 there will be global demand for 30 billion Big Data professionals. Are you prepared to cash in on that inevitability? No? Then consider this.

    One of my best friends makes his living as a completely phony Big Data Scientist. For two hundred bucks he can make you a Data Scientist or a Big Data guru. Some guys give you an education but this guy gives you immediate access to high paying jobs, sex and a life in the city. Moreover, for an extra 250 bucks you can also become a certified Big Data Trainer, which will allow you to do unto others what has been done unto you.

    6 – Creative Technology Reuse

    Big Data has heralded in the biggest innovations known in the history of computing, and arguably in the entire history of humankind. One of those new inventions has been the now widely acclaimed and revolutionary ‘flat file data base’ (FFDB), and this has been accompanied with developments in low level operating system primitives that allow for the processing of these collections and hierarchies of FFDBs. So, if one has a mind to do so, one can get some real business leverage off of these new tendencies by borrowing 21st century technology found in old operating system hacks from the sixties and seventies and eighties and nineties and… Well, the point is that in order to get serious funding it is no longer good enough to have a half page business plan, it is also necessary to eke out ‘stuff’ that works within the new paradigms of Big Data and Big Data Analytics. For my next venture I will be looking for serious funding for my ‘Arbitrary Dawdle Down Data Street’ (AD3S) Big Data Analytics platform, a platform designed to support virtual 1k bit processing and the massively parallel provision of global regular expression search and match (S&M), concatenation and listing, and cooperative data-driven and streamed data extraction and reporting. I’m hoping to attract the attention of governments, the EU, the Manic Street Preachers, the UN, China, Vladimir Putin, the DOD, HP, Oracle, Gartner, Lana Del Rey, Deloitte and IBM. So, this is going to be absolutely massive. Word!

    7 – Big Data Brokerage

    According to leading management consultants and industry watchers Gartner, McKinsey and Deloitte, data needs to be managed and accounted like any other asset, such as money. To get into a similar view-point requires a massive leap of faith, but it is a conversion that might drive dividends. One avenue to be explored in eking out value from the apparently massively valuable Big Data lakes, silos and pools is through the operation of a Big Data Brokerage. A Big Data Brokerage is a business whose main responsibility is to be an intermediary that puts Big Data buyers and Big Data sellers together in order to facilitate a transaction. Big Data Brokerage companies are compensated via commission after the Big Data transaction has been successfully completed. They may also charge introductory fees. Just imagine the wealth of business opportunities in that. You could become the Goldman Sachs of data.

    That’s it folks!

    I hope you enjoyed this piece and would be pleased to hear your views on this and other subjects.

    Whilst I understand the attraction and even the need of creating a new and significant growth industry, I would also advise a degree of restraint, and whilst I see that “Big Data” (the consideration of the potential value of All Data) has its allure, I also think that some good sense and informed caution should also prevail.

    Thank you so much for reading.

    Martyn Richard Jones

    Spain, 2015

  • The Big Data Shell Game

    The Big Data Shell Game

    To the experienced observer, Big Data propaganda may well appear to be a disorganised surfeit of half-truths, sleights of hand and boloney. Indeed, the once famously alliterative characterisation of Big Data as defined by volumes, variety and velocity, seems now more appropriately applied to the quantity, invariability and quality of the incessant self-aggrandising hype, hokum and Hadoop being astro-turfed by every dog and his guru. Indeed, the very fact that such an inevitable mega-trend needs so much hype, disingenuousness and spin to support its passage to universal applicability, is a massive contradiction, a disservice to professionals, and an artless deception worthy of our criticism and condemnation. (more…)

  • I lied about Big Data! Have an issue?

    I lied about Big Data! Have an issue?

    In a city centre office block, somewhere in Scotland, the conversation between the IT Business Manager (Bill) and the Information Management Manager (Richie) is in full swing,, Bob is irate because his successfully delivered data mart has been derided as unusable rubbish by the business people it was meant to serve.

    Let’s join the conversation:

    Bill: I hate this job. Every time we try and help the business all we get back are complaints. Complaints because it’s not what they want, complaints because it’s in the wrong format, complaints because of the cost, or the performance, or the availability. All we get are complaints, complaints and complaints.

    Richie: Well, to be fair, Little Bill, this was one clearly avoidable situation. We didn’t have to build the data mart.

    Bill: I know what you’re thinking, but you are wrong. We had to do something. Anything.

    Richie: I don’t agree, Little Bill, we always had the option of doing nothing.

    Bill: And why would we do nothing?

    Richie: Because. as I said at the time, Little Bill, without demand you don’t create supply, and at this level and on this scale, if you want to create supply, you first encourage demand. But it’s still fundamentally about meeting demand.

    Bill: But, things don’t work like that in this organisation.

    Richie: I think you will find that in fact that approach works remarkably well, Little Bill, and in almost any type of organisation. The problem is one of perception, if it has never been tried before there is no internal reference to whether it works or not, and of course repeating the mistakes of the past with absolute security, if better than doing something correct, but unproven in this setting.

    Bill: No, I still think you fail to understand the nuances of this business.

    Richie: You may well be right, Little Bill, but clearly if we really understood the even the nuances of the business, then we wouldn’t have wasted time on this effort, an effort that one of the business executive described as the expensive manifestation of an abject failure to understand the fundamentals of the business.

    Bill: They said that?

    Richie: Yes, they certainly did, Little Bill.

    Bill: Well, if that’s the case then they clearly don’t know what they are talking about.

    Richie: As may be the case, but that doesn’t help us either.

    Bill: So, you with all of your ‘knowledge and experience’, what do you suggest?

    Richie: I suggest that we take a proactive approach to encouraging demand.

    Bill: Such as?

    Richie: Well, I would revisit the recommendations that I made when I first joined this department.

    Bill: Okay, just remind me of the key points.

    Richie: We need part of IT to understand business process, and our business processes; in effect we need people who know the business of the business. These should be people who talk to the business in language the business understands, has a good grasp of a vast array of issues, and who can be confident in their everyday dealings with business.

    Bill: But, the business always thinks it knows best, how will these people succeed where we have almost always failed in the past? They think we overly complicate things; they virtually try and tell us how to do our jobs.

    Richie: That’s why we need people who can communicate with authority, persuasively and with ease, not from a basis of mistrust, lack of empathy and even disdain. We need people who can sell ideas, can frame discussions and articulate coherent and realisable proposals for business IT solutions using language the business grasps the first time. We need people who understand what is said, can lead discussion and can capture requirements in a way that IT can also understand.

    Bill: But the refuse to talk to us.

    Richie: Well, that’s perhaps rather unsurprising from people who seem to think they have articulated the same requirements to us, and repeatedly, over an extended period of time. The problem is that we have very rarely documented those requirements, and when it has happened it has not been in a way that business can understand and verify, they can’t take any of our requirements and actually understand them without resorting to a translator, so they don’t do it.

    Bill: Okay, so apart from blaming IT, what do you suggest?

    Richie: The first hurdle seems to be simple. We need to convince the business that we actually have something worth listening to, that we aren’t going back to waste their time, yet again.

    Bill: And?

    Richie: So, what I suggest is this. Part of my team will spend time on investigating existing and new technologies, methods and approaches and how these are applied in similar industries or even dissimilar settings, but with certain synergies. They will have a good grasp of the business but their focus will be on understanding technology and relating it to project opportunities within our business. They will then work with our Business Consultants to actually articulate, explain and sell the benefits of these ideas to the business.

    Bill: This, as I have repeatedly told you, is what we do now.

    Richie: I don’t think so, Little Bill. There is a marked difference between what we do now, with the “look what a marvellous data mart we have made for you, it has data and lots of menu options, and graphics and stuff” versus the “we would like you to that allows you to be able to identify tangible cross-selling opportunities between various lines of business and with a high degree of certainty, this driving increased revenue, and increased customer intensity, and therefore loyalty… and repeat business”

    Bill: So, we go begging the business for projects, with silver-tongued rhetoric.

    Richie: No, Little Bill, we give the business what it wants. They are our customers, and as any business person should know, giving the customers what they want is a sure fire route to success.

    Bill: Yes, but it would never work here. We are a very conservative company.

    Richie: If the rest of the organisation was that conservative, we wouldn’t even be in business.

    Bill: So what happens if the business says yes?

    Richie: The business consultants and the research consultants work with the architecture consultants in socialising the business requirements and in developing solutions architecture (or a domain architecture), and as part of this they will also interact with the enterprise architecture consultant. So at some point, we will have a Business Requirements document, an IT requirements document, and an IT / Business Process Architecture document, and a Project Proposal document. Then the Business Requirements document – including detailed financials, together with the Business Project Proposal are socialised with the business, and submitted to them for review and approval. We then negotiate.

    Bill: You make it sound easy.

    Richie: You have to know what you’re doing, and there is logic to it all, but it’s far more rewarding than working on projects that invariably fail to satisfy.

    Bill: So, when do we get started on all of this…?

    Richie: As soon as you want, Little Bill.

    So as we leave Bill and Richie to hammer out the details of the new approach, what can we take-away from this piece of business voyeurism?

    I sincerely believe that the hardest job of any data warehousing professional, at least one worthy of the name, is in convincing sometimes even senior IT management of the need for doing the right thing right, of the possibility of doing the right thing right, and of the dangers of confusing ignorance and wishful-thinking with pragmatism.

    So, make sure you get an expert, that your expert really is a professional, is absolutely ethical and that they really know their stuff, and when they don’t know, will not try and pretend that they do know, then trust in their professionalism, judgement and expertise, even if you then verify what you are told – and please, don’t verify this knowledge with a charlatan, spend more money and get the verification of a trusted and proven expert. Do this, and in this way and you won’t go far wrong.

    Many thanks for reading.

    Oh, and one last thng…

    According to SAP “Big Data is the ocean of information we swim in every day”. I disagree; Big Data hype is the ocean of crap that we have to navigate through every second of every day.

    Moreover, SAP contribute to that shite.

    In a big way.

    So, not only do we have Software aus Polen, we have Big Data aus Polen. With apologies to the great Polish IT professionals I know and respect.

    Champion!

    SAP! Mend your ways.

    The choice is yours.

    Many thanks for reading. 

  • Not banking on Big Data?

    Not banking on Big Data?

    First, a request, please consider joining The Big Data Contrarians.

    I have worked with clients across the entirety of financial industry for most of my career, and although this may surprise some people, I believe that I fully understand why they are being conservative about Big Data in general and Hadoop in particular. I can also understand why some people want to keep up or even ramp-up even more the Big Data market buzz, but with such a dearth of meaningful, well described and verifiable Big Data ‘success stories’, neither the banks nor I are going to be speculating in any big way on Big Data or Hadoop, anytime soon.

    Based in Spain for almost three decades, I have been up close and intimate with a few of the biggest players in the Spanish financial industry. Indeed, Spanish banks have not only lead the way in the effective, innovative and business driven use of technologies in the Spanish market, but have applied that financial industry nous around the world.

    In recent times, the big financial players in Spain have entered into the Big Data fields and stratospheres. From what I know, which may not be all, or so much, they are still watching and investigating rather than putting tangible things in production. Nevertheless, there are some interesting Big Data application ideas floating around the financial world. These are still relatively early days for Big Data in finance, and it will take some time for the hype to fade away and the cream of financial Big Data to rise to the top.

    However, it has happened before.

    If there was ever a country that quietly, diligently and consciously implemented Data Warehousing and Business Intelligence, then it has been Spain. Spanish companies were not only early adopters but also early beneficiaries of implementing Data Warehousing. Not for nothing did Bill Inmon’s company Prism Solutions chose Madrid as a major hub for its European Data Warehousing consulting, sales and support activities. Bill being the father of Data Warehousing and Prism being one of his commercial babies.

    As an aside, at Prism I had the opportunity of working alongside fantastic professionals and great people with knowledge, values and experience, such as Don and Katherine. That great gig, I will never forget.

    Which brings me to this.

    I knew what Data Warehousing would be good for, and amplified this knowledge through reasonable, rational and coherent ways of addressing a wide range of requirements. My aim was to support my claims with coherent, simple and verifiable examples of Data Warehousing success stories.

    I knew how to explain Inmon’s Data Warehousing, in business, management and technical terms. I saw when a company could benefit from DW and also when a company was not ready for DW. However, try as I might, I cannot achieve the same intensity of understanding with Big Data. Believe me I have tried.

    I’m not a contrarian just because, but isn’t it about time the Big Data BS babblers put up or shut up?

    So, if you are like me, then join The Big Data Contrarians.

    Many thanks for reading.

  • Consider this: Does all data have value?

    Consider this: Does all data have value?

    Header1

    To begin at the beginning

    You can use all the quantitative data you can get,

    but you still have to distrust it and use your own intelligence and judgment.

    Alvin Toffler

    There is a touching belief that all markets are rational and that a company’s value is accurately reflected in the current share price multiplied by the total number of shares in circulation.

    The thing is, this doesn’t square up with formal accounting practices and financial reporting, as they now stand. So, there is a sentiment that the gap between capitalisation and accounting valuation must be due to some less tangible factors, which is true, and for many years it was accepted that aspects such as good will, were not in fact measurable or even easily manageable.

    In the nineties, Leif Edvinsson and a team of accounting and finance specialists at Skandia pioneered the accounting and reporting of non-tangible assets, which they called the Skandia Navigator, the focus was clearly on reporting the progress of the creation and use of Intellectual Capital (IC). What the Skandia IC team did, how they went about it, and how they presented their results, (as a supplement to their annual report,) made a lot of good sense.

    Now there is a movement to measure, value and manage all data as if it were a highly tangible asset, because, according to some, IT has been incapable of managing data because they don’t know how to measure it and they aren’t particularly well equipped to do so, a claim that is paradoxically made by people who are very much in the IT camp.

    But this is not about treating data as an asset but focuses on potential risks due to data loss, data corruption, quality issues, misinterpretation, misuse or whatever. Simply stated, a risk prevented from arising or a risk that is mitigated is not an asset, and risk management is not primarily about rigidly theoretical asset management and vice verse.

    Now, I won’t argue that it’s not a good idea to have an idea of the value of business assets, from either a quantitative or qualitative perspective, but I think there are more moderate, coherent and less fundamentalist approaches to the understanding and valuation of data, approaches that are more aligned to contemporary views of the understanding of the value of intellectual capital assets – unstructured and structured. What follows is a brief and high-level view of some potential options.

    Data Assets in MOSCOW

    What follows is a simple explanatory technique that is part of the DW 3.0 Information Supply Framework approach. It focuses on qualitative aspects of data as an asset. The following diagram provides an overview of foundations of the first phase of this simplified risk, classification and prioritisation approach:

    On the horizontal access we have used the MOSCOW classification to break the problem down into simple terms. Here is an example of its configurable application:

    MUST – Must treat the data as an asset: If this data is compromised in anyway then it would possibly result in serious negative implications for all (or most) customers and severe financial, contractual and reputation impacts on the business

    SHOULD – Should treat the data as an asset: If this data is compromised in significant ways it would result in intolerable negative financial implications for a number of customers and would also lead to negative financial impacts on business

    COULD – Could treat the data as an asset: If this data is compromised in significant ways then it could result in financial consequences for the business, but without impacting customer churn

    WON’T – Won’t treat the data as an asset: Data that if compromised would not result in financial consequences wither for customers or for the business

    It is important to understand that these definitions are examples and that in practice the descriptions and parameters must be determined and aligned in cooperation between the distinct business stakeholders, including business IT.

    Data Value Chains

    Another aspect that is important to the understanding of ‘data as an asset’ is to be found in data value chains. The following diagram illustrates the DW 3.0 technique used to simplify the attainment of a consensual view of the understanding of this value.

    DATA: “Data is a super-class of a modern representation of an arcane symbology.” –Anon

    In order for data to be more than an operational necessity it requires context.

    Providing valid data with valid context turns that data into information.

    Data can be relevant and data can be irrelevant. That relevance or irrelevance of data may be permanent or temporary, continuous or episodic, qualitative or quantitative.

    Some data is meaningless, and there are cases whereby nobody can remember why it was collected or what purpose it has.

    Taking all this into account we can now ask the pragmatic question: what value does this data have? Which is sometimes answered with a: ‘no value whatsoever’.

    INFORMATION: “My sources are unreliable, but their information is fascinating.” –Ashleigh Ellwood Brilliant

    In order for information to be created to drive organizational operations and tactics we must have valid interpretations of data.

    Providing valid information with valid interpretations potentially turns that information into knowledge.

    Information can be correct, partially correct or incorrect; it may be relevant or irrelevant. The validity of information may also be permanent or temporary, continuous or episodic, and qualitative, quantitative or both.

    Some information may be relevant, irrelevant or misleading. Information derived from erroneous and incomplete data may be usable, but the outcomes of using that information may be unpredictable.

    Taking all this into account we can now ask the pragmatic question: what value does this information have? A question which can sometimes be answered with an ‘it is probably too early to say’.

    KNOWLEDGE: This necessarily refers to a subset of ‘business knowledge’ known as structured intellectual capital.

    The adequate, appropriate and timely application of knowledge (structured intellectual) requires wisdom; knowledge alone will no longer be good enough.

    Providing valid knowledge with valid interpretations potentially turns that knowledge into valid and executable strategies.

    Knowledge may be useless or useless; it may be relevant or irrelevant; it may even be wrong, even if we have named it ‘knowledge’. The usefulness of knowledge may also be permanent or temporary, continuous or episodic, qualitative or quantitative.

    Some knowledge may be relevant, irrelevant or misleading. Knowledge derived from erroneous and incomplete information may be usable, but the outcomes may be unpredictable. As they might say in Córdoba: ‘Give knowledge to a donkey and it will still remain a donkey’.

    Taking all of this into account we can now ask the pragmatic question: what value does this knowledge have? A question which can sometimes be answered with: ‘Isn’t it getting rather hot in here’.

    What does it all mean?

    To paraphrase George S. Patton, for the purposes of business data, an imperfectly good pragmatic valuation of data executed today is better than a perfect theoretic valuation of data made in the next life.

    The valuation of data, information and knowledge is complex and involves many intangibles, and although some may view the data, information and knowledge chain as a closed process, this is not in fact the case, as each step of the process is influenced by a number of factors that fall outside of a simplified view of any theoretical or practical progress from data to wisdom.

    When I started in IT over 30 years ago I worked on some of the first large-scale OLTP projects in Europe. On one such project, the development, test and production the databases were continuously replicated. Every day, not one, but ten backups of the data were made, and then shipped to different physical locations.

    Thirty years on and some people are saying that IT has never treated data as an asset?

    The fact of the matter is that data is managed as an asset. In fact, some IT organizations may be taking exaggerated measures in order to protect the data and information in their care. That some people have issues with the contemporary management of data does not change the facts on the ground, and the perceived shortfall in the commercial exploitation of data is frequently and erroneously interpreted as the absence of asset appreciation and management.

    Moreover, even with generally accepted accounting principles there are tangible assets that are either of dubious value or are cost-absorbing liabilities.

    Finally, there may also be unintended consequences of an overzealous approach to the financial reporting of intangible assets such as data, and just as the value of tangible assets can be accidentally or deliberately overstated, so too could the value of data, which may well lead to a significant overvaluation of a business. Moreover, it’s not beyond the realms of possibility that big data lakes, data universes and dark data dungeons – the ones that apparently will drive trillion dollar economies – are in fact somewhat worthless.

    In this respect I think there is in some quarters an unhealthy fixation on the supposed (and as yet mainly unproven) value accruable from masses of data.

    That’s all folks

    Steve Jobs didn’t turn Apple around by relying solely on data and information, it was knowledge, and more especially the wisdom of knowing where, when, why and how to apply that knowledge that made the significant difference.

    So, until the next time, hold this thought: ‘Google market cap slashed and Facebook in freefall as accountants ask “where’s the big data beef?”… Breaking news!’

    Many thanks for reading.

  • Why Sperry Univac Won Over IBM in Business Strategy

    Why Sperry Univac Won Over IBM in Business Strategy

    Martyn Richard Jones

    Here’s a clue. Because of me.

    In the eighties, there was a company called Sperry Univac, which was part of the once-famous Sperry Corporation.

    At that time, a significant manufacturing concern in the industrialised Midlands of England was looking to automate and computerise operations. This meant that they would be in the market for some serious heavy iron – to use the old euphemism for mainframe computers. (more…)

  • CONSIDER THIS: Did Big Data Kill The Statistician? 2026

    CONSIDER THIS: Did Big Data Kill The Statistician? 2026

    OLYMPUS DIGITAL CAMERA
    Blue sky data

    Hold this thought: ‘There are big lies, damn big lies and big data science’.

    Statistics is a science. Some argue that it is the oldest of sciences. It can be traced back in history to the days of Augustus Caesar, and before.

    In 1998, Lynn Billard wrote a paper that laid out the role of the Statistician and Statistics. She stated that “no science began until man mastered the concepts and arts of counting, measuring, and weighting”.[1]

    (more…)