01 Tuesday Dec 2015
The Digital Document Lifecycle
MARTYN RICHARD JONES
This is a story of the life of a digital document. Its purpose is to explain the process of analysing, designing, building, testing and delivering content rich business artefacts in today’s digital age.
13 Friday Nov 2015
22 Wednesday Jul 2015
Posted in Big Data, Consider this, good start, Good Strat, goodstart, goodstrat
Tags
As a child, I adored the USA rock band the Eagles, especially the musical talents of Joe Walsh. This explains the inspiration behind the title of this piece.
So, what’s going down at Ashley Madison?
Never heard of them? Off your radar? Surely not?
That stretches the bounds of incredulity. As even the people in Singapore’s Media Development Authority have heard of them. They even described their business site this way “it promotes adultery and disregards family values”, and subsequently will not allow them to operate in Singapore. Well, what a turn-up for the books.
On a more serious note, and as you might know, (from Wikipedia or some other ‘sites’,) Ashley Madison is a Canadian-based online dating service and social networking service marketed to people who are married or in a committed relationship. Its slogan is “Life is short. Have an affair.” It seems, if we are to believe various reports doing the rounds, that their Big Data has been compromised, big time.
Yes, I know, how could that possibly have happened, right?
According to some reports, Adison Mashley have around 37 million clients in the Big Data pool, and large caches of it have allegedly been stolen after an apparently successful hacking attempt was carried out. According to Krebs On Security, data stolen from the web site in question “have been posted online by an individual or group that claims to have completely compromised the company’s user databases, financial records and other proprietary information.”
But, again I ask, how can this happen?
I am not an avid fan of Big Data technology for core business use, and given the level of Big Data technology maturity, it sounds like a dopey idea. But each to their own.
What I will state is that my database management experience has tended to be associated with database technologies that can only be hacked as part of an inside job i.e. where people either know user IDs, passwords, IP addresses and layers of protection etc. or know of someone who does. Either someone who is a friend, part of the family (no, not that type of ‘family’) or someone who can be blackmailed into divulging the required access paths and security check workarounds.
However, taking a broader and more permissive view of this alleged hackerisation of Big Data, do we write it up as a Big Data success, i.e. The Amazing Big Data Affair? Put it down to a technical glitch and community faux pas? Or do we take a jaundiced view of the whole thing and keep it real? I await with baited breath for the enlightened opinions of the Big Data gurus.
Mitch ‘n’ Andy are not unfamiliar with ‘issues’ related to the use of people’s data. The Daily Dot carried a piece from contributing writer S. E. Smith with the headline ‘Why Ashley Madison is cheating on its users with Big Data’ in that piece, Smith states that “Like pretty much every other website on Earth, Ashley Madison spies on its users and crunches the data in a variety of ways to increase the bottom line.”
Belinda Luscombe writing in Time confirmed these suspicions with a piece titled ‘Cheaters’ Dating Site Ashley Madison Spied on Its Users’. She wrote:
In a study to be presented at the 109th Annual Meeting of the American Sociological Association in San Francisco on Saturday Aug. 16, Eric Anderson, a professor at the University of Winchester in England claims that women who seek extra-marital affairs usually still love their husbands and are cheating instead of divorcing, because they need more passion. “It is very clear that our model of having sex and love with just one other person for life has failed— and it has failed massively,” says Anderson.
“How does he know this? Because he spied on the conversations women were having on Ashley Madison, a website created for the purpose of having an affair. Professor Anderson, who as it turns out is a the “chief science officer” at Ashley Madison, looked at more than 4,000 conversations that 100 women were having with potential paramours. “I monitored their conversation with men on the website, without their knowing that I was monitoring and analyzing their conversations,” he says. “The men did not know either.”
Elsewhere, and as reported on Wikipedia, “Trish McDermott, a consultant who helped found Match.com, accused Ashley Madison of being a “business built on the back of broken hearts, ruined marriages, and damaged families.”
Wow, wow, and triple wow! What a way to run a dance hall!
Maybe they should reconsider their slogan, making it more snappy and apposite. How about “Life is short, we pimp your Big Data” as a starter? So go ahead, make your own and post it below. Have fun.
Many thanks for reading.
Oh, and one last thing before I go… GOOD-AD: Join The Big Data Contrarianshttps://www.linkedin.com/grp/home?gid=8338976
21 Tuesday Jul 2015
Posted in Big Data, Consider this, good start, Good Strat, goodstart, goodstrat, Strategy
Bore da, pobl dda. A hyfryd dydd ‘Big Data’* i bawb.
When it comes to Big Data, some people accuse me of being akin to a Luddite. Nothing could be further from the truth. Not that the facts matter. In the age of superficiality and surfaces there is as much wilfully cultivated obliviousness as there is unashamed and unabashed term abuse. Add the prevailing underlying current of anti-intellectualism into the mix, and we have an explosive combination that manifests itself in the alliterative combination of bluff, bluster and banality.
— JOIN THE BIG DATA CONTRARIANS: http://www.linkedin.com/grp/home?gid=8338976
I was reticent about writing this article, because it’s a bit like arguing against the irrational, self-interested and wilfully obtuse. Or as Ben Goldacre would have it, “You cannot reason people out of a position that they did not reason themselves into.” Therefore, a lot of care needed to be exercised. Indeed, Mark Twain once stated, “Never argue with stupid people, they will drag you down to their level and then beat you with experience.” Now, I wouldn’t go that far, and I do try to be nicely diplomatic, most of the time, but I can see where he was coming from.
Anyway, without more ado let’s get a handle on what a Luddite is, in terms I hope that most will understand.
According to Wikipedia (yes, I know) The Luddites were:
“19th-century English textile workers who protested against newly developed labour-economizing technologies from 1811 to 1816. The stocking frames, spinning frames and power looms introduced during the Industrial Revolution threatened to replace the artisans with less-skilled, low-wage labourers, leaving them without work.”
So why do I get a feeling that some people think that I am a Big Data Luddite?
Here is Peter Powell of PDP Consulting Pty Ltd putting me in my place below the line on my piece titled 7 Amazing Big Data Myths:
“With all due respect – your post does sound a little like what I could envisage an exchange between a man riding a horse and a man driving one if the first automobiles….sorry.”
Although a respectable knowledge of the technology and its evolution would inform otherwise, I assume that this means that I would be the “man riding a horse”… An interesting piece of conjecture indeed, even if hat in lacks in accuracy is made up for by the inexplicable certainty of belief. Still, it’s fascinating to discover just how many ‘experts’ think that this stuff – the sort of stuff I was doing in the mid to late eighties at Sperry and later Unisys – is bleeding edge innovation,
Sassoon Kosian a Sr. Director of Data Science at AIG, had this to tell me on my piece entitled Amazing Big Data Success Stories:
“Yes, cynical indeed… here is another amazing Big Data success story. You go on your computer, type in any search phrase and get instantaneous and highly relevant results. It is so amazing that a word has been coined. Guess what that is…”
What to say? There goes a person who seems to believe that the history of search starts and ends with the Google web search engine. Something slightly less than a munificently inapposite comment, only outdone by its tragically disconnected banality.
More recently, Bernice Blaar had this to say about my take on Big Data in general and The Big Data Contrarians in particular.
“Master Jones may well be the great and ethical strategy data architecture and management guru that the chattering-class Guardian-reading wine-sipping luvvies drool over, but he is also a brazen Big Data Luddite. No, actually far worse than a Luddite, he`s a Neddite, because with his ‘facts’ and ‘logic’ (what a laugh, you can prove anything with facts, can’t you [tou}???) he is undermining the very foundation of the Big Data work, shirk and skive ethic that has been so hardily fought for by the likes of self-sacrificing champions and evangelists of the Big Data revolution, to wit, such as those bold, proud and fine upstanding members Bernard Marr, Martin Fowler and Tom Davenport, for example, and the brave sycophants that worship at their feet. Martyn is worse than Bob Hoffman, Dave Trott, Jeremy Hardy, Mark Steel, Tab C Nesbitt and Bill Inmon, all rolled into one. He may be a great strategist, but I wouldn’t hire him. Contrarian Luddite!”
And then followed it up with this broadside:
“The Big Data Contrarians group are nothing more than a bunch of over-educated clown-shoes who are trying to scupper the hard-work of decent people out to earn a crust from leveraging the promise of a bright future. In a decent society of capital and consumers, they would be banned off the face of the internets.”
How does one reciprocate such flattering flatulence? How can one possible respond to such a long concatenation of meaningless clichés? Though to be fair, I quite liked being referred to as a Neddite, whatever that is.
Anyway, to set the record straight, this is where I stand.
A contrarian is a person who takes up a contrary position, especially a position that is opposed to that of the majority, regardless of how unpopular it may be.
Like others, I am a Big Data Contrarian, not because I am contrary to the effective use of large volumes, varieties and velocities of data, but because I am contrary to the vast quantities of hype, disinformation and biased mendaciousness surrounding aspects of Big Data and some of the attendant technologies and service providers that go with the terrain. I don’t mind people guilding the lily (to use an English aphorism for exaggeration), but I do draw the line at straight out deception., which could lead to unintended consequences, such as creating false expectations, diverting scarce resources to wasteful projects or doing people out of a livelihood. That’s just not tight.
Does that make me a Luddite (or a Neddite)? I don’t think so, but do make sure that your opinion is your own and is arrived at through reason, not some other persons bullying hype. As I wrote elsewhere some moments ago “If you have to lie like an ethically challenged weasel to sell Big Data then clearly there is something amiss.”
As always I would love to hear your opinions and comments on this subject and others, and also please feel free to reach out and connect, so we can keep the conversation going, here on LinkedIn or elsewhere (such as Twitter).
Many thanks for reading.
— JOIN THE BIG DATA CONTRARIANS: http://www.linkedin.com/grp/home?gid=8338976
Photograph: Delegates at my Big Data Summer Camp in Carmarthen (Wales).
*Data mawr
20 Monday Jul 2015
Posted in Big Data, Big Data Analytics, Consider this, Data Warehousing, good start, Good Strat, goodstart, goodstrat
Okay, before we get started I have to declare the real intent for posting this piece. It is to get you to join The Big Data Contrarians professional group here on LinkedIn.
To apply to join the best Big Data community on the web simply navigate to this address http://www.linkedin.com/grp/home?gid=8338976 (or paste it into your browser) and request membership, the process is quick and painless and well worth the effort.
Now for the rest of the news…
There are many common misconceptions amongst the Big Data collective about Data Warehousing. There are common fallacies that need clearing up in order avoid unnecessary confusion, avoidable risks and the damaging perpetuation of disinformation.
In the dim and distant past of business IT, the best information that senior executives could expect from their computer systems were operational reports typically indicating what went right or wrong or somewhere in between. Applied statistical brilliance made up for what data processing lacked in processing power, up to a point, because even heavy lifting statistics requires computing horsepower, which in those days was really a question of serious capital expenditure, which not all companies were willing to commit to.
Then, and curiously coincidentally, people around the world started to posit the need for using data and information to address significant business challenges, to act as input into the processes of strategy formulation, choice and execution. Reports would no longer just be for the Financial Directors or the paper collectors, but would support serious business decision making.
Many initiatives sprang up to meet the top-level decision-making data requirements; they were invariably expensive attempts, with variable outcomes. Some approaches were quite successful, but far too many failed, until the advent of Data Warehousing.
Back then, most of the data that could potentially aid decision-making was in operational systems. Both an advantage and a problem. Data in operational systems was like having data in gaol. Getting data into operational systems was relatively easy, getting it out and moving it around was a nightmare. However, one of the advantages of operational data is that it was generally stored in a structured format, even if data quality was frequently of a dubious nature, and ideas such as subject orientation and integration were far from being widespread.
Of course, data also came in from external sources, but usually via operational databases as well. An example of such data is instrument pricing in financial services.
Therefore, briefly, a lot of Data Warehousing started as a means to provide data to support strategic decision-making. Data Warehousing ways not about counting cakes, widgets or people, which was the purview of operational reporting, or to measure sentiment, likes or mouse behaviour, but to assist senior executives, address the significant business challenges of the day.
Bill Inmon, the father of Data Warehousing, defines it as being “a subject-oriented, integrated, time-variant and non-volatile collection of data in support of management’s decision making process.”
Subject Oriented: The data in the Data Warehouse is organised conceptually (the big canvas), logically (detailing the big picture and) and physically (detailing how it is implemented) by subjects of interest to the business, such as customer and product.
The thing to remember about subject areas is that they are not created ad-hoc by IT according to the sentiments of the time, e.g. during requirements gathering, but through a deeper understanding of the business, its processes and its pertinent business subject areas.
Integrated: All data entering the data warehouse is subject to normalisation and integration rules and constraints to ensure that the data stored is consistently and contextually unambiguous.
Time Variant: Time variance gives us the ability to view and contrast data from multiple viewpoints over time. It is an essential element in the organisation of data within the data warehouse and dependent data marts.
Non-Volatile: The data warehouse represents structured and consistent snapshots of business data over time. Once a data snapshot is established, it is rarely if ever modified.
Management Decision Making: This is the principal focus of Data Warehousing, although Data Warehouses have secondary uses, such as complementing operational reporting and analysis.
In plain language, if what your business has or is planning to have does not fully satisfy the Inmon criteria then it probably is not a Data Warehouse, but another form of data-store.
The thing to remember about informed management decision making is that it needs to be as good as required but it does not need to achieve technical perfection. This observation underlies the fact that Data Warehouse is a business process, and not an obsessive search for zero defects or the application of so called ‘leading edge’ technologies – faddish, appropriate or not.
JOIN THE BIG DATA CONTRARIANS: http://www.linkedin.com/grp/home?gid=8338976
Before we delve into the meaning of Data Warehousing, there are a couple of terms that need to be understood first, so, by way of illustration:
Let’s follow the numbers in the simplification of the process.
This is a very simplified view, but it serves to convey the fundamental chain of events. The most important aspect being that we start (1) and end (7) with the user, and we fully involve them in the non-technical aspects of the process.
JOIN THE BIG DATA CONTRARIANS: http://www.linkedin.com/grp/home?gid=8338976
Essentially, a Data Warehouse is a business driven, enterprise centric and technology based solution for continual quality improvement in the sourcing, integration, packaging and delivery of data for strategic, tactical and operational modelling, reporting, visualisation and decision-making.
A data warehouse is business centric and nothing happens unless there is a business imperative for doing so. This means that there is no second-guessing the data requirements of the business users, and every piece of data in the data warehouse should be traceable to a tangible business requirement. This tangible business requirement is usually a departmental or process specific dimensional data model produced together in requirements workshops with the business. We build the Data Warehouse over time in iterative steps, based on the criteria that the requirements should be small enough to be delivered in a short timeframe and large enough to be significant.
Typically, a Data Warehouse iteration results in a new Data Mart or the revision of an existing Data Mart.
As we build up the collection of Data Marts, we are also building up the central logical store of data known as the Enterprise Data Warehouse that serves as a structured, coherent and cohesive central clearing area for data that supports enterprise decision making. Therefore, whilst we are addressing specific departmental and process requirements through Data Marts we are also building up an overall view of the enterprise data.
By technology, I mean technology in the broadest sense of techniques, methods, processes and tools, and not just a question of products, brands or badges.
Unfortunately, there is a popular misconception that Data Warehousing is primarily about competing popular and commercial available technology products. It isn’t, but they do play an important role.
The following is an example of a very high-level Data Warehouse architecture diagram.
Various methodologies support the building, expansion and maintenance of a Data Warehouse. Here is one example of a professional data integration methodology, produced, maintained and used by Cambriano Energy.
And here is an information value-chain map as used by Cambriano Energy as part of its Iter8 process management. There are alternatives, many of which do a satisfactory job.
Last but not least, this was (from memory) the way that Bill Inmon’s Prism Solutions ETL company used to view the iterative EDW building process.
JOIN THE BIG DATA CONTRARIANS: http://www.linkedin.com/grp/home?gid=8338976
At this point, I decided to cut short further explanations on aspects on Data Warehousing. However, if you have any question then please address them to me and I will do my best (or something close) to answer them.
Hold this thought for another time: If you think you can replace a Data Warehouse, that is not a Data Warehouse, with another approach to ‘Data Warehousing’ that doesn’t produce a Data Warehouse, for as fast and cheap as one can do it, then you still don’t have a Data Warehouse to show for all of your efforts. That is not a great place to be.
Therefore, you see, Data Warehousing was never about a haphazard approach to providing random structured, semi-structured and unstructured data of various qualities, provenance, volumes, varieties and velocities, to whomever was of a mind to want it.
Many thanks for reading.
If you want to connect then please send a request. I you have any questions or comments then fire them off below. Cheers :-)
Oh… and one last thing before I go… DON’T FORGET TO JOIN THE BIG DATA CONTRARIANS: http://www.linkedin.com/grp/home?gid=8338976
03 Friday Jul 2015
Posted in Big Data, community, Good Strat, goodstart, goodstrat
Tags
Friends, peers and colleagues, lend me your bandwidth and 10 minutes of your time. Gather around and let me tell you about the greatest, most interesting and fantastically diverse Big Data and Data community right here in our very midst on this amazing LinkedIn community.
We have a new Big Data/Data group, and the group is aptly named The Big Data Contrarians, and yet it is neither a ‘me too’ group, of which there are too many to mention, or a ‘belief circle’, of which the less said, the better. Not, The Big Data Contrarians group is a place for cool opinion pieces, creative abrasion, practical insight and (within the realms of the possible) BS free comment.
However, before going into more detail about the group, I would like to digress for a moment.
Like many people, I take a lot of inspiration from outside my own professional spheres of practice, principles and technologies, and this is no less true when it comes to advertising.
Two of my real influencers – the real kind not the LinkedIn kind – are advertising legends Dave Trott (also author of Predatory Thinking) and Bob Hoffman (the Ad Contrarian), who are exceptionally experienced, talented and creative people, of the NoBS (no flim-flam) kind. Indeed, it was after reading some of Bob’s and Dave’s recent articles that I decided to get this group registered on LinkedIn, which, love it or loath it, is where many of us connect.
So, I hear you ask “What’s The Big Data Contrarians, Mart?”
Okay, to be fair, The Big Data Contrarians group is about far more than just being contrarian and a legitimate means of inciting discussion, for as reasonable as that is. It’s also about arguing against or openly rejecting mistakenly cherished and contrived Big Data beliefs and ‘institutions’ and established Big Data hype, speculation and opinion. It’s about separating Big Data fads, fantasises and folk-tales from Big Data reality.
What we seek to understand and convey is where, when, how and for what ends data (including Big Data) can be used to derive legitimate benefits. Moreover, stated from a position of reason and facts, and not simply projected as an issue of Big Data faith, speculation and clairvoyance.
On the other side, we can call out the Big Data hype for what it is, and just as Bob Hoffman calls out the social media and advertising BS babblers in his trade, this too lends a platform for people to do the same with the disreputable and dubious practices of Data gurus, courtesans and ‘influencers’.
“So, Mart, is being a Big Data Contrarian a bit like being a Big Data Luddite?”
Well, not really, but the problem with having so many people who are new to IT is that the past is a mystery top them, so anything that is new to them is actually taken as new, whether it is new or not.
Those who know will know that technologies of distributed file stores and search over unstructured data has been around for quite some time, and some of the “new” technologies that we big-up today, are actually simple developments of data technologies that go back to the seventies and eighties, or maybe even before.
However, this is not essentially about being anti-technology or even in advances in the application of technology, but of understanding that it isn’t helpful for the media, the big industry players and their indentured acolytes, to railroad, cajole and bully businesses into buying Big Data technology they don’t need, to solve Big Data problems and opportunities they don’t have.
That said, it’s up to the members of The Big Data Contrarians to decide on what shape the community should take, and as it is an open forum in democratic terms, the members have equal rights in presenting their own opinions, lessons learned and other insights.
So, if you haven’t yet drunk the Big Data kool-aid, come on down to The Big Data Contrarians, the place for everyone interested in Big Data/Data and its many potential uses.
Many thanks for reading.
Of course, this piece will also not feature on LinkedIn’s Big Data channel, because apparently that channel editor (naming no names) doesn’t like anyone raining on their particular Big Data flim-flam parade.
#BigData #BigDataChannel
22 Friday May 2015
Posted in business strategy, Consider this, Good Strat, goodstrat, IT strategy, Strategy
I used to work for an affable person from Chicago. His two favourite phrases were “Let’s talk strat” and “Brought your cheque book with you?”
There are many misconceptions about strategy. But, I particularly want to address two things:
So, without more ado, let’s get the baby off the ground.
Continue reading16 Saturday May 2015
Posted in Big Data, Consider this, Good Strat, Good Strategy, Martyn Richard Jones, Strategy
Tags
Listen up Big Data playmates! The ubiquitous Big Data gurus, tied up in their regular chores of astroturfing mega-volumes, velocities and varieties of superficial flim flam, may not have noticed this, but, Hadoop is getting set up for one mighty fall – or a fast-tracked and vertiginous black run descent. Why do I say that? Well, let’s check the market. Continue reading
01 Friday May 2015
Posted in Big Data, Big Data Analytics, Consider this, Good Strat, goodstart
Simply stated, the best application of Big Data is in systems and methods that will significantly reduce the data footprint.
Why would we want to reduce the data footprint?
20 Friday Mar 2015
Posted in Big Data, Consider this, good start, Good Strat, goodstart, Martyn Jones
Most of us would probably like to work in a profession recognised for its legality, decency and honesty. At least I hope so. In my line of work, what we have right now is palpable evidence that the IT industry lacks a moral compass.
Imagine this. A major sensationalist tabloid pulls together a team of diverse journalists who are set to work on a national campaign to promote very high usage of sunbeds as a cure for cancer. Why? The newspaper owner’s son owns the sunbed franchise.
The health experts criticise the publisher for being irresponsible, unprofessional and lacking in scruples.
Continue reading14 Saturday Mar 2015
Posted in Consider this, good start, Good Strat, goodstart, Martyn Richard Jones
Tags
careers, Consider this, good start, Good Strat, Good Strategy, goodstart, Martyn Jones, Martyn Richard Jones, quit
You are the boss. You are the leader, coach and manager, and there are some things that you just got to learn, like it or not. One of these skills is to be able to identify when someone has quit. “How dare they?” I here you ask.
The first time I quit a job and didn’t tell anybody was when I was in the RAF working as a fighter pilot in World War 2, and I accidentally bombed Newport in South Wales, and was given a stern talking to for my troubles. Well, I didn’t actually quit and I was never in the armed forces and I was born into the era of the Beat Generation, but that’s by the by, it’s just there for effect, to create some artificial empathy between me and those who have actually quit a job and not told anyone about it. Myself, I would never do such a thing. Although to be fair, Newport has looked like it has been freshly bombed with dark green, brown and grey shades of poster paints and self-raising flour, since forever. Continue reading
10 Tuesday Mar 2015
Posted in All Data, Big Data, Consider this, dark data, Good Strat
Tags
All Data, Big Data, dark data, data architecture, data management, Good Strat, Martyn Jones, Martyn Richard Jones
Dark data, what is it and why all the fuss?
First, I’ll give you the short answer. The right dark data, just like its brother right Big Data, can be monetised – honest, guv! There’s loadsa money to be made from dark data by ‘them that want to’, and as value propositions go, seriously, what could be more attractive?
Let’s take a look at the market.
Gartner defines dark data as “the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes” (IT Glossary – Gartner)
Techopedia describes dark data as being data that is “found in log files and data archives stored within large enterprise class data storage locations. It includes all data objects and types that have yet to be analyzed for any business or competitive intelligence or aid in business decision making.” (Techopedia – Cory Jannsen)
Cory also wrote that “IDC, a research firm, stated that up to 90 percent of big data is dark data.”
In an interesting whitepaper from C2C Systems it was noted that “PST files and ZIP files account for nearly 90% of dark data by IDC Estimates.” and that dark data is “Very simply, all those bits and pieces of data floating around in your environment that aren’t fully accounted for:” (Dark Data, Dark Email – C2C Systems)
Elsewhere, Charles Fiori defined dark data as “data whose existence is either unknown to a firm, known but inaccessible, too costly to access or inaccessible because of compliance concerns.” (Shedding Light on Dark Data – Michael Shashoua)
Not quite the last insight, but in a piece published by Datameer, John Nicholson wrote that “Research firm IDC estimates that 90 percent of digital data is dark.” And went on to state that “This dark data may come in the form of machine or sensor logs” (Shine Light on Dark Data – Joe Nicholson via Datameer)
Finally, Lug Bergman of NGDATA wrote this in a sponsored piece in Wired: “It” – dark data – “is different for each organization, but it is essentially data that is not being used to get a 360 degree view of a customer.
Okay, let’s see if we can be a bit more specific about the content of dark data?
Items on the dark data ticket include: Email; Instant messages; documents; Sharepoint content; content of collaboration databases; ZIP files; log files; archived sensor and signal data; archived web content; aged audit trails; operational database backups – full and incremental; roll-back, redo and spooled data files; sunsetted applications (code and documentation); partially developed and then abandoned applications; and, code snippets.
Most importantly, dark data is data that is not actively in use, is underutilised, or is something else. Seriously.
So, the conclusion that some have come to is this: there is a vast collection of data in various formats waiting to be monetised.
Personally, the idea that really grabs my attention is the potential ability to do novel forensic research on email. If only to find out what happened in the past.
For example, maybe it would be fascinating to see how significant challenges were identified, flagged and discussed; how strategic responses to those challenges were formulated, chosen and executed; and, how the outcomes of all of that process were reflected in email communications.
I think that this line of work can be very interesting for some people, and that interesting insights may be uncovered, but I would hate to have to put a tangible value on it, if only to avoid adding to the already galactic magnitudes of nonsense and hype surrounding certain data topics.
There are other more mundane uses of dark data.
Imagine that you are just about to embark on a Data Warehouse project (you really are a late adopter aren’t you), and you want establish a base collection of historical data. Where do you get that historical data from?
Right! Operational databases are not characteristically used to store significant amounts of historical reference data and historical transactions beyond a certain time window; there are performance and other reasons for keeping OLTP systems as lean as possible, so, initial loads of historical data is typically recreated in the Data Warehouse from backups, audit trails or logs.
You don’t need a Chief Data Officer in order to be able to catalogue all your data assets. However, it is still good idea to have a reliable inventory of all your business data, including the euphemistically termed Big Data and dark data.
If you have such an inventory, you will know:
What you have, where it is, where it came from, what it is used in, what qualitative or quantitative value it may have, and how it relates to other data (including metadata) and the business.
What needs to be kept, and for how long, and what can be safely discarded, and when.
The risks associated with the retention or loss of that data.
If you don’t have such a catalogue and have never done a data inventory then a full data inventory and audit seems to be your new best friend.
Simply stated, you may have dark data that has value, or it may be a simple collection of worthless digital nostalgia. But if you don’t know what you have, it may pay to find out what’s there, and if necessary, to let it go.
There is no point in hoarding unneeded and unwanted rubbish data. That is simply not good data management.
Finally a word on all the fuss surrounding dark data.
Failure to monetize when there is value to be obtained from dark data is one thing, claiming that value can be invariably obtained whilst actually not knowing what the data is, or how it could be monetised, is just adding to the mountain of data related ‘nonsense and hype’ doing the rounds these days. Please consider not adding to that mountain.
British Rail, the national UK rail Company, used to be notorious for the number of delays and cancellations to services, and their reasons for failing to meet their obligations became stranger and stranger.
In winter, it would snow and there would be problems. And people would ask ‘how come you couldn’t deal with the snow this year, we’ve had snow for centuries?’ And back came the answers ‘Yes, Sir, but this year it was the wrong type of snow’. In autumn (the fall), it was ‘the wrong types of leaves, and ‘the wrong type of rain’, and in Summer, the ‘wrong type of sunshine’ and so on and so forth.
I hope this will not be the excuse from the Big Data and dark data pundits and punters when the much-vaunted and ‘almost’ guaranteed monetisation isn’t frequently realised.
‘Of course Big Data gives you big dollar benefits, it was just littered with the wrong type of data’ or ‘you just weren’t trying hard enough’.
Many thanks for reading.
06 Friday Mar 2015
Posted in Big Data, Consider this, Data Warehousing, Good Strat, hadoop, hdfs, Martyn Jones
Tags
Big Data, enterprise data warehousing, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Jones
Hold this thought: To paraphrase the great Bob Hoffman, just when you think that if the Big Data babblers were to generate one more ounce of bull**** the entire f****** solar system would explode, what do they do? Exceed expectations.
I am a mild mannered person, but if there is one thing that irks me, it is when I hear variations on the theme of “Data Warehousing is Big Data”, “Big data is in many ways an evolution of data warehousing” and “with Big Data you no longer need a Data Warehouse”.
Big Data is not Data Warehousing, it is not the evolution of Data Warehousing and it is not a sensible and coherent alternative to Data Warehousing. No matter what certain vendors will put in their marketing brochures or stick up their noses.
In spite of all of the high-visibility screw-ups that have carried the name of Data Warehousing, even when they were not Data Warehouse projects at all, the definition, strategy, benefits and success stories of data warehousing are known, they are in the public domain and they are tangible.
Data Warehousing is a practical, rational and coherent way of providing information needed for strategic and tactical option-formulation and decision-making.
Data Warehousing is a strategy driven, business oriented and technology based business process.
We stock Data Warehouses with data that, in one way or another, comes from internal and optional external sources, and from structured and optional unstructured data. The process of getting data from a data source to the target Data Warehouse, involves extraction, scrubbing, transformation and loading, ETL for short.
Data Warehousing’s defining characteristics are:
Subject Oriented: Operational databases, such as order processing and payroll databases and ERP databases, are organized around business processes or functional areas. These databases grew out of the applications they served. Thus, the data was relative to the order processing application or the payroll application. Data on a particular subject, such as products or employees, was maintained separately (and usually inconsistently) in a number of different databases. In contrast, a data warehouse is organized around subjects. This subject orientation presents the data in a much easier-to-understand format for end users and non-IT business analysts.
Integrated: Integration of data within a warehouse is accomplished by making the data consistent in format, naming and other aspects. Operational databases, for historic reasons, often have major inconsistencies in data representation. For example, a set of operational databases may represent “male” and “female” by using codes such as “m” and “f”, by “1” and “2”, or by “b” and “g”. Often, the inconsistencies are more complex and subtle. In a Data Warehouse, on the other hand, data is always maintained in a consistent fashion.
Time Variant: Data warehouses are time variant in the sense that they maintain both historical and (nearly) current data. Operational databases, in contrast, contain only the most current, up-to-date data values. Furthermore, they generally maintain this information for no more than a year (and often much less). In contrast, data warehouses contain data that is generally loaded from the operational databases daily, weekly, or monthly, which is then typically maintained for a period of 3 to 10 years. This is a major difference between the two types of environments.
Historical information is of high importance to decision makers, who often want to understand trends and relationships between data. For example, the product manager for a Liquefied Natural Gas soda drink may want to see the relationship between coupon promotions and sales. This is information that is almost impossible – and certainly in most cases not cost effective – to determine with an operational database.
Non-Volatile: Non-volatility means that after the data warehouse is loaded there are no changes, inserts, or deletes performed against the informational database. The Data Warehouse is, of course, first loaded with cleaned, integrated and transformed data that originated in the operational databases.
We build Data Warehouses iteratively, a piece or two at a time, and each iteration is primarily a result of business requirements, and not technological considerations.
Each iteration of a Data Warehouse is well bound and understood – small enough to be deliverable in a short iteration, and large enough to be significant.
Conversely, Big Data is characterised as being about:
Massive volumes: so great are they that mainstream relational products and technologies such as Oracle, DB2 and Teradata just can’t hack it, and
High variety: not only structured data, but also the whole range of digital data, and
High velocity: the speed at which data is generated, transmitted and received.
These are known as the three Vs of Big Data, and they are subject to significant and debilitating contradictions, even amongst the gurus of Big Data (as I have commented elsewhere: Contradictions of Big Data).
From time to time, Big Data pundits slam Data Warehousing for not being able to cope with the Big Data type hacking that they are apparently used to carrying out, but this is a mistake of those who fail to recognise a false Data Warehouse when they see one.
So let’s call these false flag Data Warehouse projects something else, such as Data Doghouses.
“Data Doghouse, meet Pig Data.”
Failed or failing Data Doghouses fail for the same reasons that Big Data projects will frequently fail. Both will almost invariably fail to deliver artefacts on time and to expectations; there will be failures to deliver value or even simply to return a break even in costs versus benefits; and of course, there will be failures to deliver any recognisable insight.
Failure happens in Data Doghousing (and quite possibly in Big Data as well) because there is a lack of coherent and cohesive arguments for embarking on such endeavours in the first place; a lack of real business drivers; and, a lack of sense and sensibility.
There is also a willing tendency to ignore the advice of people who warn against joining in the Big Data hubris. Why do some many ignore the ulterior motives of interested parties who are solely engaged in riding on the faddish Big Data bandwagon to maximise the revenue they can milk off punters? Why do we entertain pundits and charlatans who ‘big up’ Big Data whilst simultaneously cultivating an ignorance of data architecture, data management and business realities?
Some people say that the main difference between Big Data and Data Warehousing is that Big Data is technology, and Data Warehousing is architecture.
Now, whilst I totally respect the views of the father of Data Warehousing himself, I also think that he was being far too kind to the Big Data technology camp. However, of course, that is Bill’s choice.
Let me put it this way, if Oracle gave me the code for Oracle 3, I could add 256 bit support, parallel processing and give it an interface makeover, and it would be 1000 times better than any Big Data technology currently in the market (and that version of Oracle is from about 1983).
Therefore, Data Warehousing has no serious competing paragon. Data Warehousing is a real architecture, it has real process methodologies, it is tried and proven, it has success stories that are no secrets, and these stories include details of data, applications and the names of the companies and people involved, and we can point at tangible benefits realised. It’s clear, it’s simple and it’s transparent.
Just like Big Data, right?
Well, no.
See what I mean?
Therefore, the next time someone says to you that Big Data will replace Data Warehousing or that Data Warehousing is Big Data, or any variations on that sort of ‘stupidity’ theme, you can now tell them to take a hike, in the confidence that you are on the side of reason.
Many thanks for reading.
Aligning Big Data: http://www.linkedin.com/pulse/aligning-big-data-martyn-jones
Big Data and the Analytics Data Store: http://www.linkedin.com/pulse/big-data-analytics-store-martyn-jones
A Modern Manager’s Guide to Big Data:http://www.linkedin.com/pulse/managers-guide-big-data-context-martyn-jones
Accomodating Big Data
01 Sunday Mar 2015
Posted in Big Data, Consider this, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Jones
Please note: This is an edited version of a previous piece with a similar name, but focusing solely on the three main Vs of Big Data.
What we’ve been toldWe’ve been told that business Big Data is the greatest thing since sliced bread, and that its major characteristics are:
Which is a simple and straightforward means of classification. Big Data is about massive volumes, high variety and high velocity. Right?
It’s not about bigI have never bought into the idea that more data is necessarily better data, or that it provides better focus or leads to increased insight, in fact I have been quite vocal with my contrarian opinion, but now this view is getting some additional support, and from some surprising corners.
In a recent blog piece on IBM’s Big Data and Analytics Hub (Big data: Think Smarter, not bigger), Bernard Marr wrote that “the truth is, it isn’t how big your data is, it’s what you do with it that matters!”
Over at Fierce Big Data it was Pam Baker who stated that “the term big data is unfortunate because it’s really not about the size of the data”. (Big data is not about petabytes, but complex computing).
Elsewhere, SAS echoed similar sentiments on their web site: “The real issue is not that you are acquiring large amounts of data. It’s what you do with the data that counts.”
Well, apparently Big Data isn’t about “massive volumes” of data.
Strike 1!
It is claimed that 20% of digital data is structured, it is based on the problematic suggestion that structured data is uniquely relational.
It is also said that unstructured data includes CSV files and XML data, and this makes up far more than the 20% of the data generated. But this definition is wrong.
If anything, CSV data is structured, and XML data is highly structured, and it’s typically regular ASCII data. So there it does not add variety, even though it is not structured in the ways that some someone might expect, especially if that someone lacks the required knowledge and experience. Simply stated, CSV data is structured, it’s just that it lacks rich metadata, but that doesn’t make it unstructured.
“But”, I hear you say “what about all the non-textual data such as multi-media, and what about the masses of unstructured textual data?”
Take it from me, most businesses will not be basing their business strategies on the analysis of a glut of selfies, juvenile twittering, home videos of cute kittens, or the complete works of William Shakespeare. Almost all business analysis (whether done by a professional statistician or a data scientist) will continue to be carried out using structured data obtained primarily from internal operational systems and external structured data providers.
Variety, Sir? No problem.
Strike two!
So, if we accept that Big Data isn’t really about the massive data volumes or high data variety then that leaves us with velocity. Because if it isn’t about record breaking VLDB or significant data variety, then for most commercial businesses the management of data velocity becomes either less of an issue or just is no issue.
Even in some extreme circumstances, one can explore the suggestion that data sampling can remove issues with data volume as well as velocity.
However, the fact that some software vendors and IT service suppliers set up this‘straw man’ velocity argument and then knock it down with the ‘amazing powers’ of their products and services, is quite another matter.
So, is it really about velocity?
Strike three!
Big Data is a dopey term, applied necessarily ambiguously to a surfeit of tenuously connected vagaries, and its time has come and gone. Let’s dump the Big Data moniker, and the 3 Vs along with it, and embrace the fact that data is data, there will always be more of it.
So, let’s consider ‘all data’ and principally for its time and place utility.
If there is something that you are not sure about or have questions with then please leave a comment below or email me.
Thanks very much for reading.
17 Tuesday Feb 2015
Posted in Big Data, Consider this, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Jones, Strategy
Tags
Analytics, Big Data, data management, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Jones
Hold this thought: Big Data is King.
Is there just nothing that Big Data isn’t capable of fixing? From terrorism, world hunger, Ebola, HIV, fraud, money laundering and hiring the ‘right’ people through to winning the lottery, curing hangovers, arranging entrapment and finding the love of your life. Big Data is King. Continue reading
14 Saturday Feb 2015
A helpful slideset that is used to explain the purposes, positions and roles of Knowledge Management.
Enjoy! Please tell me what you think about this slide deck. Many thanks for viewing.
12 Thursday Feb 2015
Posted in Big Data, Consider this, Good Strat, Information Management, Martyn Jones
Normally I would send such people to see a specialist – no, not a guru, but a sort of health specialist, but because this has happened to me so many times now, I eventually decided to put pen to paper, push the envelope, open up the kimono, and to record my advice for posterity and the great grandchildren.
So, here are my top seven tips for cashing in quick on the new big thing on the block.
1 – A business opportunity for faith
Like every new religion, trend or fad, Big Data has its own founding myths, theology and liturgy, and there is money to be made in it; loadsa lovely jubbly money. By predicating and evangelising Big Data you will be welcomed with open arms into the Big Data faith, and will receive all the attendant benefits that will miraculously and mysteriously fall upon you and your devout friends. Go on, I dare you. Be a Big Data guru, a shepherd to a flock of sheep, and enjoy the wealth, health and happiness that most surely will come your way. You too can look cool in red Prada slippers, a flattering and flowing gown and matching accessories.
2 – Acquire it, multiply it, weigh it, mark it up and sell it on
Simply stated, this is about acquiring other people’s data, by sacred means or profane, marking it up and then selling it on. The value you add is that you act as a trusted conduit, a conduit for good. You may care to enrich the data, swop the order of data, replicate and embellish data, make stuff up, etc. which all serves to ‘add value’ to the data. You may even consider adding nuggets of value to the data, just for kicks and giggles. My best friend’s favourite is injecting the good old ‘diaper and beer’ and ‘friends and family’ clichés into every Big Data collection, as it never fails to thrill, please and delight.
3 – Anything can be anything
The good thing about making money from Big Data is that it doesn’t need to be anything to do with Big Data. Make a 20GB Enterprise Data Warehouse? Call it a Big Data success. Sell 20 boxes of dodgy doughnuts down the alternative market? Proclaim a Big Data triumph. Sell your digital porn stash to your best mate? Point to the incredible invisible hand of the Big Data market at work. See what I’m doing there. Anything can be anything, and you too can cash in on that opportunity, big time.
4 – Big Data Patronage
Tense, nervous headaches? Do you like making up stories about Big Data, or for that matter anything else? Are you a natural born fibber but are strapped for cash? Then worry no longer. If you get a Big Data patron you will be sorted for ‘life’; get two and you’ll be sorted for the afterlife as well. With a Big Data patron you can get the most tenuous, crappiest and superficial of pieces published, promoted and vaunted – globally. Can’t make it up yourself, then outsource and offshore it, after all, just get the keywords right for SEO ranking and the gullible will flock to you in droves. The down side of this profession is that you will be targeted for writing half-truths, quarter-truths and downright lies, and you will be pilloried as a purveyor of rank hyperbole. But don’t worry, take heart and never lose the faith, you will be in good company. As one Big Data guru was want to say ” If you repeat a lie often enough, people will believe it, and you will even come to believe it yourself.” Amen! brother.
5 – Big Data Certification
By 2016 there will be global demand for 30 billion Big Data professionals. Are you prepared to cash in on that inevitability? No? Then consider this.
One of my best friends makes his living as a completely phony Big Data Scientist. For two hundred bucks he can make you a Data Scientist or a Big Data guru. Some guys give you an education but this guy gives you immediate access to high paying jobs, sex and a life in the city. Moreover, for an extra 250 bucks you can also become a certified Big Data Trainer, which will allow you to do unto others what has been done unto you.
6 – Creative Technology Reuse
Big Data has heralded in the biggest innovations known in the history of computing, and arguably in the entire history of humankind. One of those new inventions has been the now widely acclaimed and revolutionary ‘flat file data base’ (FFDB), and this has been accompanied with developments in low level operating system primitives that allow for the processing of these collections and hierarchies of FFDBs. So, if one has a mind to do so, one can get some real business leverage off of these new tendencies by borrowing 21st century technology found in old operating system hacks from the sixties and seventies and eighties and nineties and… Well, the point is that in order to get serious funding it is no longer good enough to have a half page business plan, it is also necessary to eke out ‘stuff’ that works within the new paradigms of Big Data and Big Data Analytics. For my next venture I will be looking for serious funding for my ‘Arbitrary Dawdle Down Data Street’ (AD3S) Big Data Analytics platform, a platform designed to support virtual 1k bit processing and the massively parallel provision of global regular expression search and match (S&M), concatenation and listing, and cooperative data-driven and streamed data extraction and reporting. I’m hoping to attract the attention of governments, the EU, the Manic Street Preachers, the UN, China, Vladimir Putin, the DOD, HP, Oracle, Gartner, Lana Del Rey, Deloitte and IBM. So, this is going to be absolutely massive. Word!
7 – Big Data Brokerage
According to leading management consultants and industry watchers Gartner, McKinsey and Deloitte, data needs to be managed and accounted like any other asset, such as money. To get into a similar view-point requires a massive leap of faith, but it is a conversion that might drive dividends. One avenue to be explored in eking out value from the apparently massively valuable Big Data lakes, silos and pools is through the operation of a Big Data Brokerage. A Big Data Brokerage is a business whose main responsibility is to be an intermediary that puts Big Data buyers and Big Data sellers together in order to facilitate a transaction. Big Data Brokerage companies are compensated via commission after the Big Data transaction has been successfully completed. They may also charge introductory fees. Just imagine the wealth of business opportunities in that. You could become the Goldman Sachs of data.
I hope you enjoyed this piece and would be pleased to hear your views on this and other subjects.
Whilst I understand the attraction and even the need of creating a new and significant growth industry, I would also advise a degree of restraint, and whilst I see that “Big Data” (the consideration of the potential value of All Data) has its allure, I also think that some good sense and informed caution should also prevail.
Thank you so much for reading.
Martyn Richard Jones
12 Thursday Feb 2015
Posted in Big Data, Consider this, Good Strat, Martyn Jones
Tags
Fueled by the new fashions on the block, principally Big Data, the Internet of Things, and to a lesser extent Cloud computing, there’s a debate quietly taking please over what statistics is and is not, and where it fits in the whole new brave world of data architecture and management. For this piece I would like to put aspects of this discussion into context, by asking what ‘Core Statistics’ means in the context of the DW 3.0 Information Supply Framework.
The following diagram illustrates the overall DW 3.0 framework:
There are three main concepts in this diagram: Data Sources; Core Data Warehousing; and, Core Statistics.
Data Sources: All current sources, varieties, velocities and volumes of data available.
Core Data Warehousing: All required content, including data, information and outcomes derived from statistical analysis.
Core Statistics: This is the body of statistical competence, and the data used by that competence. A key data component of Core Statistics is the Analytics Data Store, which is designed to support the requirements of statisticians.
The focus of this piece is on Core Statistics. It briefly looks at the aspect of demand driven data provisioning for statistical analysis and what ‘statistics’ means in the context of the DW 3.0 framework.
The DW 3.0 Information Supply Framework isn’t primarily about statistics it’s about data supply. However, the provision of adequate, appropriate and timely demand-driven data to statisticians for statistical analysis is very much an integral part of the DW 3.0 philosophy, framework and architecture.
Within DW 3.0 there are a number of key activities and artifacts that support the effective functioning of all associated processes. Here are some examples:
All Data Investigation: An activity centre that carries out research into potential new sources of data and analyses the effectiveness of existing sources of data and its usage. It is also responsible for identifying markets for data owned by the organization.
All Data Brokerage: An activity that focuses on all aspects of matching data demand to data supply, including negotiating supply, service levels and quality agreements with data suppliers and data users. It also deals with contractual and technical arrangements to supply data to corporate subsidiaries and external data customers.
All Data Quality: Much of the requirements for clean and useable data, regardless of data volumes, variety and velocity, have been addressed by methods, tools and techniques developed over the last four decades. Data migration, data conversion, data integration, and data warehousing have all brought about advances in the field of data quality. The All Data Quality function focuses on providing quality in all aspects of information supply, including data quality, data suitability, quality and appropriateness of data structures, and data use.
All Data Catalogue: The creation and maintenance of a catalogue of internal and external sources of data, its provenance, quality, format, etc. It is compiled based on explicit demand and implicit anticipation of demand, and is the result of an active scanning of the ‘data markets’, ‘potential new sources’ of data and existing and emerging data suppliers.
All Data Inventory: This is a subset of the All Data Catalogue. It identifies, describes and quantifies the data in terms of a full range of metadata elements, including provenance, quality, and transformation rules. It encompasses business, management and technical metadata; usage data; and, qualitative and quantitative contribution data.
Of course there are many more activities and artifacts involved in the overall DW 3.0 framework.
Statistics, it is said, is the study of the collection, organization, analysis, interpretation and presentation of data. It deals with all aspects of data, including the planning of data collection in terms of the design of surveys and experiments; learning from data, and of measuring, controlling, and communicating uncertainty; and it provides the navigation essential for controlling the course of scientific and societal advances[i]. It is also about applying statistical thinking and methods to a wide variety of scientific, social, and business endeavors in such areas as astronomy, biology, education, economics, engineering, genetics, marketing, medicine, psychology, public health, sports, among many.
Core Statistics supports micro and macro oriented statistical data, and metadata for syntactical projection (representation-orientation); semantic projection (content-orientation); and, pragmatic projection (purpose-orientation).
The Core Statistics approach provides a full range of data artifacts, logistics and controls to meet an ever growing and varied demand for data to support the statistician, including the areas of data mining and predictive analytics. Moreover, and this is going to be tough for some people to accept, the focus of Core Statistics is on professional statistical analysis of all relevant data of all varieties, volumes and velocities, and not, for example, on the fanciful and unsubstantiated data requirements of amateur ‘analysts’ and ‘scientists’ dedicated to finding causation free correlations and interesting shapes in clouds.
This has been a brief look at the role of DW 3.0 in supplying data to statisticians.
One key aspect of the Core Statistics element of the DW 3.0 framework is that it renders irrelevant the hyperbolic claims that statisticians are not equipped to deal with data variety, volumes and velocity.
Even with the advent of Big Data alchemy is still alchemy, and data analysis is still about statistics.
If you have any questions about this aspect of the framework then please feel free to contact me, or to leave a comment below.
Many thanks for reading.
Catalogue under: #bigdata #technology big data, predictive analytics
[i] Davidian, M. and Louis, T. A., 10.1126/science.1218685
12 Thursday Feb 2015
Posted in Big Data, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Jones
Good morning fellow consumers; here’s a pop quiz question: What does Big Data have in common with Robitussin? Think about, take your time.
Okay, times up!
Robitussin is a legal pharmaceutical product commonly associated with coughs, colds and flu combinations. Continue reading
12 Thursday Feb 2015
Posted in Good Strat, goodstartegy, Martyn Jones
It’s no wonder that truth is stranger than fiction. Fiction has to make sense.
Mark Twain
It’s Friday morning in London’s trendy Canary Wharf, and I have been asked to facilitate a local meeting of the Digital Violence and Dogma Victims Group, the self-help recovery chain for those who have fallen foul of the pernicious and debilitating effects of IT dogma, organisational autism and insider thuggery and blackmail.
There are twelve of us in the old church hall. We sit in a circle, to facilitate communication. After a more formal welcome and brief introduction the floor is opened up for people to talk about whatever they want to talk about. There is silence. This is normal. There are a few new faces.
“Pantxo!” I look across at Pantxo; he is staring out the window at the falling rain. He can usually talk the legs off a giraffe, but today he is having none of it. Sensing that things are not going too well, I enter into my routine of floridly and inanely relating well-worn anecdotes from the distant annals of IT history.
As I am entering my tenth lap of the track of tedium, one of the new members picks up enough courage to chime in, first nervously and then with the increasing confidence of someone who knows exactly what they are talking about and precisely what they are going to say.
“Hello. My name is Crème”. A woman in a blue adidas tracksuit looks around the room.
“Yes. My name is Crème Brûlée; you may well have heard of it from twitter, the tabloids and the TV… oh, and the novel Absolute Beginners… I used to be the CIO of a major household name.”
She pauses and looks into the middle distance, searching for the truth, tip toeing around the pain.
“This is a bit embarrassing – awkward maybe would be a better word – but what I want to unburden upon you all today is the story of how I outsourced my Data Warehouse, my Business Intelligence, my Big Data, my MDM, my CRM, my family and my life”.
She takes a deep breath and continues; making a point of looking at each of her fellow members in turn as she does so.
“About five summers ago, I feeling a bit lost, which was unusual for me, a strange and novel experience, so I decided that I really needed to do something to turbo-charge my career prospects and to get things moving faster in my part of the organisation. I wanted to excel, and I wanted to be seen doing so, by the right people, and recognised as such.”
“In the spring of that year I had been to a management conference with some of our senior IT management team, some of whom are also here today. Okay, I won’t single out any one of you, because you know who you are.”
“As part of the week-long conference we were wined and dined, stroked and cajoled, flattered and sweet talked by a whole entourage of sales execs from the technology and service providers. They were telling us that the future was in outsourcing and offshoring as much as we could, yes even Big Data and Data Warehousing and Analytics, and they were bewitching us with stories of future successes, of IT paradise and professional nirvana. We in turn wanted to believe, needed to believe, desired to believe. All of this was reinforced by the so called independent industry analysts who insisted, in their agnostic way, that we should seize the moment, with courage, determination and illusion.”
“When I got back to the ranch my mind became occupied with other things, but I didn’t entirely forget the compelling messages that I had brought away with me from the conference.”
“Nothing happened for a couple of months until, one day and out of the blue, things came to a head.”
“We had recently acquired a media news and entertainments business – Media Macaroni International, and we were planning on integrating their general ledger into the corporate IT landscape. One morning I received a call from the CIO of the newly acquired company, inviting me to their site for a meet and greet event.”
“So I moseyed on down to Tinsel Town and got a briefing from not only the CIO, but the full board of directors of Media Macaroni, the ‘hasta la pasta’ of Big Data Analytics ad-hoc performance alignment.”
“To cut a long story short, they basically put me on the spot. Either I integrate the entire Data Warehousing, MIS, Big Data, Analytics and MDM across the expanded corporate body in 9 to 15 months, or we would have serious problems of convergence and market credibility. The message couldn’t have been clearer. Either I got our act together and made this acquisition work, or what looked like a humongous hot potato could land in my lap anytime soon. It was a career changing risk that I needed to address.”
“I told the directors there and then that the mission was going to be incredibly difficult to fulfil. However, the mood quickly changed.”
“Their CIO looks across the table and tells me that he can help me out of my hole. My hole? What the freak! You see, we have employed a service company that does most of our IT work for us, and according to us at Media Macaroni they are simply the bee’s knees, the best thing since chopped liver on rye, the biz.”
“So ‘who are these guys’, I ask. And after a brief hiatus that seemed to last forever, back came the ominous response: The Taffia Connection.”
To be continued…
Many thanks for reading.
Channel: #IT #BigData
As always, please share your questions, views and criticisms on this piece using the comment box below. I frequently write about strategy, organisational leadership and information technology topics, trends and tendencies. You are more than welcome to keep up with my posts by clicking the ‘Follow’ link and perhaps even send me aLinkedIn invite. Also feel free to connect via Twitter, Facebook and the Cambriano Energy website.
File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro
08 Sunday Feb 2015
Posted in Big Data, Consider this, Good Strat, Good Strategy, Martyn Jones
Tags
Analytics, Big Data, data lake, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Joens, Stories, trading
The Wow Wow Factor! Trading, Big Data and 7 HabitsHi, I’m Ricky Jones, boss and co-founder of Becci Boo International Investments. Last week we said goodbye to our best ever Big Data energy commodity trader. He’d been with us for years.
Sadly, Coco Jones was determined to retire to the countryside, to his birthplace, to his real home, a snug little village in the hills of Montseny, and there was absolutely nothing we could do to convince him to stay. The thing about Coco is that he is not like you or me, he’s a highly intelligent Catalan sheepdog.
So, you might ask, how did Coco get to be a star trader at the Becci Boo Hedge Fund? Was it the tools he used? Was it the techniques he adopted? Was it the food he ate? What was so special about him and his amazing abilities? It’s a long story that I will relate as briefly as I can.
Back in time, there was one particularly disastrous week of trading at Becci Boo. Something had gone really wrong with our once reliable Big Data Trade Analytics platform, and wrong bets were being placed right, left and centre – and against trader’s better judgement. The CFO was livid. Out he comes onto the trading floor, swearing and blinding. “God! You guys are the damn pits! What the hell do you think you are doing? Can’t you get anything right? A Catalan sheepdog could trade more effectively than you feckless lot of feckless things.”
I try and diffuse the situation. “Come on, Jordi, don’t be like that, we’re only human and this is a tough business.” “You don’t believe me” he replies. “Sure, but you’re not going to convince me that a Catalan sheepdog could be a substitute for a highly experienced human trader are you?” “How much do you want to bet? These superior four legged beings would never place any faith in things that they don’t understand or can’t control, and they certainly wouldn’t need your Big Data gizmo to make a success of things.”
So, in the following weeks we arrange to run an experiment. We ring around all the owners of registered Catalan sheepdogs and ask them if they would like their dog to take part in our simple trading experiment. Food, lodging and generous expenses all included. We easily manage to get together 64 Catalan sheepdogs and their owners from all over Europe. The experiment we design is quite simple.
We give each dog a gadget (and actually for the more sophisticated dogs it was a smart phone and specially designed app) with two big buttons on it, a red button and a yellow button.
Every Monday morning we ask the assembled dogs to press one of two buttons depending on whether they think that the chosen energy commodities market will close higher or lower at the end of the week.
Red button for a higher closing price, yellow button for a lower closing price. As you might imagine dear reader it’s a walk in the park for these cunning canines. To make sure that all of the participating dogs have equal and fair chances in the experiment, we provide guardians with access to all our big data analytics, and we have multiple screens set up in each studio apartment so that each dog can follow all of the financial news that they need to take in and then view the results of machine learning, data mining and predictive analytics.
At the end of week one, 32 dogs remain in the experiment, and the other 32 are sadly sent home. Every week the exercise is repeated, and every week less dogs move on to the next round. By the end of week 5 only two dogs remain. By the end of week 6 we have a sole winner, Afi Bastò, who has accurately predicted the market movement of our energy commodities market for a record six weeks in succession. It’s an amazing success that proves Jordi right. Unfortunately, this is where things start to go wrong. Our marketing department tweets the news of the amazing Afi and it immediately goes viral.
Articles appear in the Wall Street Journal, Financial Times, Economist, Cinco Dias and Les Échos, hailing the amazing brilliance of the newly discovered Catalan commodity trader.
Hundreds of interviews are held and millions of photos make the rounds of the social and professional networks. Over the following year Afi is interviewed, researched, studied and investigated. Academic papers are written about him.
Amazing claims are made about his canine knowledge, wisdom and experience. This is shortly followed by the publication of a plethora of bestselling business books, with titles such as: The 7 Habits of the Highly Effective Canine – Personal Lessons in Trading; Be More Afi and Grow Rich; The Gos d’Atura That Conquered Chicago; Learn to Trade like Afi; Good Sheepdog, Great Afi; Who Moved My Dog Food; How Afi outperformed Big Data Analytics; etc.
Which unfortunately all goes to Afi’s head, and he begins to seriously lose his commodity trading mojo. So much so, that by the end of the year, we let Afi go, and we bring back “the incredible” Coco Jones, the dog that managed to get the market movements right for five weeks in a row, and who was only pipped at the post by “the amazing” Afi.
Since then, Coco has helped us to win far more than we lose, he has had an innate knack of being able to combine market knowledge with statistical analysis and a certain indescribable ‘insight’ that no one quite manages to understand never mind emulate, and all combined with an amazing caring character.
So, as you might imagine, we were really sad to see him go. After his epic leaving party, where he is acclaimed by all and sundry, but especially by Jordi, all the staff assembled outside the office entrance to see him off, and as he was slowly chauffeured away down the driveway he turned and looked back, and his face said it all. “I was lucky — I found what I loved to do early in life”.
Many thanks for reading.
Here’s Coco paddling in his Big Data Lake
File under: Good Strat, Good Strategy, Martyn Richard Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro
06 Friday Feb 2015
Posted in Consider this, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Jones
See no evil, speak no evil, and hear no evil. Bad managers love to hear good news, leaders thrive on adversity, contradiction and criticism, but some bosses don’t know their right foot from their left ear.
Sure, there are things a lot of us would prefer not to hear. But sometimes things are just unavoidable. We are told that honesty is the best policy, but what happens when honesty goes wrong?
Here are some examples of comments that might piss your boss off, together with some suggestions on how to finesse your way out of such situations and crawl back into favour.
One: “Honey, I shrunk the Big Data…!”
This is a really difficult one. On the one hand your boss might take the news badly and run around like Chicken Licken for days on end lamenting the ‘fact’ that the sky has just fallen in. On the other hand, you might have a sensible, intelligent and sane boss, in which case you might like to follow it up with a “shall I stick it back in the spin dryer and give it another whirl?”
Two: “Isn’t it your bedtime already?”
No, no and no! This is so wrong, and on so many levels. First, avoid a question that ends with an ‘already’, this is far too formal for office banter. Next, consider the time. If it’s before 21:00 it is really not the moment to start asking these sorts of questions. Save this type of question for the regular night out with the project team or for anonymous SMSs.
Three: “Yes, your bum does look big in that 1k USD suit…”
Nobody likes being told that they have spent ‘loadsa’ money on sharp ‘schmutter’ that doesn’t exactly flatter, especially when it comes to naturally portly or big boned types. One way out of this difficult situation, if you really want a way out of this difficult situation, is to add a quick “sorry, only joking, you don’t look even half as bad as me dear old granddad”.
Four: “What did your last slave die of?”
Say you were busily serving tea in the Oval Office and President Obama asked you to pour some more milk into his cup, this would not be the phrase to use under any circumstances if this happened. In other circumstances it might be perfectly acceptable. Just imagine that instead of Obama it was Uncle Joe Biden who was asking. Then you would possibly be right to use that phrase, and would be free to follow it up with a “and who gave you permission to use the office of the POTUS to entertain your mates, huh?”
Five: “My Mum won’t be happy with this… and you know what that means”
Yes, he gets it, its blackmail, and he won’t like it. He won’t like the thought of not getting any… Well, you know what I mean. No need to spell out these sorts of things, is there, especially before the nine o’clock watershed. But, if this just slipped out accidentally then the best way of retracting it is to deny that you ever said it in the first place. Yes, I know this is not very ethical, but it’s the height of modern day ‘professionalism’.
Six: “You run like a girl…”
Your boss may behave like a highly socialised two year old, but the use of gender specific insults is a definite ‘no, no’. Of course there are exceptions. Your boss may be from a tribal ethnic minority and may have been named Runs Like Girl by his parents, so in that case it might be totally acceptable to use the name. But, it’s really best to ask first, just to be on the safe side.
Seven: “Okay, keep your hair on Mussolini, you’ll never sell any ice creams with an attitude like that!”
Dodgy one, under almost any conditions. There are however times when this might just work positively in your favour. For example, if the bald headed and rotund boss is a keen and nostalgic fascist sympathiser, one with a penchant for the old gelati celesti. In which case, you might want to follow it up with a rousingly jolly accusation such as “Fascist!” If that’s not the case or you are unsure, then it’s really best to avoid such language.
None of these expressions particularly bother me, but there are horses for courses, braces for races and boors for moors. If you are one of those charming ‘holier than thou’ thin-skinned puritans then you probably have a plethora of pet peeves of your own, in which case please join in the fun, and contribute your own ‘phrases I like to hate’, below.
Many thanks for reading.
Channels: #Careers: Getting Started #leadership
File under: Good Strat, Good Strategy, Martyn Richard Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro
03 Tuesday Feb 2015
Posted in Big Data, Consider this, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Jones
You walk down the street, maybe in London, Paris and Rome – well, of course, not all at the same time.
You look out of your spacious and luxurious apartment in Manhattan, Unter den Linden or Mahim.
You are cycling slowly along the streets of Amsterdam, Bonn or Zurich.
Got the picture? Continue reading
30 Friday Jan 2015
Posted in Banking, Consider this, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Jones, Strategy
Tags
Banking, finance, funds, Good Strat, Good Strategy, hedge funds, Martyn Jones, Martyn Richard Jones, money
Hedge Funds are evil, right? Go on, you know you want to say yes. Almost everyone has an opinion about them, but very few people can actually tell you what they are.
Indeed, there’s am awful lot of nonsense written about Hedge Funds, and this piece might just end up being a worthy addition to that body of baloney. But, the intention is somewhat different.
The objective behind this piece is to provide a quick look at where the modern hedge fund started; what they are; how they work; the mechanics of participation; and who traditionally has put their money into them.
Of course, this piece is a necessary simplification of what is a fascinating aspect of the alternative investment universe.
In 1966 Carol J. Loomis[i] blew the lid on one of the best kept investment secrets of the 20th century.
In an article penned for Fortune titled “The Jones that nobody keeps up with”, Loomis revealed that over a five year period a fund run by Alfred Winslow Jones had consistently outperformed the Fidelity Fund, the most successful mutual fund of that time, by a remarkable 44%.
Not only that, but between 1956 and 1966 the Jones fund had outperformed the Dreyfus Fund, the best performing mutual fund of that decade, by a massive 87%.
Jones was born in Melbourne, Australia, but from the age of four he lived in the USA. He graduated from Harvard in 1923, and before becoming involved in the finance industry he toured the world working on steamships. He was to serve as a diplomat in Germany, and also worked as a journalist covering the Spanish civil war.
In 1941, with conflict raging in Europe, Jones returned to the USA. He then studied for, and obtained a doctorate in sociology at Columbia University, and became a reporter for Fortune.
His thesis, Life, Liberty and Property, is a reference text in sociology.
In 1949 Jones formed a company, A. W. Jones & Co., arguably the first modern Hedge Fund.
Robert A. Jaeger characterised the fund as “an opportunistic equity hedge fund”, that relied heavily on discerning stock picking abilities, combined with bets on long positions (rising prices) and short positions (falling prices).
In 1952 the fund was converted into a limited partnership, and during the 50s other such partnerships were set up, including the sage of Omaha’s Buffet Partners, and WJS Partners, founded by Walter Schloss.
Hedge funds are loosely regulated, exclusive and limited-membership investment clubs, usually run as partnerships or as corporations. They focus on absolute returns on investments, for themselves and their members, regardless of market conditions.
Direct participation in Hedge Funds is theoretically limited to between 100 or 500 investors, depending on the class of investor. Moreover, because of the unregulated status of most hedge funds, they are not allowed to actively market their products, so they have to use more exclusive means to attract investors, typically word of mouth.
Hedge Funds typically invest in traditional securities, such as stocks, bonds and commodities, but they can also invest in real estate, art, wine or any number of other non-traditional areas of investment. In fact, they are free to use virtually any pick and mix of strategies from the entire range of investment possibilities.
That said, a lot of Hedge Funds will opt for a specific investment strategy and will stick with that strategy for the life-time of the fund.
So what’s in it for the Hedge Fund managers and administrators?
Hedge Fund Managers typically charge a management fee of between 1and 3 percent of the value of the assets under management, regardless of performance. They may also – almost always in the past – charge a performance fee, which can start at around 20 percent of any fund gains above a certain minimum performance hurdle or target value. Managers also generally ‘eat their own dog food’, in that they will also invest in their own Hedge Fund.
Investors in Hedge Funds are informed of the value of their investments via a statement that shows the calculated value of shares in the fund, the Net Asset Value (the NAV). This can be calculated monthly, quarterly or even yearly, depending on the fund. In addition funds are free to choose if they wish to publicly disclose performance figures or not.
Some hedge funds may require additional fees and commissions, and may impose lock up periods, and strict and narrow redemption periods. They may also reject some applications for subscriptions without giving any reasons, and they may also forcibly redeem shares held, and without having to justify their actions.
In addition, some hedge funds use equalization methods – and there are a number of variants – to equitably distribute hedge fund fees amongst its partners, yet other hedge funds do not use equalization methods at all.
The mechanics of participationSo, briefly, how do you get to invest in a Hedge Fund (subscribe), how do you liquidate that participation (redeem), and what happens between ‘subscription’ and ‘redemption’.
Offer: A Hedge Fund details what’s involved in a particular offer in an ‘Offering Memorandum’, also known as a ‘private placement memorandum’. This is typically an ‘enriched’ business plan tied to a specific issuance of shares in a fund. It basically sets out the stall.
Subscription: In order to subscribe to a fund the potential participant in the fund signs up to a subscription agreement, and the conditions laid out in that agreement. Conditions may cover aspects such as minimum subscription amounts; minimum share increments, rules governing the liquidation of participation in the fund; management and performance fees; and, so on and so forth.
Redemption: This is the liquidation of shares in a fund. Typical redemption points can occur from anything from 15 days to up to 180 days, and sometimes more than this, depending on the fund and the rules related to lock-ups, redemption. In addition, redemption options may be linked to other financial charges, penalties and constraints.
So what happens between subscription and redemption?
Custody: A hedge fund subscriber may wish to use the services of a large and reputable financial service provider to act as custodian of their hedge fund shares. In addition to holding securities for safekeeping, most custodians also offer other services such as account administration, transaction settlements, collection of dividends and interest payments, tax support and foreign exchange. (Source: Investopedia).
Dividends: It is very uncommon for hedge funds to pay dividends, as any accruable earnings are realized upon redemption of the shares. However, some fundsdo incentivize the maintenance of subscriptions through the payment of dividends.
Calculating NAV: Periodically – or even on a real-time basis – a hedge fund will recalculate the Net Asset Value of the fund shares. This is done by dividing total value of all securities held by the number of shares. The NAV is more or less subjective in cases where the associated assets are more or less liquid. An extreme example of this may be the calculation of a NAV that has to take into account the theoretical market value of art.
Of course, the mechanics of participation is typically more involved and complex than this.
Who plays, who paysInvestors in Hedge Funds are individuals and institutions (such as foundations, endowments, family offices, pension funds, insurance companies, private banks and funds of funds).
One of the key criteria in the Hedge Fund business is that only people and institutions with money can invest in them. On face value this prerequisite seems a tad bizarre, but there are some very valid reasons for it.
In order to be able to invest in Hedge Funds an investor will need to meet certain legal requirements. They have to be either a credited investor or a qualified purchaser. The qualification is based on net worth and individual income. The qualified purchaser has a higher net worth than a credited investor.
In my opinion the practical rules of Hedge Funds are clear, albeit wrapped up in more indirect language. The biggest rule is: ‘do not put any money into a Hedge Fund that you are not prepared to lose’. The second big rule is: ‘only subscribe to a Hedge Fund with money that you can lose and without the risk of significantly and adversely affecting your lifestyle.’
Of course, this didn’t stop people from jumping on the Hedge Fund bandwagon with little or no clue about what they were getting themselves into.
That’s all folksHedge Funds are subject to a dire circle of misleading, banal and frequently reactionary published and public opinion. Which is unfortunate, because it ignores that almost all of the Hedge Funds reflect a culture and style of their managers and administrators, and that in the business there is a a lot of plurality and diversity.
Some Hedge Funds have been the epitome of sharp investment practice, hubris and good old fashioned albeit legal duplicity.
There are macho funds, and non-macho funds, high risk funds and risk-averse funds, highly leveraged funds and funds that use little or no leverage.
Some funds are discrete, some are ugly, some are charming, and some are boisterous and incredibly aggressive. Some are socially responsible, and others might ask “is social responsibility part of the NAV calculation?”
Some funds delight in planning raids on markets, mounting shark attacks on political systems and find ‘justifiable’ enchantment in destabilizing economies and currencies. Other Hedge Funds – in my opinion the vast majority – would never dream of doing such things.
Furthermore, some funds actively encourage responsible investment in developing countries and in other ethical investment strategies. Moreover, there are plenty of examples of funds that sit somewhere between the extremes, so there is no one size fits all when it comes to characterizing funds.
But whatever the style of the Hedge Fund, at the heart of each individual Hedge Fund culture is the culture of the team leaders and team players.
So, are Hedge Funds intrinsically evil?
No, I don’t think so. But that sort of headline grabs a lot of people’s attention, and frequently for all the wrong reasons.
Thank you so much for reading.
[i] Loomis, Caroll J. The Jones that nobody keeps up with. Fortune, April 1966
File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro