Category Archives: Information Supply Frameowrk

Data Supply Framework 3.0 – ETL Patterns

26 Thursday Jan 2017

Posted by in 4th generation Data Warehousing, All Data, Big Data, Big Data 7s, Big Data Analytics, business strategy, dark data, data architecture, Data governance, Data Lake, Data Supply Framework, Extract, Good Strategy, goodstart, governance, Information Management, Information Supply Frameowrk, Information Supply Framework, IT strategy, Strategy, The Big Data Contrarians

≈ Leave a comment

Martyn Richard Jones

Mountain View, 22nd January 2015

image3This article is the first in a series of articles that discuss aspects of the use of architectural patterns in the Cambriano Information Supply Framework 3.0

The term architectural pattern may sound grand, misleading or daunting, but it’s really quite a simple concept. It’s like writing a function in a programming language to log in to a database, check that the connection is alive and working and report back the success of the connection request. If that function can be reused either in the same application development, in the same IT shop or in IT in general (e.g. Java code to connect and test the connection to SQL Server) then it’s well on its way to becoming an architectural pattern. Of course, there are much more sophisticated architectural patterns. But generally a pattern is a simplified and generic template for address a generally occurring problem. But as with much in architecture, less usually turns out to be more.

Continue reading →

Big Data is Bullshit – 2017

19 Thursday Jan 2017

Posted by in 4th generation Data Warehousing, All Data, Big Data, Big Data 7s, Big Data Analytics, dark data, data architecture, Data governance, Data Lake, data management, data science, Data Supply Framework, Data Warehouse, Data Warehousing, disinformation, governance, information, Information Management, Information Supply Frameowrk, Information Supply Framework, pig data, The Amazing Big Data Challenge, The Big Data Contrarians

≈ Leave a comment

Marty Richard Jones

Mountain View, 19th January 2017

“I’ve been accused of vulgarity. I say that’s bullshit.” – Mel Brooks

If you enjoy this piece or find it useful then please consider joining The Big Data Contrarians. Continue reading →

A data superhero is something to be

02 Wednesday Mar 2016

Posted by in 4th generation Data Warehousing, Consider this, Data Supply Framework, Data Warehouse, Data Warehousing, DW 3.0, EDW, Excellence, Good Strat, Good Strategy, Information Management, Information Supply Frameowrk, Knowledge, leadership, Methodology, Professional Networking, The Big Data Contrarians

≈ Leave a comment

SuperHeroA data warehousing superhero is something to be

Not all that glitters is Big Data, and Big Data has a long way to go before it can deliver anything like the same satisfying results, tangible benefits and organisational agility that a properly implemented Inmon Enterprise Data Warehouse can provide.

Therefore, I have a question for you.

Continue reading →

In the beginning was the Big Data Plan

23 Tuesday Feb 2016

Posted by in 4th generation Data Warehousing, Big Data, Big Data 7s, Big Data Analytics, Data governance, Data Lake, data science, Data Warehouse, Dogma, DW 3.0, Information Management, Information Supply Frameowrk, Infotrends, Inmon, sentiment analysis

≈ Leave a comment

Lucas_Cranach_d._Ä._035

In the beginning was the Big Iron, the Big Data, and the Big Data Plan.

And then came the Big Data Assumptions.

And the Big Data Assumptions were without form.

And the Big Data Plan was without substance.

And the Big Iron was without movement.

And the Big Data was without velocity, variety and volume.

And darkness was upon the face of the data workers.

And they spoke amongst themselves, saying: “Big Data, is a crock of shit, and it stinketh mucho”.

And the data workers went unto their Data Supervisors and said: “This here Big Data is a pile of putrid crappy keech”, for they were from Govan, and continued, “and none may abide the odour thereof”.

And the Data Supervisors went unto their Information Managers, saying: “Big Data is a container of excrement, and it is very strong, such that none may abide by it.”

And the Information Managers went unto their Business Directors, saying: “This here Big Data doodoo is a vessel of fertilizer, and none may abide its strength.”

And the Business Directors spoke amongst themselves, saying to one another: “Big Data contains that which aids plant growth, and it is very powerful.”

And the Vice Presidents went unto the President, saying unto him: “This new Big Data will actively promote the growth and vigour of the company, with powerful effects.”

And the President looked upon the Big Iron, the Big Data and the Big Data Plan, and saw that they were good.

Many thanks for reading

Join The Big Data Contrarians

The Big Data Contrarians

Stories from the Data Warehousing front-line

21 Sunday Feb 2016

Posted by in 4th generation Data Warehousing, Big Data, Big Data 7s, Data Warehouse, Data Warehousing, Good Strat, Good Strategy, goodstart, goodstrat, Information Management, Information Supply Frameowrk, Information Supply Framework, Inmon, Martyn Jones, Martyn Richard Jones, Strategy, The Big Data Contrarians

≈ Leave a comment

NB THIS IS FICTION

All characters appearing in this work are fictitious. Any resemblance to real persons, living or dead, is purely coincidental.

Data warehousing, what is she like?

Although the answers are probably obvious, and to be honest, compared to the Big Data hype-circus this is a walk in the park, I have often wondered why Data Warehousing attracts such a surfeit of lazy, socially inept and shallow-thinking chancers.

I could go on about this at length, about how I convened a meeting recently (held on the outskirts of Bornheim, a small town in Germany ) to discuss how to move rapidly forward with a new strategic data-warehousing project, and how, whilst putting aside the crass impertinence and barely-disguised arrogance of my guests, I was still amazed by the unabashed and brazen snow-job that I was subjected to.

I imagined that this was a deliberate tactic used in the craven hope that I would be overcome by the depth and breadth of their ‘inside knowledge’, and would consent to having my workshop hijacked and reframed

But, I wasn’t having any of that. I know bluff and bluster when I see it, and as a reformed bullshitter I will not willingly accept bullshit from anyone else.

So, as their bullshit came in fast and furious I started making notes, and thinking of the most adequate response that a Project Manager could make in the circumstances, but I soon tired of note taking and was rapidly becoming irritated by a total lack of empathy and an utter lack of engagement.

Irritated as I was, I still tried putting things back on the rails. Therefore, I continued to be as engaging and constructive as one should, whilst internally suppressing the urge to ask ‘what the feck is going on here?’ So, I talked about lifting, shifting and dropping a legacy data warehouse and marts from one box to another – thinking that this would be the minimum that Data Warehouse experts could engage with – and the need to get estimates of the effort required to do so (you know, things like roles and number of days, Big ballpark estimate stuff). The ‘Data Warehouse Architect’ and I use that term loosely, went off on a tangent. Vague, fuzzy and disjointed. The architect threw in some nonsensical vagaries about the need for Master Data Management to be an integral component of any future data warehouse. I half-managed to avoid the incredulous Jeremy Paxman look of ‘what on earth are you talking about?’ just as the gathered augmented the assault on MDM with a call to Information Lifecycle arms. Therefore, when things were becoming even weirder, the weird turned pro, with the train kicked off the tracks, rolled down the hill, and then set on fire, by so-called professionals, passing themselves off as supine yobs, and reciting, in close harmony, “Proof of Concept! Proof of Concept! What about the f****** proof of Concept! Uh? Uh? Uh?”

Well, well, well… what a way to run a dance hall!

We were opening up all technological fronts, apart from the ones that would actually be relevant. I felt like a PG Tips chimp getting bananas and cups of tea thrown in their general direction. I was Martyn, the Project Chimp, plaintively calling out “‘ere mate, do you know this is not the way to do Data Warehousing”, and half-expecting a response along the lines of “you plan it, Son, we’ll muddle along “. I didn’t get a response, all I got was what looked like the human equivalent (if there is such a thing) of a page-fault, a glazing-over of the eyes and a rapid reboot into full-on bullshit mode.

I could go on and on about this all day, but I would rather not. Just the day in the life of a PM tasked with getting sense, sensibility, and work out of profane variations on the theme of Blackadder’s stupid Prince George. “I don’t need Inmon or Kimball, I know data! And… I have been to Ikea!” Sorry, that was just an example of how utterly obtuse things can get on the front-line of Data Warehousing.

So, to close, I would like to pose a question, one that goes beyond Data Warehousing and Big Data. Do people have the same or similar issues in other parts of IT or indeed in other businesses and technical related activities?

Bamberg

22nd September 2012

 

All Data: It’s about statistics

30 Friday Jan 2015

Posted by in All Data, Consider this, DW 3.0, Good Strat, Good Strategy, Information Supply Frameowrk, Martyn Jones, Martyn Richard Jones, statistics

≈ Leave a comment

Tags

, , , , , , ,

LinkedInHeader1

A big computer, a complex algorithm and a long time does not equal science.

Robert Gentleman

To begin at the beginning

Fueled by the new fashions on the block, principally Big Data, the Internet of Things, and to a lesser extent Cloud computing, there’s a debate quietly taking please over what statistics is and is not, and where it fits in the whole new brave world of data architecture and management. For this piece I would like to put aspects of this discussion into context, by asking what ‘Core Statistics’ means in the context of the DW 3.0 Information Supply Framework.

Core Statistics on the DW 3.0 Landscape

The following diagram illustrates the overall DW 3.0 framework:

There are three main concepts in this diagram: Data Sources; Core Data Warehousing; and, Core Statistics.

Data Sources: All current sources, varieties, velocities and volumes of data available.

Core Data Warehousing: All required content, including data, information and outcomes derived from statistical analysis.

Core Statistics: This is the body of statistical competence, and the data used by that competence. A key data component of Core Statistics is the Analytics Data Store, which is designed to support the requirements of statisticians.

The focus of this piece is on Core Statistics. It briefly looks at the aspect of demand driven data provisioning for statistical analysis and what ‘statistics’ means in the context of the DW 3.0 framework.

Demand Driven Data Provisioning

The DW 3.0 Information Supply Framework isn’t primarily about statistics it’s about data supply. However, the provision of adequate, appropriate and timely demand-driven data to statisticians for statistical analysis is very much an integral part of the DW 3.0 philosophy, framework and architecture.

Within DW 3.0 there are a number of key activities and artifacts that support the effective functioning of all associated processes. Here are some examples:

All Data Investigation: An activity centre that carries out research into potential new sources of data and analyses the effectiveness of existing sources of data and its usage. It is also responsible for identifying markets for data owned by the organization.

All Data Brokerage: An activity that focuses on all aspects of matching data demand to data supply, including negotiating supply, service levels and quality agreements with data suppliers and data users. It also deals with contractual and technical arrangements to supply data to corporate subsidiaries and external data customers.

All Data Quality: Much of the requirements for clean and useable data, regardless of data volumes, variety and velocity, have been addressed by methods, tools and techniques developed over the last four decades. Data migration, data conversion, data integration, and data warehousing have all brought about advances in the field of data quality. The All Data Quality function focuses on providing quality in all aspects of information supply, including data quality, data suitability, quality and appropriateness of data structures, and data use.

All Data Catalogue: The creation and maintenance of a catalogue of internal and external sources of data, its provenance, quality, format, etc. It is compiled based on explicit demand and implicit anticipation of demand, and is the result of an active scanning of the ‘data markets’, ‘potential new sources’ of data and existing and emerging data suppliers.

All Data Inventory: This is a subset of the All Data Catalogue. It identifies, describes and quantifies the data in terms of a full range of metadata elements, including provenance, quality, and transformation rules. It encompasses business, management and technical metadata; usage data; and, qualitative and quantitative contribution data.

Of course there are many more activities and artifacts involved in the overall DW 3.0 framework.

Yes, but is it all statistics?

Statistics, it is said, is the study of the collection, organization, analysis, interpretation and presentation of data. It deals with all aspects of data, including the planning of data collection in terms of the design of surveys and experiments; learning from data, and of measuring, controlling, and communicating uncertainty; and it provides the navigation essential for controlling the course of scientific and societal advances[i]. It is also about applying statistical thinking and methods to a wide variety of scientific, social, and business endeavors in such areas as astronomy, biology, education, economics, engineering, genetics, marketing, medicine, psychology, public health, sports, among many.

Core Statistics supports micro and macro oriented statistical data, and metadata for syntactical projection (representation-orientation); semantic projection (content-orientation); and, pragmatic projection (purpose-orientation).

The Core Statistics approach provides a full range of data artifacts, logistics and controls to meet an ever growing and varied demand for data to support the statistician, including the areas of data mining and predictive analytics. Moreover, and this is going to be tough for some people to accept, the focus of Core Statistics is on professional statistical analysis of all relevant data of all varieties, volumes and velocities, and not, for example, on the fanciful and unsubstantiated data requirements of amateur ‘analysts’ and ‘scientists’ dedicated to finding causation free correlations and interesting shapes in clouds.

That’s all folks

This has been a brief look at the role of DW 3.0 in supplying data to statisticians.

One key aspect of the Core Statistics element of the DW 3.0 framework is that it renders irrelevant the hyperbolic claims that statisticians are not equipped to deal with data variety, volumes and velocity.

Even with the advent of Big Data alchemy is still alchemy, and data analysis is still about statistics.

If you have any questions about this aspect of the framework then please feel free to contact me, or to leave a comment below.

Many thanks for reading.

Catalogue under: #bigdata #technology

[i] Davidian, M. and Louis, T. A., 10.1126/science.1218685

File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro

Consider this: Social Media Big Data killed Ad-land

21 Wednesday Jan 2015

Posted by in Big Data, Consider this, Information Supply Frameowrk

≈ Leave a comment

Tags

, , , , ,

Hold this thought: Big Data is the future of online business and interactive advertising is its profit.

Much is being made of Big Data and its role in social media and online interactive advertising. The advertising industry itself has a “big crush” on Big Data, and it fuels the elevated revenues, profits and share prices of a number of online companies.

In 2011, a predominant social media company in preparation for their IPO announced that 85% of their $3.5B revenue came from advertising.

In the same year another high profile internet company made 96% of their revenue of $37B from advertising.

There are quite a few examples of internet companies that make the overwhelming bulk of their revenue from online advertising.

Much of the attraction of these companies can be found in the tremendous volumes of ‘social media’ data that they gather, mine and use to in the target the interactive advertising of brands, products and services.

Now consider this: advertising legend Dave Trott reckons that 89% of advertising is ignored, and who would argue with Dave Trott.

What does that mean?

Out of every $1M spent on advertising, $890K is money down the drain.

Let’s raise the stakes a little. If $37B is spent on advertising, and our assumptions about the effectiveness of advertising is true, then $32B went west, and no one even noticed.

So, what does that mean?

You may recall the frenetic DOT COM gold rush of the late nineties. That went well didn’t it?

The sub-prime revolution came later. But how did that turn out?

If 89% of advertising is ignored, and people online don’t want to engage with brands, interactive advertising or have meaningful discussions about either, then the ‘ignore’ rate for online advertising might even surpass that bar-setting 89%.

McKinsey (in 2014) stated that “E-mail remains a significantly more effective way to acquire customers than social media—nearly 40 times that of Facebook and Twitter combined”. What should that tell us? What sort of signal is that sending to the market?

Interactive advertising and online viewing were supposed to signal the end of traditional channel advertising and TV as we knew it. But, according to US advertising guru and contrarian Bob Hoffman, in 2013, 97% of video viewing in the USA was done on TVs, 3% was done on-line. In a presentation he made at Advertising Week Europe he commented on interactive internet advertising, the sort of thing we get in social media sites, he concluded that ‘people have no interest whatsoever in interacting with advertising, in fact online banner ads have a click-through rate lower than one in a thousand. This is not interactivity, this is absence of interactivity.’

For good measure, it has been claimed that more than 35% of the online traffic channeled through ads is actually fake. According to digital analytics provider comScore, 36% of online traffic is suspicious or non-human – source: The Guardian 16/May/2014. Michael Tiffany, CEO of fraud detection firm White Ops, has stated that some fraud today is so egregious that the only way it can be succeeding is because nobody cares – source: Advertising Age 22/July/2013.

Of course advertising expenditure won’t stop, it won’t mutate and it won’t die. It will just relocate; probably back to the more traditional channels and forms of advertising.

But what is going to happen when online advertisers get wise to the probable ineffectiveness of their advertising expenditure – an ineffectiveness that is magnificently amplified in online-land?

What will happen to the social media and other internet companies that make their money from advertising, when online advertisers rethink their advertising strategies and start to go elsewhere? What will happen to the organizations that are built on the idea that digital advertising is the future?

How about this for starters:

  • Markets will lose confidence
  • Share prices will plummet
  • People will lose money, big time
  • Big companies will radically reduce or curtail their expenditure on online advertising
  • Share prices will plummet even further
  • Smaller companies will follow the big companies and reduce or curtail their expenditure on online advertising

So what will these internet companies be able to do in order to compensate for the loss of what is essentially a massive slice of their core business? They have no bricks or mortar; they have no obvious intellectual capital worth shaking a stick at; and, they will be left without any clear market differentiators.

Will they actually start charging people to use their services?

As I wrote at the outset, much is being made of Big Data and its role in social media and online interactive advertising. But is its role being overplayed? Are we entirely candid about the effectiveness of online interactive advertising? And, will this “irrational exuberance” simply end in tears, as it has done so many times before?

How can the impending collapse be averted? I will leave my thoughts on that conundrum for another time.

Many thanks for reading.

File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro