02 Wednesday Mar 2016
Posted in 4th generation Data Warehousing, Consider this, Data Supply Framework, Data Warehouse, Data Warehousing, DW 3.0, EDW, Excellence, Good Strat, Good Strategy, Information Management, Information Supply Frameowrk, Knowledge, leadership, Methodology, Professional Networking, The Big Data Contrarians
A data warehousing superhero is something to beNot all that glitters is Big Data, and Big Data has a long way to go before it can deliver anything like the same satisfying results, tangible benefits and organisational agility that a properly implemented Inmon Enterprise Data Warehouse can provide.
Therefore, I have a question for you.
23 Tuesday Feb 2016
In the beginning was the Big Iron, the Big Data, and the Big Data Plan.
And then came the Big Data Assumptions.
And the Big Data Assumptions were without form.
And the Big Data Plan was without substance.
And the Big Iron was without movement.
And the Big Data was without velocity, variety and volume.
And darkness was upon the face of the data workers.
And they spoke amongst themselves, saying: “Big Data, is a crock of shit, and it stinketh mucho”.
And the data workers went unto their Data Supervisors and said: “This here Big Data is a pile of putrid crappy keech”, for they were from Govan, and continued, “and none may abide the odour thereof”.
And the Data Supervisors went unto their Information Managers, saying: “Big Data is a container of excrement, and it is very strong, such that none may abide by it.”
And the Information Managers went unto their Business Directors, saying: “This here Big Data doodoo is a vessel of fertilizer, and none may abide its strength.”
And the Business Directors spoke amongst themselves, saying to one another: “Big Data contains that which aids plant growth, and it is very powerful.”
And the Vice Presidents went unto the President, saying unto him: “This new Big Data will actively promote the growth and vigour of the company, with powerful effects.”
And the President looked upon the Big Iron, the Big Data and the Big Data Plan, and saw that they were good.
Many thanks for reading
Join The Big Data Contrarians
30 Friday Jan 2015
Tags
All Data, Big Data, business intelligence, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Jones, statistics
A big computer, a complex algorithm and a long time does not equal science.
Robert Gentleman
Fueled by the new fashions on the block, principally Big Data, the Internet of Things, and to a lesser extent Cloud computing, there’s a debate quietly taking please over what statistics is and is not, and where it fits in the whole new brave world of data architecture and management. For this piece I would like to put aspects of this discussion into context, by asking what ‘Core Statistics’ means in the context of the DW 3.0 Information Supply Framework.
The following diagram illustrates the overall DW 3.0 framework:
There are three main concepts in this diagram: Data Sources; Core Data Warehousing; and, Core Statistics.
Data Sources: All current sources, varieties, velocities and volumes of data available.
Core Data Warehousing: All required content, including data, information and outcomes derived from statistical analysis.
Core Statistics: This is the body of statistical competence, and the data used by that competence. A key data component of Core Statistics is the Analytics Data Store, which is designed to support the requirements of statisticians.
The focus of this piece is on Core Statistics. It briefly looks at the aspect of demand driven data provisioning for statistical analysis and what ‘statistics’ means in the context of the DW 3.0 framework.
The DW 3.0 Information Supply Framework isn’t primarily about statistics it’s about data supply. However, the provision of adequate, appropriate and timely demand-driven data to statisticians for statistical analysis is very much an integral part of the DW 3.0 philosophy, framework and architecture.
Within DW 3.0 there are a number of key activities and artifacts that support the effective functioning of all associated processes. Here are some examples:
All Data Investigation: An activity centre that carries out research into potential new sources of data and analyses the effectiveness of existing sources of data and its usage. It is also responsible for identifying markets for data owned by the organization.
All Data Brokerage: An activity that focuses on all aspects of matching data demand to data supply, including negotiating supply, service levels and quality agreements with data suppliers and data users. It also deals with contractual and technical arrangements to supply data to corporate subsidiaries and external data customers.
All Data Quality: Much of the requirements for clean and useable data, regardless of data volumes, variety and velocity, have been addressed by methods, tools and techniques developed over the last four decades. Data migration, data conversion, data integration, and data warehousing have all brought about advances in the field of data quality. The All Data Quality function focuses on providing quality in all aspects of information supply, including data quality, data suitability, quality and appropriateness of data structures, and data use.
All Data Catalogue: The creation and maintenance of a catalogue of internal and external sources of data, its provenance, quality, format, etc. It is compiled based on explicit demand and implicit anticipation of demand, and is the result of an active scanning of the ‘data markets’, ‘potential new sources’ of data and existing and emerging data suppliers.
All Data Inventory: This is a subset of the All Data Catalogue. It identifies, describes and quantifies the data in terms of a full range of metadata elements, including provenance, quality, and transformation rules. It encompasses business, management and technical metadata; usage data; and, qualitative and quantitative contribution data.
Of course there are many more activities and artifacts involved in the overall DW 3.0 framework.
Statistics, it is said, is the study of the collection, organization, analysis, interpretation and presentation of data. It deals with all aspects of data, including the planning of data collection in terms of the design of surveys and experiments; learning from data, and of measuring, controlling, and communicating uncertainty; and it provides the navigation essential for controlling the course of scientific and societal advances[i]. It is also about applying statistical thinking and methods to a wide variety of scientific, social, and business endeavors in such areas as astronomy, biology, education, economics, engineering, genetics, marketing, medicine, psychology, public health, sports, among many.
Core Statistics supports micro and macro oriented statistical data, and metadata for syntactical projection (representation-orientation); semantic projection (content-orientation); and, pragmatic projection (purpose-orientation).
The Core Statistics approach provides a full range of data artifacts, logistics and controls to meet an ever growing and varied demand for data to support the statistician, including the areas of data mining and predictive analytics. Moreover, and this is going to be tough for some people to accept, the focus of Core Statistics is on professional statistical analysis of all relevant data of all varieties, volumes and velocities, and not, for example, on the fanciful and unsubstantiated data requirements of amateur ‘analysts’ and ‘scientists’ dedicated to finding causation free correlations and interesting shapes in clouds.
This has been a brief look at the role of DW 3.0 in supplying data to statisticians.
One key aspect of the Core Statistics element of the DW 3.0 framework is that it renders irrelevant the hyperbolic claims that statisticians are not equipped to deal with data variety, volumes and velocity.
Even with the advent of Big Data alchemy is still alchemy, and data analysis is still about statistics.
If you have any questions about this aspect of the framework then please feel free to contact me, or to leave a comment below.
Many thanks for reading.
Catalogue under: #bigdata #technology
[i] Davidian, M. and Louis, T. A., 10.1126/science.1218685
File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro
