Monthly Archives: January 2015

Consider this: Hedge Funds are Evil

30 Friday Jan 2015

Posted by in Banking, Consider this, Good Strat, Good Strategy, Martyn Jones, Martyn Richard Jones, Strategy

≈ Leave a comment

Tags

, , , , , , , ,

Hedge Funds are evil, right? Go on, you know you want to say yes. Almost everyone has an opinion about them, but very few people can actually tell you what they are.

Indeed, there’s am awful lot of nonsense written about Hedge Funds, and this piece might just end up being a worthy addition to that body of baloney. But, the intention is somewhat different.

The objective behind this piece is to provide a quick look at where the modern hedge fund started; what they are; how they work; the mechanics of participation; and who traditionally has put their money into them.

Of course, this piece is a necessary simplification of what is a fascinating aspect of the alternative investment universe.

To begin at the beginning

In 1966 Carol J. Loomis[i] blew the lid on one of the best kept investment secrets of the 20th century.

In an article penned for Fortune titled “The Jones that nobody keeps up with”, Loomis revealed that over a five year period a fund run by Alfred Winslow Jones had consistently outperformed the Fidelity Fund, the most successful mutual fund of that time, by a remarkable 44%.

Not only that, but between 1956 and 1966 the Jones fund had outperformed the Dreyfus Fund, the best performing mutual fund of that decade, by a massive 87%.

Jones was born in Melbourne, Australia, but from the age of four he lived in the USA. He graduated from Harvard in 1923, and before becoming involved in the finance industry he toured the world working on steamships. He was to serve as a diplomat in Germany, and also worked as a journalist covering the Spanish civil war.

In 1941, with conflict raging in Europe, Jones returned to the USA. He then studied for, and obtained a doctorate in sociology at Columbia University, and became a reporter for Fortune.

His thesis, Life, Liberty and Property, is a reference text in sociology.

In 1949 Jones formed a company, A. W. Jones & Co., arguably the first modern Hedge Fund.

Robert A. Jaeger characterised the fund as “an opportunistic equity hedge fund”, that relied heavily on discerning stock picking abilities, combined with bets on long positions (rising prices) and short positions (falling prices).

In 1952 the fund was converted into a limited partnership, and during the 50s other such partnerships were set up, including the sage of Omaha’s Buffet Partners, and WJS Partners, founded by Walter Schloss.


What they are; how they work

Hedge funds are loosely regulated, exclusive and limited-membership investment clubs, usually run as partnerships or as corporations. They focus on absolute returns on investments, for themselves and their members, regardless of market conditions.

Direct participation in Hedge Funds is theoretically limited to between 100 or 500 investors, depending on the class of investor. Moreover, because of the unregulated status of most hedge funds, they are not allowed to actively market their products, so they have to use more exclusive means to attract investors, typically word of mouth.

Hedge Funds typically invest in traditional securities, such as stocks, bonds and commodities, but they can also invest in real estate, art, wine or any number of other non-traditional areas of investment. In fact, they are free to use virtually any pick and mix of strategies from the entire range of investment possibilities.

That said, a lot of Hedge Funds will opt for a specific investment strategy and will stick with that strategy for the life-time of the fund.

So what’s in it for the Hedge Fund managers and administrators?

Hedge Fund Managers typically charge a management fee of between 1and 3 percent of the value of the assets under management, regardless of performance. They may also – almost always in the past – charge a performance fee, which can start at around 20 percent of any fund gains above a certain minimum performance hurdle or target value. Managers also generally ‘eat their own dog food’, in that they will also invest in their own Hedge Fund.

Investors in Hedge Funds are informed of the value of their investments via a statement that shows the calculated value of shares in the fund, the Net Asset Value (the NAV). This can be calculated monthly, quarterly or even yearly, depending on the fund. In addition funds are free to choose if they wish to publicly disclose performance figures or not.

Some hedge funds may require additional fees and commissions, and may impose lock up periods, and strict and narrow redemption periods. They may also reject some applications for subscriptions without giving any reasons, and they may also forcibly redeem shares held, and without having to justify their actions.

In addition, some hedge funds use equalization methods – and there are a number of variants – to equitably distribute hedge fund fees amongst its partners, yet other hedge funds do not use equalization methods at all.

The mechanics of participation

So, briefly, how do you get to invest in a Hedge Fund (subscribe), how do you liquidate that participation (redeem), and what happens between ‘subscription’ and ‘redemption’.

Offer: A Hedge Fund details what’s involved in a particular offer in an ‘Offering Memorandum’, also known as a ‘private placement memorandum’. This is typically an ‘enriched’ business plan tied to a specific issuance of shares in a fund. It basically sets out the stall.

Subscription: In order to subscribe to a fund the potential participant in the fund signs up to a subscription agreement, and the conditions laid out in that agreement. Conditions may cover aspects such as minimum subscription amounts; minimum share increments, rules governing the liquidation of participation in the fund; management and performance fees; and, so on and so forth.

Redemption: This is the liquidation of shares in a fund. Typical redemption points can occur from anything from 15 days to up to 180 days, and sometimes more than this, depending on the fund and the rules related to lock-ups, redemption. In addition, redemption options may be linked to other financial charges, penalties and constraints.

So what happens between subscription and redemption?

Custody: A hedge fund subscriber may wish to use the services of a large and reputable financial service provider to act as custodian of their hedge fund shares. In addition to holding securities for safekeeping, most custodians also offer other services such as account administration, transaction settlements, collection of dividends and interest payments, tax support and foreign exchange. (Source: Investopedia).

Dividends: It is very uncommon for hedge funds to pay dividends, as any accruable earnings are realized upon redemption of the shares. However, some fundsdo incentivize the maintenance of subscriptions through the payment of dividends.

Calculating NAV: Periodically – or even on a real-time basis – a hedge fund will recalculate the Net Asset Value of the fund shares. This is done by dividing total value of all securities held by the number of shares. The NAV is more or less subjective in cases where the associated assets are more or less liquid. An extreme example of this may be the calculation of a NAV that has to take into account the theoretical market value of art.

Of course, the mechanics of participation is typically more involved and complex than this.

Who plays, who pays

Investors in Hedge Funds are individuals and institutions (such as foundations, endowments, family offices, pension funds, insurance companies, private banks and funds of funds).

One of the key criteria in the Hedge Fund business is that only people and institutions with money can invest in them. On face value this prerequisite seems a tad bizarre, but there are some very valid reasons for it.

In order to be able to invest in Hedge Funds an investor will need to meet certain legal requirements. They have to be either a credited investor or a qualified purchaser. The qualification is based on net worth and individual income. The qualified purchaser has a higher net worth than a credited investor.

In my opinion the practical rules of Hedge Funds are clear, albeit wrapped up in more indirect language. The biggest rule is: ‘do not put any money into a Hedge Fund that you are not prepared to lose’. The second big rule is: ‘only subscribe to a Hedge Fund with money that you can lose and without the risk of significantly and adversely affecting your lifestyle.’

Of course, this didn’t stop people from jumping on the Hedge Fund bandwagon with little or no clue about what they were getting themselves into.

That’s all folks

Hedge Funds are subject to a dire circle of misleading, banal and frequently reactionary published and public opinion. Which is unfortunate, because it ignores that almost all of the Hedge Funds reflect a culture and style of their managers and administrators, and that in the business there is a a lot of plurality and diversity.

Some Hedge Funds have been the epitome of sharp investment practice, hubris and good old fashioned albeit legal duplicity.

There are macho funds, and non-macho funds, high risk funds and risk-averse funds, highly leveraged funds and funds that use little or no leverage.

Some funds are discrete, some are ugly, some are charming, and some are boisterous and incredibly aggressive. Some are socially responsible, and others might ask “is social responsibility part of the NAV calculation?”

Some funds delight in planning raids on markets, mounting shark attacks on political systems and find ‘justifiable’ enchantment in destabilizing economies and currencies. Other Hedge Funds – in my opinion the vast majority – would never dream of doing such things.

Furthermore, some funds actively encourage responsible investment in developing countries and in other ethical investment strategies. Moreover, there are plenty of examples of funds that sit somewhere between the extremes, so there is no one size fits all when it comes to characterizing funds.

But whatever the style of the Hedge Fund, at the heart of each individual Hedge Fund culture is the culture of the team leaders and team players.

So, are Hedge Funds intrinsically evil?

No, I don’t think so. But that sort of headline grabs a lot of people’s attention, and frequently for all the wrong reasons.

Thank you so much for reading.

[i] Loomis, Caroll J. The Jones that nobody keeps up with. Fortune, April 1966

File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro

All Data: It’s about statistics

30 Friday Jan 2015

Posted by in All Data, Consider this, DW 3.0, Good Strat, Good Strategy, Information Supply Frameowrk, Martyn Jones, Martyn Richard Jones, statistics

≈ Leave a comment

Tags

, , , , , , ,

LinkedInHeader1

A big computer, a complex algorithm and a long time does not equal science.

Robert Gentleman

To begin at the beginning

Fueled by the new fashions on the block, principally Big Data, the Internet of Things, and to a lesser extent Cloud computing, there’s a debate quietly taking please over what statistics is and is not, and where it fits in the whole new brave world of data architecture and management. For this piece I would like to put aspects of this discussion into context, by asking what ‘Core Statistics’ means in the context of the DW 3.0 Information Supply Framework.

Core Statistics on the DW 3.0 Landscape

The following diagram illustrates the overall DW 3.0 framework:

There are three main concepts in this diagram: Data Sources; Core Data Warehousing; and, Core Statistics.

Data Sources: All current sources, varieties, velocities and volumes of data available.

Core Data Warehousing: All required content, including data, information and outcomes derived from statistical analysis.

Core Statistics: This is the body of statistical competence, and the data used by that competence. A key data component of Core Statistics is the Analytics Data Store, which is designed to support the requirements of statisticians.

The focus of this piece is on Core Statistics. It briefly looks at the aspect of demand driven data provisioning for statistical analysis and what ‘statistics’ means in the context of the DW 3.0 framework.

Demand Driven Data Provisioning

The DW 3.0 Information Supply Framework isn’t primarily about statistics it’s about data supply. However, the provision of adequate, appropriate and timely demand-driven data to statisticians for statistical analysis is very much an integral part of the DW 3.0 philosophy, framework and architecture.

Within DW 3.0 there are a number of key activities and artifacts that support the effective functioning of all associated processes. Here are some examples:

All Data Investigation: An activity centre that carries out research into potential new sources of data and analyses the effectiveness of existing sources of data and its usage. It is also responsible for identifying markets for data owned by the organization.

All Data Brokerage: An activity that focuses on all aspects of matching data demand to data supply, including negotiating supply, service levels and quality agreements with data suppliers and data users. It also deals with contractual and technical arrangements to supply data to corporate subsidiaries and external data customers.

All Data Quality: Much of the requirements for clean and useable data, regardless of data volumes, variety and velocity, have been addressed by methods, tools and techniques developed over the last four decades. Data migration, data conversion, data integration, and data warehousing have all brought about advances in the field of data quality. The All Data Quality function focuses on providing quality in all aspects of information supply, including data quality, data suitability, quality and appropriateness of data structures, and data use.

All Data Catalogue: The creation and maintenance of a catalogue of internal and external sources of data, its provenance, quality, format, etc. It is compiled based on explicit demand and implicit anticipation of demand, and is the result of an active scanning of the ‘data markets’, ‘potential new sources’ of data and existing and emerging data suppliers.

All Data Inventory: This is a subset of the All Data Catalogue. It identifies, describes and quantifies the data in terms of a full range of metadata elements, including provenance, quality, and transformation rules. It encompasses business, management and technical metadata; usage data; and, qualitative and quantitative contribution data.

Of course there are many more activities and artifacts involved in the overall DW 3.0 framework.

Yes, but is it all statistics?

Statistics, it is said, is the study of the collection, organization, analysis, interpretation and presentation of data. It deals with all aspects of data, including the planning of data collection in terms of the design of surveys and experiments; learning from data, and of measuring, controlling, and communicating uncertainty; and it provides the navigation essential for controlling the course of scientific and societal advances[i]. It is also about applying statistical thinking and methods to a wide variety of scientific, social, and business endeavors in such areas as astronomy, biology, education, economics, engineering, genetics, marketing, medicine, psychology, public health, sports, among many.

Core Statistics supports micro and macro oriented statistical data, and metadata for syntactical projection (representation-orientation); semantic projection (content-orientation); and, pragmatic projection (purpose-orientation).

The Core Statistics approach provides a full range of data artifacts, logistics and controls to meet an ever growing and varied demand for data to support the statistician, including the areas of data mining and predictive analytics. Moreover, and this is going to be tough for some people to accept, the focus of Core Statistics is on professional statistical analysis of all relevant data of all varieties, volumes and velocities, and not, for example, on the fanciful and unsubstantiated data requirements of amateur ‘analysts’ and ‘scientists’ dedicated to finding causation free correlations and interesting shapes in clouds.

That’s all folks

This has been a brief look at the role of DW 3.0 in supplying data to statisticians.

One key aspect of the Core Statistics element of the DW 3.0 framework is that it renders irrelevant the hyperbolic claims that statisticians are not equipped to deal with data variety, volumes and velocity.

Even with the advent of Big Data alchemy is still alchemy, and data analysis is still about statistics.

If you have any questions about this aspect of the framework then please feel free to contact me, or to leave a comment below.

Many thanks for reading.

Catalogue under: #bigdata #technology

[i] Davidian, M. and Louis, T. A., 10.1126/science.1218685

File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro

Consider this: Does all data have value?

30 Friday Jan 2015

Posted by in Assets, Consider this, Data governance, data science, Good Strat, Martyn Jones

≈ 1 Comment

Tags

, , , , , , ,

Header1

To begin at the beginning

You can use all the quantitative data you can get,

but you still have to distrust it and use your own intelligence and judgment.

Alvin Toffler

There is a touching belief that all markets are rational and that a company’s value is accurately reflected in the current share price multiplied by the total number of shares in circulation.

The thing is, this doesn’t square up with formal accounting practices and financial reporting, as they now stand. So, there is a sentiment that the gap between capitalisation and accounting valuation must be due to some less tangible factors, which is true, and for many years it was accepted that aspects such as good will, were not in fact measurable or even easily manageable.

In the nineties, Leif Edvinsson and a team of accounting and finance specialists at Skandia pioneered the accounting and reporting of non-tangible assets, which they called the Skandia Navigator, the focus was clearly on reporting the progress of the creation and use of Intellectual Capital (IC). What the Skandia IC team did, how they went about it, and how they presented their results, (as a supplement to their annual report,) made a lot of good sense.

Now there is a movement to measure, value and manage all data as if it were a highly tangible asset, because, according to some, IT has been incapable of managing data because they don’t know how to measure it and they aren’t particularly well equipped to do so, a claim that is paradoxically made by people who are very much in the IT camp.

But this is not about treating data as an asset but focuses on potential risks due to data loss, data corruption, quality issues, misinterpretation, misuse or whatever. Simply stated, a risk prevented from arising or a risk that is mitigated is not an asset, and risk management is not primarily about rigidly theoretical asset management and vice verse.

Now, I won’t argue that it’s not a good idea to have an idea of the value of business assets, from either a quantitative or qualitative perspective, but I think there are more moderate, coherent and less fundamentalist approaches to the understanding and valuation of data, approaches that are more aligned to contemporary views of the understanding of the value of intellectual capital assets – unstructured and structured. What follows is a brief and high-level view of some potential options.

Data Assets in MOSCOW

What follows is a simple explanatory technique that is part of the DW 3.0 Information Supply Framework approach. It focuses on qualitative aspects of data as an asset. The following diagram provides an overview of foundations of the first phase of this simplified risk, classification and prioritisation approach:

On the horizontal access we have used the MOSCOW classification to break the problem down into simple terms. Here is an example of its configurable application:

MUST – Must treat the data as an asset: If this data is compromised in anyway then it would possibly result in serious negative implications for all (or most) customers and severe financial, contractual and reputation impacts on the business

SHOULD – Should treat the data as an asset: If this data is compromised in significant ways it would result in intolerable negative financial implications for a number of customers and would also lead to negative financial impacts on business

COULD – Could treat the data as an asset: If this data is compromised in significant ways then it could result in financial consequences for the business, but without impacting customer churn

WON’T – Won’t treat the data as an asset: Data that if compromised would not result in financial consequences wither for customers or for the business

It is important to understand that these definitions are examples and that in practice the descriptions and parameters must be determined and aligned in cooperation between the distinct business stakeholders, including business IT.

Data Value Chains

Another aspect that is important to the understanding of ‘data as an asset’ is to be found in data value chains. The following diagram illustrates the DW 3.0 technique used to simplify the attainment of a consensual view of the understanding of this value.

DATA: “Data is a super-class of a modern representation of an arcane symbology.” –Anon

In order for data to be more than an operational necessity it requires context.

Providing valid data with valid context turns that data into information.

Data can be relevant and data can be irrelevant. That relevance or irrelevance of data may be permanent or temporary, continuous or episodic, qualitative or quantitative.

Some data is meaningless, and there are cases whereby nobody can remember why it was collected or what purpose it has.

Taking all this into account we can now ask the pragmatic question: what value does this data have? Which is sometimes answered with a: ‘no value whatsoever’.

INFORMATION: “My sources are unreliable, but their information is fascinating.” –Ashleigh Ellwood Brilliant

In order for information to be created to drive organizational operations and tactics we must have valid interpretations of data.

Providing valid information with valid interpretations potentially turns that information into knowledge.

Information can be correct, partially correct or incorrect; it may be relevant or irrelevant. The validity of information may also be permanent or temporary, continuous or episodic, and qualitative, quantitative or both.

Some information may be relevant, irrelevant or misleading. Information derived from erroneous and incomplete data may be usable, but the outcomes of using that information may be unpredictable.

Taking all this into account we can now ask the pragmatic question: what value does this information have? A question which can sometimes be answered with an ‘it is probably too early to say’.

KNOWLEDGE: This necessarily refers to a subset of ‘business knowledge’ known as structured intellectual capital.

The adequate, appropriate and timely application of knowledge (structured intellectual) requires wisdom; knowledge alone will no longer be good enough.

Providing valid knowledge with valid interpretations potentially turns that knowledge into valid and executable strategies.

Knowledge may be useless or useless; it may be relevant or irrelevant; it may even be wrong, even if we have named it ‘knowledge’. The usefulness of knowledge may also be permanent or temporary, continuous or episodic, qualitative or quantitative.

Some knowledge may be relevant, irrelevant or misleading. Knowledge derived from erroneous and incomplete information may be usable, but the outcomes may be unpredictable. As they might say in Córdoba: ‘Give knowledge to a donkey and it will still remain a donkey’.

Taking all of this into account we can now ask the pragmatic question: what value does this knowledge have? A question which can sometimes be answered with: ‘Isn’t it getting rather hot in here’.

What does it all mean?

To paraphrase George S. Patton, for the purposes of business data, an imperfectly good pragmatic valuation of data executed today is better than a perfect theoretic valuation of data made in the next life.

The valuation of data, information and knowledge is complex and involves many intangibles, and although some may view the data, information and knowledge chain as a closed process, this is not in fact the case, as each step of the process is influenced by a number of factors that fall outside of a simplified view of any theoretical or practical progress from data to wisdom.

When I started in IT over 30 years ago I worked on some of the first large-scale OLTP projects in Europe. On one such project, the development, test and production the databases were continuously replicated. Every day, not one, but ten backups of the data were made, and then shipped to different physical locations.

Thirty years on and some people are saying that IT has never treated data as an asset?

The fact of the matter is that data is managed as an asset. In fact, some IT organizations may be taking exaggerated measures in order to protect the data and information in their care. That some people have issues with the contemporary management of data does not change the facts on the ground, and the perceived shortfall in the commercial exploitation of data is frequently and erroneously interpreted as the absence of asset appreciation and management.

Moreover, even with generally accepted accounting principles there are tangible assets that are either of dubious value or are cost-absorbing liabilities.

Finally, there may also be unintended consequences of an overzealous approach to the financial reporting of intangible assets such as data, and just as the value of tangible assets can be accidentally or deliberately overstated, so too could the value of data, which may well lead to a significant overvaluation of a business. Moreover, it’s not beyond the realms of possibility that big data lakes, data universes and dark data dungeons – the ones that apparently will drive trillion dollar economies – are in fact somewhat worthless.

In this respect I think there is in some quarters an unhealthy fixation on the supposed (and as yet mainly unproven) value accruable from masses of data.

That’s all folks

Steve Jobs didn’t turn Apple around by relying solely on data and information, it was knowledge, and more especially the wisdom of knowing where, when, why and how to apply that knowledge that made the significant difference.

So, until the next time, hold this thought: ‘Google market cap slashed and Facebook in freefall as accountants ask “where’s the big data beef?”… Breaking news!’

Many thanks for reading.

Consider this: Social Media Big Data killed Ad-land

21 Wednesday Jan 2015

Posted by in Big Data, Consider this, Information Supply Frameowrk

≈ Leave a comment

Tags

, , , , ,

Hold this thought: Big Data is the future of online business and interactive advertising is its profit.

Much is being made of Big Data and its role in social media and online interactive advertising. The advertising industry itself has a “big crush” on Big Data, and it fuels the elevated revenues, profits and share prices of a number of online companies.

Continue reading →

Consider this: Big Data and the Analytics Data Store

19 Monday Jan 2015

Posted by in Analytics, Big Data, Consider this, statistics

≈ Leave a comment

Tags

, , , ,

To begin at the beginning

Hold this thought: If Data Warehousing was Tesco then Big Data would be the “try something different”.

Since the publication of the article Aligning Big Data, which basically laid out a draft view of DW 3.0 Information Supply Framework and placed Big Data within a larger framework, I have been asked on a number of occasions recently to go into a little more detail with regards to the Analytics Data Store (ADS) component. This is an initial response to those requests. Continue reading →

The World’s Best Data Quotes… Including Big Data quotes

17 Saturday Jan 2015

Posted by in Analytics, Architecture, Big Data, Business Intelligence, Consider this, Data Warehousing, statistics

≈ 4 Comments

Tags

, , , , , ,

OLYMPUS DIGITAL CAMERA

A Random walk down Data Street

If you enjoy, abhor or are simply bored with the massive surfeit of hype surrounding Big Data, Data Warehousing and Analytics, then you might just hate these less than faithful quotes as well.

If you enjoy one or two of the quotes, well, then that’s an acceptable bonus too.

So, to begin at the beginning…xHound

Data Sources

“My data sources are unreliable, but their information is fascinating.” – Ashleigh Brilliant

“I give no data sources, because it is indifferent to me whether what data I have sourced has already been sourced before me by another.” – Ludwig Wittgenstein

“In the kitchen of a great Data Warehouse, the data source chef is a soloist.” – Fernand Point

“It is better to be hated for what data sources you have than to be loved for what data sources you do not have.” – André Gide

“In England, there are sixty different types of Data Warehouse and only one data source.” – Attributed to Voltaire

“It is a capital mistake to theorize before one has data sources. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” – Arthur Conan Doyle, Sherlock Holmes

“From such a gentle thing, from such a source of all data, my every pain is born.” –Michelangelo

“Noise free data is a source of great strength.” – Lao Tzu

“In three words I can sum up everything I’ve learned about data: it goes on.” – Robert Frost

“Data enrichment improves a mighty fine data source” – Anonymous

xButcherBig Data

“Junk food, empty calories and carbs are the Big Data of the masses” – Karl Marx

“We live, I regret to say, in an age of Big Data hype.” – Oscar Wilde

“We are not rich by the Big Data we possess but by what Big Data we can do without.” – Immanuel Kant

“He who has Big Data hype on his side has no need of proof.” – Theodor Adorno

“The religion of Big Data sets itself the goal of fulfilling man’s unattainable desires, but for that very reason ignores her attainable needs.” – Ludwig Feuerbach

“The flesh endures the storms of the present alone; the mind in our social network interactions, those of the past and future as well as the present. Big Data is a covetousness of the mind.” – Thomas Hobbes

“Big Data is negative and dialectical, because it resolves the determinations of the understanding of things into nothings.” – Georg Wilhelm Friedrich Hegel

“I am trapped in this Big Data, and there is nothing I can do about it.” – Dudley Moore

“And remember, never take the ruby case off your iPad for a moment, or you will be at the mercy of the Big Data Witch of the West.” – The Wizard of Oz

“Imagine there’s no Big Data…” – John Lennon

Abacus3Data Transformation

“Analysis does not transform data.” – Jiddu Krishnamurtu

“I live in a data landscape, which every single day of my life is enriching data.” – Daniel Day-Lewis

“Data opportunities multiply as the data is transformed” – Sun Tzu

“He who integrates data badly is lost.” – Theodor Adorno

“Today we transform the data; tomorrow, the whole enchilada” – Leon Trotsky

“Well, it’s all about the ETL law of the transformation of data quantity into data quality, and vice versa. Innit!” – Friedrich Engels

“The management consultants have only interpreted the business data, in various ways. The point, however, is to transform it.” – Karl Marx

“Hey! What’s going down here in the Hollyweird of data?” – Joe McCarthy

“The Big Data alchemists in their transformational search for gold discovered much data of greater value.” – Arthur Schopenhauer

“That Schopenhauer yolk was a bit of an old Big Data ‘procurer’ wasn’t he now Rodge?” – Pádraig Judas O’Leprosy

IMGQBusiness Intelligence

“The trouble with the world is that the cocksure have Big Data and that Data Science and Business Intelligence are all sexed up.” – Bertrand Russell

If people never did silly things no Business Intelligence would ever get done.” – Ludwig Wittgenstein

“The best Business Intelligence user is intelligent, well-educated and a little drunk.” –Alben W Barkley

“The Master said, “If your conduct is determined solely by considerations of Business Intelligence and profit you will arouse great resentment.” ― Confucius

“That’s cricket, Harry, you get these sort of things in Business Intelligence” – Frank Bruno

“Business Intelligence without ambition is a bird without wings.” – Salvador Dali

“I would prefer a Business Intelligence hell to a Big Data paradise.” – Blaise Pascal

“Many much-learned business men have no Business Intelligence.” – Democritus

“We should not only use the brains we have, but all that we can borrow.” – Woodrow Wilson

“The reason we have Business Intelligence is so we don’t have to think all the time” –Homer Simpson

P3160034Data Warehousing

“The study of Data Warehousing, like the Nile, begins in Inmon and ends in magnificence.” – Charles Caleb Colton

“Big Data wins games, but Data Warehousing wins championships.” – Michael Jordan

“Big Data is no substitute for Data Warehousing.” – Frank Herbert

“It’s in me blood, Clive, without Data Warehousing I’d be nothing,” – Alan Latchley

“The trouble with the world is that the cocksure have Big Data and that Data Science is all sexed up.” – Bertrand Russell

If people never did silly things no Business Intelligence would ever get done.” – Ludwig Wittgenstein

“The best Business Intelligence user is intelligent, well-educated and a little drunk.” –Alben W Barkley

“You can catch all the whales in the ocean and stack them together and they still do not make a minnow.” – Ralph Wiggum

“Well, the smarter I practice Inmon Data Warehousing, the luckier I get.” – Gary Player

“Well, I’ve cleaned up facts and dimensions in a star-schema ‘data warehouse’. That was pretty terrible. But I can’t complain because I’m sure other people have done worse.” – Cee Lo Green

“You can give a person a bowl of Big Data Gruel and feed them for a day, or teach them Inmon Data Warehousing and feed them for a lifetime.” – Proverb

“A Data Warehouse is like a tea bag; you never know how strong it is until you are in hot water.” – Eleanor Roosevelt

” οἶδα δ᾽ ἐγὼ ψάμμου τ᾽ ἀριθμὸν καὶ μέτρα θαλάσσης, καὶ κωφοῦ συνίημι, καὶ οὐ φωνεῦντος ἀκούω. ὀδμή μ᾽ ἐς φρένας ἦλθε κραταιρίνοιο χελώνης ἑψομένης ἐν χαλκῷ ἅμ᾽ ἀρνείοισι κρέεσσιν, ᾗ χαλκὸς μὲν ὑπέστρωται, χαλκὸν δ᾽ ἐπιέσται.” – An Oracle to Croesus of Lydia

IMGThat’s all folks!

Well, now that that’s done I can always ask for forgiveness. Not that I will of course.

 Many thanks for reading.

abfab111

Founder and CEO, Cambriano Energy

File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro

Continue reading →

Consider this: Aligning Big Data

13 Tuesday Jan 2015

Posted by in Big Data, Consider this

≈ Leave a comment

Tags

Intro

In order to bring some semblance of simplicity, coherence and integrity to the Big Data debate I am sharing an evolving model for pervasive information architecture and management.

This is an overview of the realignment and placement of Big Data into a more generalized architectural framework, an architecture that integrates data warehousing (DW 2.0), business intelligence and statistical analysis.

The model is currently referred to as the DW 3.0 Information Supply Framework, or DW 3.0 for short.

A recap

In a previous piece with the name of Data Made Simple – Even ‘Big Data’, I looked at three broad-brush classes of data: Enterprise Operational Data; Enterprise Process Data; and, Enterprise Information Data. The following is a diagram taken from that piece:

Fig. 1 – Data Made Simple

In simple terms the classes of data can be defined in the following terms:

Enterprise Operational Data – This is data that is used in applications that support the day to day running of an organisation’s operations.

Enterprise Process Data – This is measurement and management data collected to show how the operational systems are performing.

Enterprise Information Data – This is primarily data which is collected from internal and external data sources, the most significant source being typically Enterprise Operational Data.

These three classes form the underlying basis of DW 3.0.

The overall view

The following diagram illustrates the overall framework:

Fig. 2 – DW 3.0 Information Supply Framework

There are three main elements within this diagram: Data Sources; Core Data Warehousing (the Inmon architecture and process model); and, Core Statistics.

Data Sources – This element covers all the current sources, varieties and volumes of data available which may be used to support processes of ‘challenge identification’, ‘option definition’, decision making, including statistical analysis and scenario generation.

Core Data Warehousing – This is a suggested evolution path of the DW 2.0 model. It faithfully extends the Inmon paradigm to not only include unstructured and complex data but also the information and outcomes derived from statistical analysis performed outside of the Core Data Warehousing landscape.

Core Statistics – This element covers the core body of statistical competence, especially but not only with regards to evolving data volumes, data velocity and speed, data quality and data variety.

The focus of this piece is on the Core Statistics element. Mention will also be made of how the three elements provide useful synergies.

Core Statistics

The following diagram focuses on the Core Statistics element of the model:

Fig. 3 – DW 3.0 Core Statistics

What this diagram seeks to illustrate is the flow of data and information through the process of data acquisition, statistical analysis and outcome integration.

What this model also introduces is the concept of the Analytics Data Store. This is arguably the most important aspect of this architectural element.

Data Sources

For the sake of simplicity there are three explicitly named data sources in the diagram (of course there can be more, and the Enterprise Data Warehouse or it’s dependent Data Marts may also act as a data source), but for the purpose of this blog piece I have limited the number to three: Complex data; Event data; and, Infrastructure data.

Complex Data – This is unstructured or highly complexly structured data contained in documents and other complex data artefacts, such as multimedia documents.

Event Data – This is an aspect of Enterprise Process Data, and typically at a fine-grained level of abstraction. Here are the business process logs, the internet web activity logs and other similar sources of event data. The volumes generated by these sources will tend to be higher than other volumes of data, and are those that are currently associated with the Big Data term, covering as it does that masses of information generated by tracking even the most minor piece of ‘behavioural data’ from, for example, someone casually surfing a web site.

Infrastructure Data – This aspect includes data which could well be described as signal data. Continuous high velocity streams of potentially highly volatile data that might be processed through complex event correlation and analysis components.

The Revolution Starts Here

Here I will backtrack slightly to highlight some guiding principles behind this architectural element.

Without a business imperative there is no business reason to do it: What does this mean? Well, it means that for every significant action or initiative, even a highly speculative initiative, there must be a tangible and credible business imperative to support that initiative. The difference is as clear as that found between the Sage of Omaha and Santa Claus.

All architectural decisions are based on a full and deep understanding of what needs to be achieved and of all of the available options: For example, rejecting the use of a high performance database management product must be made for sound reasons, even if that sound reason is cost. It should not be based on technical opinions such as “I don’t like the vendor, much”. If a flavour of Hadoop makes absolute sense then use it, if Exasol or Oracle or Teradata make sense, then use them. You have to be technology agnostic, but not a dogmatic technology fundamentalist.

That statistics and non-traditional data sources are fully integrated into the future Data Warehousing landscape architectures: Building even more corporate silos, whether through action or omission, will lead to greater inefficiencies, greater misunderstanding and greater risk-

The architecture must be coherent, coherent, usable and cost-effective: If not, what’s the point, right?

That no technology, technique or method is discounted: We need to be able to cost-effectively incorporate any relevant, existing and emerging technology into the architectural landscape.

Reduce early and reduce often: Massive volumes of data, especially at high speed, are problematic. Reducing those volumes, even if we can’t theoretically reduce the speed is absolutely essential. I will elaborate on this point and the following separately.

That only the data that is required is sourced. That only the data that is required is forwarded: Again, this harks back on the need for clear business imperatives tied to the good sense of only shipping data that needs to be shipped.

Reduce Early, Reduce Often

Here I expand on the theme of early data filtering, reduction and aggregation. We may be generating increasingly massive amounts of data, but that doesn’t mean we need to hoard all of it in order to get some value from it.

In simplistic data terms this is about putting the initial ET in ETL (Extract and Transform) as close to the data generators as possible. It’s the concept of the database adapter, but in reverse.

Let’s look at a scenario.

A corporation wants to carry out some speculative analysis on the many terabytes of internet web-site activity log data being generated and collected every minute of every day.

They are shipping massive log files to a distributed platform on which they can run data mapping and reduction.

Then they can analyse the resulting data.

The problem they have, as with many web sites that were developed by hackers, designers and stylists, and not engineers, architects and database experts, is that are lumbered with humungous and unwieldy artefacts such as massive log files of verbose, obtuse and zero-value adding data.

What do we need to ensure that this challenge is removed?

We need to rethink internet logging and then we need to redesign it.

  • We need to be able to tokenise log data in order to reduce the massive data footprint created by badly designed and verbose data.
  • We need to have the dual option of being able to continuously send data to an Event Appliance that can be used to reduce data volumes on an event by event and session basis.
  • If we must use log files, then many small log files are preferable to fewer massive log files, and more log cycles are preferable to few log cycles. We must also maximise the benefits of parallel logging. Time bound/volume bound session logs are also worth considering and in more depth.

So now, we are either getting log data to the point of use either via log files, log files produced by an Event Appliance (as part of a toolkit of Analytic Data Harvesting Adapters) or sent by that appliance to a reception point via messaging.

Once that data has been transmitted (conventional file transfer/sharing or messaging) we can then move to the next step: ET(A)L – Extract, Transform, Analyse and Load

For log files we would typically employ ET(A)L but for messages of course we do not need the E, the extract, as this is about direct connect.

Again the ET(AL) is another form of reduction, which is why the analytics aspect is included to ensure that the data that gets through is the data that is needed, and that the junk that has no recognisable value, gets cleaned out early and often.

The Analytics Data Store

The ADS (which can be a distributed data store on a Cloud somewhere) supports the data requirements of statistical analysis. Here the data is organised, structured, integrated and enriched to meet the ongoing and occasionally volatile needs of the statisticians and data scientists focusing on data mining. Data in the ADS can be accumulative or completely refreshed. It can have a short life span or have a significantly long life-time.

The ADS is the logistics centre for analytics data. Not only can it be used to provide data into the statistical analysis process, but it can also be used to provide persistent long term storage for analysis outcomes and scenarios, and for future analysis, hence the ability to ‘write back’.

The data and information in the ADS may also be augmented with data derived from data stored in the data warehouse, it may also benefit from having its own dedicated Data Mart specifically designed for this purpose.

Results of statistical analysis on the ADS data may also result in feedback being used to tune the data reduction, filtering and enrichment rules further downstream, either in smart data analytics, complex event and discrimination adapters or in ET(AL) job streams.

That’s all folks.

This has been necessarily a very brief and high-level view of what I currently label DW 3.0.

The model doesn’t seek to define statistics or how statistical analysis is to be applied, which has been done more than adequately elsewhere, but how statistics can be accommodated in an extended DW 2.0 architecture, and without the need to come up with almost reactionary and ill-fitting solutions to problems that can be solved in better and more effective ways through good sense, sound engineering principles and the judicious application of appropriate methods, technologies and techniques.

If you have questions or suggestions or observations regarding this framework, then please feel free to contact me either here or via LinkedIn mail.

Many thanks for reading.

File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro

Who’s Who Mugs

12 Monday Jan 2015

Posted by in Condiser this

≈ Leave a comment

Tags

whoswho

A couple of day’s back I was looking through some really interesting LinkedIn articles published via Pulse, and my mind started to wonder.

On each article page an advert for a professional Who’s Who service appeared, the same advert, again and again, just begging to be clicked.

So, I did something that I very rarely do, and that was to click through to the web site, completely ignoring long held views regarding professional Who’s Who services and other types of scams, and putting aside for once the tried and trusted heuristic of ‘there is no such thing as a free lunch’.

Curiosity had got the better of me. Superficially it looked like yet another networking opportunity, and the impression they carefully projected was that it was ‘for free’. So I clicked on the link. To be fair – to myself, I also wanted to see how elaborate the scam would be and when and how they would actually ask for more than information.

So after passing through to a registration web site I was then asked to enter some basic qualifying information which was supposedly to be used in a first stage assessment of my professional experience.

I entered the information, it wasn’t much and it wasn’t potentially compromising, and then completely forgot about the whole thing. That is, until I received a call today.

“Hello, this is Mini Beamer from Brie and Stilton ‘Big Cheeses in Who’s Who’. I am calling you in order to qualify you for entry into our prestigiously professional international networking and recognition hall of fame”.

So, for the best part of twenty five minutes I answer – candidly even – a succession of questions, and talked about myself and my business. We touched on education, skills, strong points, key aspects in personal and professional success, business and personal web sites, public speaking, lectures, published work, blogs, hobbies, sports, voluntary work, blah, blah, bloody blah, and more.

Towards the end of the conversation I was informed that given my exceptional professional knowledge and experience (oh, here we go) that I certainly qualified for inclusion in their Who’s Who. It was odd, because even after twenty five minutes I still hadn’t been asked for money.

Then the moment came.

“So, do you want the 5 year plan at 800 bucks or the life-long plan at 1200 bucks?”

Then the pitch came on the difference between the two offers.

“Most of our professionals go for the life-long plan because it’s the most cost effective.”

So I answered “let me think about it, and I will get back to you”.

Wrong answer!

I was told that this was not possible and that I must decide, there and then, between the five year plan and the life-long plan. I presumed that the credit card number would be the next piece of ‘necessary’ information.

The sales person then ratcheted up their game, and insisted on the extensive benefits that would accrue to me from having them extend my network ‘to the max’ and in them winning well-deserved global recognition of my work in my chosen professional fields.

Being a reasonable person I tried to calmly reason with the caller. My basic message was that I was not interested. The caller failed to hear me. I tried to stop the flow of the sales pitch – several times again and without success, and I eventually hung up the phone.

Did I know that there would be a cost involved? Of course I didn’t know, but I knew that this was a 99.9% possibility.

Did I really want to be included in such a Who’s Who? I wasn’t sure, but I wanted to find out what their angle was.

The strangest thing about the conversation was that when it came to websites and networking to achieve sales leads and real business, which was in the first five minutes of the call, I basically said:

“My company has a web site, I have a web site, I have a blog. We make no business through them. We need the corporate web site because people like to work with companies that have a web site. It’s like a relationship comfort-blanket. I am on Facebook, Twitter and LinkedIn, but I don’t use them to generate business, at all. That’s not my shtick or that of my company. And there’s a reason for it”

I made it very clear, at least in my mind, that our business model was not based on generating any business via internet channels. I know, I know, it’s strange. I deal in bleeding-edge and leading-edge technologies, innovative strategies that address significant challenges, and in business performance and risk, amongst other things, and it’s totally on the commercial edge of ‘new’, innovative and predatory thinking. So it may sound odd that all of my business engagements and dealings, and that of my associated company, are made entirely from old fashioned networking and lead generation, qualification and actions – old boys and girls networks, just like how it was done in the days when there were only bricks and mortar businesses.

That said, I have been given business leads by people I am connected with on the internet sites such as LinkedIn, but in my case it’s because I know these people personally, some are even really good friends, I get new business because I know them and they know me, not from being on a social or professional network site. That’s just the way the business has evolved, even if for other people the story may be completely different.

Maybe in the future we will start to generate leads from internet based networking. At that time I would need to consider what strategy to adopt. But right now we’re not there, nor need to be, and I will certainly never contemplate paying 1200 bucks to a dodgy Who’s Who company to manage my network and to generate professional recognition.

Later I looked up the web site of the Who’s Who Company. I just wanted to see if any of my closest partners, friends and colleagues appeared on their listings. Not one of them was registered. Then I looked for some big names; Branson, Gates, Ellison, Dell, and Bezos? Again, not even one.

What little rascals!

Thanks for reading.

File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro

Big Data? Dopey Quotes?

10 Saturday Jan 2015

Posted by in Big Data

≈ Leave a comment

Tags

Imagen1

I came across a blog post titled “The Best Big Data Quotes Of All Times”.

At first I ignored it. I really like the author, but I’m generally not a fan of this type of ‘content’; of top 10 lists or cute collections or amazing pussy cat videos on YouTube, although I respect other people’s tastes.

But because of its continued high visibility on the LinkedIn network site – every damn page I visited seemed to have a link to it, I eventually succumbed and took a look.

I read all the quotes, and wondered, ‘what the bloody hell was that all about then?’

You may want to read the piece yourself, or you could save yourself a lot of exasperation and the wasteful death of brain cells, by simply avoiding that piece and the rest of this piece altogether.

You have been warned.

Now here for your delight is a blow by blow flashback of the experience – with the names removed to protect the guilty.

Quote #1: “Big data is at the foundation of all the megatrends that are happening today, from social to mobile to cloud to gaming.”

Gobsmacked reaction: “Boloney in a handcart going to hell! If there are people who think that Big Data is anywhere other than on the peripheral edge of a peripheral edge then they clearly need to get out more often and listen to real people.”

Quote #2: “Big data is not about the data”

Gobsmacked reaction: “Say what? Big Data missionaries have always primarily made it about the data, you know, the 3 or 4 or 10 Vs of Big Data and all that jazz. In fact one of the shticks that the Big Data proselytizers have schlepped around is that Data Warehouse can’t do bigger data because it can’t handle the volumes. What they forget to add is that if you first remove the wheat from the chaff it becomes a non-issue.”

Quote #3: “There were 5 exabytes of information created between the dawn of civilization through 2003, but that much information is now created every 2 days.”

Gobsmacked reaction: “Yet another dope who thinks that one can calculate the value of data by simply weighing it? A ‘sexed up’ version of my n exabytes of ignorance is just as good as your 2 exabytes knowledge”

Quote #4: “Information is the oil of the 21st century, and analytics is the combustion engine.”

Gobsmacked reaction: “This assertion is so dopey that it defies the conventional laws of stupid. For the best part of two centuries we have appreciated the power of having the right information and that our enemies trust in disinformation. But to compare this with the historic and contemporary use and utility of energy sources is off the wall. In a word the comparison is vapid.”

Quote #5: “I keep saying that the sexy job in the next 10 years will be statisticians, and I’m not kidding”

Gobsmacked reaction: “What? When people say weird shit like this one should be wary about asking for clarification.”

Quote #6: “You can have data without information, but you cannot have information without data.”

Gobsmacked reaction: “Of course you can have information without data; it’s just that we usually associate information with the availability of underlying data, which isn’t always the case.”

Quote #7: “Hiding within those mounds of data is knowledge that could change the life of a patient, or change the world.”

Gobsmacked reaction: “Sure! There’s gold in them there hills… or not. Isn’t that right Madame Soleil?”

Quote #8: “Errors using inadequate data are much less than those using no data at all.”

Gobsmacked reaction: “What? Probability might indicate that this claim could possibly be erring on the wrong side of likely. Apart from that, the bizarre and intellectually questionable juxtaposition and conflation of this quote with Big Data is somewhat embarrassing.”

Quote #9: “To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem – he may be able to say what the experiment died of.”

Gobsmacked reaction: To call in a qualified surgeon after a piece of hobbyist DIY open heart surgery goes awry may be no more than asking him to perform a post-mortem – but he may be able to say what the bodger’s patient died of.”

Quote #10: “Without big data, you are blind and deaf in the middle of a freeway”

Gobsmacked reaction: “Has the Cosa Nostra gone into Big Data now? This looks more like a threat than an observation. Unless of course the observation was made after smoking forty blunts.”

Unfortunately there are many more like this and worse, and I would love not to hear your favourite ones, I am absolutely sick of them all. Why not make this list a little longer – but without my knowledge – by adding your favourite crap big data quotes in the comments below?

You might also want to share or download this little PowerPoint slide-dreck:

File under: Good Strat, Good Strategy, Martyn Richard Jones, Martyn Jones, Cambriano Energy, Iniciativa Consulting, Iniciativa para Data Warehouse, Tiki Taka Pro