Learning journey – Agile@Scale – The video
22 Wednesday May 2019
Posted in agile, 4th generation Data Warehousing, agile@scale
Category Archives: agile
Learning to hate Agile-to-scale
11 Saturday May 2019
Afilonius Rex
Cordoba, 12th May 2019
Ladies and gentlemen, the performance is about to start!
In 1998, as Programme, Product and Competence Centre manager at Hewlett-Packard, I signed the Agile Manifesto.
At that time, I had been using elements of the Agile-method mindset for the best part of sixteen years.
Previously, in 1995 I was trained and certified in Iterations, an iterative and agile-like methodology for building, maintaining and expanding data warehouses and data marts. Continue reading
Getting Agile Right
17 Monday Apr 2017
Martyn Richard Jones
Madrid, 16th April 2017
I think Agile has its place, especially in the world of software application development, especially in building the kind of software that businesses frequently use in the peripheral aspects of their day-to-day operations.
Done right, Agile can make the difference between a great success and a painful failure. Done badly, and you might suffer a worse fate than a badly applied Software Development Life Cycle such as Waterfall. Done well, and Agile will oil the wheels of the machine that gets things done.
Big Data Predictions for 2017
01 Sunday Jan 2017
Posted in 4th generation Data Warehousing, accountability, agile, All Data, Big Data, Big Data 7s, Big Data Analytics, dark data, data architecture, Data governance, Data Lake, data management, data science, Data Supply Framework, Data Warehouse, Data Warehousing, pig data, The Amazing Big Data Challenge, The Big Data Contrarians
Big Data Predictions for 2017
Prologue
You want Big Data predictions for 2017?
You’ve got ’em!
These are my Big Data, Data Warehousing and Analytics extrapolations for 2017. They are based on extensive, exhaustive and enigmatic work carried out by top-notch researcher gurus at Cambriano Energy, between December 2015 and December 2016.
So, stick with us as we survey the landscape that will be Big Data in 2017. Continue reading
The Million-Dollar Big Data Briefing
11 Wednesday Nov 2015
If you enjoy this piece or find it useful then please consider joining The Big Data Contrarians: https://www.linkedin.com/grp/home?gid=8338976
Many thanks, Martyn.
Big Data, together with Cloud computing, the Internet of Things and Machine Learning, are topics that are very much to the fore in contemporary trends in Information Management. But is Big Data really the revolution that people have been waiting for or is it simply about the next steps in the evolution of business data architecture and management?
Whilst it is true that we are increasingly generating more and more data, do we really need to rethink our whole approach to data demand and data supply, or can we build on the best of what we have already accomplished in order to move with intelligence, rigour and stability to the next level?
We are told that all organisations, small, medium, large and gigantic, will be overwhelmed with an existential threat if they don’t recognise and act on the fundamental need to leverage all data – massive data, in all formats, all volumes and all qualities. From wherever it may come from, legally or legally.
But is this really the case? Are we being urged to action by caring, principled and disinterested good data Samaritans? Or are we being railroaded into a shallow, anti-intellectual and mass-reactionary fervour that will run ultimately against the grain of our own best interests? Whether as individuals or as stakeholders.
I am not entirely convinced by the breadth or depth or intellectual veracity of the claims for Big Data, which frequently border the hyperbolic and the absurd, but I also think it is wise to be forearmed as well as forewarned.
That is why I am encouraging a discussion about where Big Data and Data Science fit into a much larger picture of business information supply and demand.
To begin at the beginning
There are many dimensions to business data and information, and here I will touch on some of those dimensions here, albeit with a broad brush.
There are also many potential sources of data; these can be divided into two broad dimensions: analogue data and digital data.
Analogue data – Any type of data not stored in a digital format. This covers the data found in analogue form. For the purpose of this piece this dimension is overlooked.
Digital data – Any type of data stored in a digital format. The source may be a conventional database, a log file, and audit trail, a spreadsheet, document, presentation, project plan or other similar source.
What is being contemplated is data that could help us to interpret history; manage the present; and, plan for the future.
Past – What happened and what can we learn from it?
Present – What is happening and what, if anything, should we do differently?
Future – What might or might not happen and what might we do or not do if this happens, or to prevent this from happening?
If the data in focus cannot help us with any of these dimensions does that data really matter?
The third dimension of data that I want to identify concerns place (internal and external):
Internal data – This is the information that belongs to the organisation and is obtained from internal sources, such as operational systems.
External data – This is the data from external sources, such as market data providers. that an organisation can legally and legitimately acquire and use.
And finally, for the purpose of brevity, the dimensions of demand and supply:
Demand driven data – This is when business is the driver for the flow of information. Data is delivered because there is a business demand. In simple terms this is about giving people what they want, continuously.
Supply driven data – This is about delivering data and information based upon what is available. If done on the right scale and with the aim of creating new internal markets for data, then it can be beneficial. This is where IT needs to be far more sales, marketing and Ad-land oriented.
But what do all of these considerations mean to a managers need for adequate, appropriate and timely information? Let’s take a look.
The following diagram illustrates the overall drivers for the Information Supply Framework.
From the right, the consumers and prospective consumers of data and information create the demand for information and data.
The middle process of ‘data processing, enrichment and information creation’ strives to meet the business demands and also ‘provokes’ business demands for data and information.
From the left the data sources provide data to meet real and secondary demands.
So how do we ‘make it happen’?
How do we try and obtain the business advantages supposedly accruable to Big Data without succumbing to yet another anarchic information management aberration that will take us one step forward, and two steps back?
Let me explain.
Is this the sort of diagram that gives you a warm and fuzzy feeling?
Of course not, it’s a mess. Print it out on a plotter and place it on the office wall, by all means. You can even call it art, but it’s also a recipe for data bedlam hell.
Now contrast that colourful fun-loving anarchical approach to something I prepared earlier.
There, isn’t that better?
It’s clean, cohesive and coherent. It’s a rational, structurally sound and practical model to support the business demands for data and for the speculative supply of data. The model is based on unifying strands of business data requirements, (with specific regard to operational, tactical and strategic aspects,) into to an integrated, agile and flexible whole.
In short, it is a coalescing model for business data and information.
That’s all folks
Is it absolutely unnecessary in this day and age to take a maverick, piecemeal and DIY approach to the provision of non-traditional business data and information?
No, evidently it’s not.
There is no rational business reason to use speculative, quick and dirty – and characteristically ‘throw away’ approaches to data supply, when quick, clean, useful, maintainable, cost effective and usable alternatives are available.
The Information Supply Framework provides a strategic approach to meeting the data and information needs of an organisation. It accurately reflects business process, it can change in step with the changing needs of the organisation, and it is capable of handling the emerging requirements associated with ever increasing data volumes, data variety and data velocity.
Simply stated, the DW 3.0 Information Supply Framework provides a nascent opportunity for businesses to easily transition to the inclusion of more formal and rigorous methods of statistical analysis, analysis that can now be powered by the inclusion of new, alternative and complementary sources of data.
Naturally the choice is yours. But ensure that you weigh up your options with your ‘eyes wide open’, or it may well all end in tears.
Many thanks for reading.
If you enjoyed this piece or found it useful then please consider joining The Big Data Contrarians: https://www.linkedin.com/grp/home?gid=8338976
Many thanks, Martyn.
Consider this: Big Data and the Curse of the Temple of Java
13 Friday Mar 2015
Posted in agile, Big Data, Consider this, java
“Rats, rats for sale. Get your rats. Good for rat stew, rat soup, or the ever-popular ratatouille”. – Mel Brooks
Hold this thought: Everything that the Templars of Java touch turns to dreck.
In a small and timeless village in misty and mountainous Transylvania, the locals mourn the passing of yet another victim.
On the wind swept beaches of a wintry Costa Blanca, the reverberating voice of childish despair is barely perceptible through the crashing of the waves on the grey, cold and craggy rocks.
In Victorian London, a hobgoblin of indescribable and vacuous insanity stalks the silent and rain drizzled streets.
Cracking this curse will take more than the combined powers of Clint Eastwood, Mel Brooks and Homer Simpson.
A spectre haunts the face of Europe, the spectre of Big Data and the Curse of the Temple of Java.
Everything that the disciples of the Temple touch turns to blah. Everything that the disciples call their own has been blagged from elsewhere.
Take the very language of Java itself, an authentic eccentricity amongst computing languages. If Java code were real coffee grains, it would be used to make the shittiest coffee in the history of humankind.
Given the vast amounts of knowledge and experience that was washing around IT at the time of Java’s hatching, it must be considered to be the most demonic aberration of a programming language ever conceived by woman, man or beast.
“Cats have a scam going – you buy the food, they eat the food, they go away; that’s the deal.” – Eddie Izzard
If ever there was an excuse in IT for failing to deliver or for delivering badly and late, then Java is your friend.
In the hands of the right people, Java can turn a one year and $3M project into a five year and $300M project, and still not deliver anything of use.
Yet magically, and out of the people directly responsible for these debacles, no one is sacked, sued or busted as a result, the incumbent supplier either quietly leaves the scene or is rewarded for their gross incompetence and dishonesty, and in many cases a success is hailed, even if that success looks remarkably like abject failure. It is totally false, absolutely dishonest and thoroughly unprofessional. But that’s what we have, like it or not.
Java sucks, it is a horrid language, aesthetically and functionally, it’s a piecemeal pile of do-do, a dirty old ragbag of ‘object-oriented’ hacks, logical aberrations and lagoons of missing structure, dysfunctional rationality and discontinuity – and that that’s not just my opinion:
“I spent several months programming in Java. Contrary to its authors’ prediction, it did not grow on me. I did not find any new insights – for the first time in my life, programming in a new language did not bring me new insights. It keeps all the stuff that I never use in C++ – inheritance, virtuals – OO gook – and removes the stuff that I find useful.” – Alexander Stepanov
“Claiming Java is easier than C++ is like saying that K2 is shorter than Everest.” – Larry O’Brien
“I would rather use Java than Perl. And I’d rather be eaten by a crocodile than use Java.”
“If I wanted plastic scissors I’d use Java. Give me my scalpel back.”
And for the record, even Linus Torvald hates it.
But if you thought Java was a horrid, hype infested viper’s den of programming bad practice and hyper-hype, just wait until you see what’s behind Hadoop.
As long as the world is turning and spinning, we’re gonna be dizzy and we’re gonna make mistakes. – Mel Brooks
Hadoop must be the biggest piece of technical and rhetorical bullshit in the history of data management.
Repackage a series of Unix primitives (cat, grep, awk, cut, sed, wc) built on top of parallel Linux or Unix. Dress it up, take it out on the town, and call it the greatest thing since sliced bread. It is nothing less than a brazen and blatant con. Want to count words? Use wc (Unix wordcount).
Let me repeat that, using other words. If you made a compilation of extracts from the works of the world’s greatest thinkers and authors, randomised replacement of some of the words, and produced and published this compilation, as all your own work, what would you call that?
So back to when this happens, frequently, in IT.
This might fool the foolish who don’t have the first idea about anything technical, objective or rational beyond whatsapp, kiddy scripting and HTML, but if you have a clue, you know that this is a scam, a very big one. It is also dishonest.
So how do they (the scammers) get away with it?
Easy. You have bad apples everywhere. But there is another reason. For well over a decade the world of IT has become the dumping ground for the stupid, lazy and indolent kids of the comfortable middle-classes and also a hunting ground for unscrupulous wide-boys.
Listen up parents!
Do you think that your kid is way too thick to be a doctor, scientist, lawyer, researcher, professor, teacher, statistician, health worker, politician, bus driver, street cleaner, entrepreneur, sandwich maker or economist?
Your kid has no creativity beyond messing with their food?
Your kid has no sporting ability apart from skills at gaming?
The only academic ability your kid has is your money?
No worries!
IT for you, my son!
So if that’s you, then lap it up. Real knowledge and experience will not come your way, but you will learn the dogma of the Temple of Java, and you will be able to repeat it to perfection, just like Pavlov’s favourite dog.
You will learn to be be pliable, usable and even more gullible. You will know bugger all about practical IT or the architecture, evolution and application of information technology and data, and vendors will love you for it, for you will be just an extension of their idea of increasing the profit rate.
This is how IT business has become the refuge of liars, cheats, pimps and the chronically dopey, and this is why Java and Hadoop have become the ultimate expression in programming and data. It’s a geeky Greek tragedy being played out as we speak. O tempora, o morons.
But it isn’t just about Java and Hadoop. Everything the Templars of Java touch turns to dreck. Whether we are looking at aberrations and failures in rapid joint application development, end user computing, database design (refractor this, dimwits!), or solutions and domain architecture, and more, the dead cold hand of the Java Mafia is invariably behind it.
And now, to top it all off, the miserable Templars of Java want to take over and displace Bill’s Data Warehousing. You couldn’t make it up.
So, who will save the IT world from the evil doers?
To paraphrase Homer Simpson: I’m not normally a praying man, but if you’re up there, please save us, Wonderwoman.
Thank you so much for reading.
