Data warehousing eschews iterative development?

Martyn Jones, Santander, 10th October 2024

Narrator: The development method for data warehousing, whether following Inmon or Kimball, has always been iterative with apparent aspects of self-service, agility, rapid development and end-user development.  

Dud: We always built data warehouses iteratively, didn’t we, Pete?

Pete: Yes, that’s right, Dud. Iterations are delivered in increments. Small enough to be quickly deliverable. Large enough to be significant to the business.

Dud: So, why do they say we don’t and didn’t?

Pete: Ignorance, Dud. Sheer ignorance. The data mesh mob and other guttersnipes and hangers push the idea that data warehousing is monolithic, maladroit, and brittle in all aspects. They do this to remove the good things they wish to destroy and replace. The only trouble is that it’s a pack of lies.

Dud: How would you define monolithic, Pete?

Pete: Many ignorant punters claim that traditional data warehouses were designed as monolithic systems where all data processing, storage, and querying functionalities reside within a single architecture. But this is to miss the point entirely. Data warehouses are not primarily about technology and infrastructure architectures. This is a lie peddled by hardware vendors and IT services companies who would otherwise not be able to eat at the data warehousing table because they are the opportunistic prostitutes of IT.

In architecture, a monolithic structure is constructed from a single material or is a solid piece that appears uniform, such as a poured concrete building or a large stone sculpture. This term also suggests a robust and solid construction that gives an impression of permanence. So, labelling data warehousing as inherently monolithic is a load of old unjustified and unjustifiable nonsense, Dud.

Dud: So, is data warehousing agile, Pete?

Pete: Agile is an approach to project management and product development that emphasises flexibility, collaboration, and customer-focused practices. Originally derived from the Agile Manifesto published in 2001, the agile methodology prioritizes people and interactions over processes and tools, working software over extensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan.

Is data warehousing agile? I would say so. From the beginning, data warehouses were built iteratively and designed to deliver business benefits quickly. Data warehouses were built with progressive and reiterative deliveries of data mart extensions that strove to provide something that could be delivered promptly and simultaneously in increments that were significant enough for the business and small enough to be delivered promptly. The balance between scope, scale and significance.

Another important aspect was that if an iteration failed, it wouldn’t be the end of the world but a temporary setback.

Dud: Interesting, Pete. So, what real advantages does data mesh have over contemporary data warehousing?

Pete: None whatsoever, Pete.

Dud: Let’s put this idiocy to bed before it gets into another sugar-rush-driven beserker episode!

Pete: Amen!