Data Warehousing means monolithic and siloed teams?

“Great things in business are never done by one person;

they’re done by a team of people.”

Steve Jobs

Martyn Richard Jones, Tours, 4th October 2024

Narrator: There is a widespread belief amongst the know-it-all crowd that data warehousing and business intelligence necessarily mean monolithic and siloed teams. And that the only way of moving away from such team organisations is to kill off data warehousing. But is this really a rational, coherent, and cohesive approach, as some people say it is? Or is it destructive stupidity born out of conceit, ignorance, and arrogance?

Dud: Consider this, Pete. There is a widespread view that data warehousing must have massively problematic monolithic and siloed teams.

Pete: This is not a problem with data warehousing, analytics, reporting or business intelligence, Dud. This is a problem of how IT companies and their customers reframed the idea of data warehousing development and infrastructure. All the top players in IT did this. Everything with a data warehouse label on it at places like HP in 1998 got chucked into the infrastructure bucket, and much valuable knowledge and experience was lost. At times, CEOs in technology corporations are the dumbest of animals.

Dud: Blimey, talk about shooting yourself in the foot and then the head. That’s got to hurt!

Pete: Before IT got its grubby mitts on data warehousing, the initiatives in this solution space were highly dynamic and focused and consisted of small multi-disciplinary teams working in close conjunction with the business. The data warehouses would be populated with fully integrated data, and the data marts would be built on-demand in chunks that were small enough to be doable in an iteration and large enough to have business significance. By working this way, we achieved 100% customer satisfaction and 100% customer engagement. It was a revolution in terms of meeting user demands and needs. It was one of the most satisfying periods in business and IT.

Dud: So, what you are saying then, Pete, is that we should not lay the blame for the demise of this approach on data warehousing but blame it on the greedy and ignorant vendors and the big “systems integration” providers focusing on revenue funnels, software licensing margins and their consultant utilisation figures.

Pete: Yes, Dud. Credit where credit is due. Dud, when data warehousing first came to Europe in the early nineties, the IT departments wanted nothing to do with it. For them, it was a fad that would die as soon as it was born. They did not understand it, and they did not want to understand it. They were so dismissive, frivolous, and aloof!

So, these projects were entirely business-driven initiatives with minimal IT support. For a while, we kept the data warehousing and business intelligence servers by the side of the desk of our project manager. And the entire project team sat in the same room. Mind you, the only fundamental BI tool we had was Business Objects. Everything else we had was hand-built. I was responsible for the ETL framework and its implementation and the mechanism for ensuring that everything kept running even when there were failures. I also acted as the data architect.

Dud: Yes, Pete, I hear you. I also hear that the data mesh folk still claim that the traditional approach to data management is to have a centralised data team that manages organization-wide business needs to access, manage, and deliver data to different stakeholders. Is that approach really so traditional?

Pete: Well, yes. Sometimes it is like that. Many times, it is not. One swallow does not make a spring, Dud. Nor one sunny day. We are clearly dealing with folks who only have, at best, a passing acquaintance with professional data architecture and management. When it comes to data, they are not precisely the worldly wise.

Dud: So, what are your more general views on team organisation, seen from both a business and technical perspective, especially regarding data management?

Pete: What we need to understand, Dud, is that there are several modes, levels of abstraction, dependencies, and transversality when it comes to data architecture and management, especially in large and complex organisations.

Dud: I see. In an ideal world, how would you organise that?

Pete: I would start at the top, Dud. But first, I would want to define what we want to do concerning data, information, and structured intellectual capital. From my perspective, we usually want to protect, curate, and legitimately leverage our digital assets. First, I would have an umbrella organisation responsible at the highest level for all digital assets. Subordinate to this organisation, I would have enterprise teams with responsibilities for governance and quality for applications, interfaces, and AI governance – the first team and the second team would be dedicated to data, information and knowledge management and the promotion of the use of assets in this governance space; and a third team with responsibility for contract, asset supply agreements, service level agreements and other aspects of agreement governance.

Dud: Blimey, but won’t that upset the people who think that data governance should be the queen of the tribes?

Pete: No doubt, Dud. But progress also means some things come and others go, and change can impact anything. We might admire the canals, Dud, but few people use them these days to move things around the country.

Dud: Give me an example of a drill down into one of these three teams.

Pete: Let us talk about digital asset governance’s data and information aspect. The second team that I mentioned previously. They should be responsible and accountable for enterprise data and information models at the highest level. There should be responsibilities for business line models, such as information and data models for corporate real estate, manufacturing, sustainability, and purchase-to-pay. It should be emphasised that these teams will have three major modes of working; strategy, project, and operational.

Dud: Interesting, Pete. And what team configuration options do we have at our disposal? 

Pete: Well, it isn’t constrained to siloed teams or data mesh siloed teams, Dud. There are quite a few forms of teams found in the data space. For example, functional teams are the option that the data mesh people take aim at, and not in a friendly way. Regarding building the data warehouse, I do not think functional teams should be at the heart of the ongoing development. However, functional teams that provide API interconnection services could be unbelievably valuable function teams. I would also go for a cross-functional team for each discrete iteration of the data warehouse. However, I would also want a matrix team to support cross-functional teams at a higher level of abstraction. Of course, other team options exist; some are appropriate to the data sphere, and others are not. I am thinking of teams such as project teams; self-managed teams, especially when doing Agile properly; virtual teams, local, regional, and global; networked teams; hierarchical teams; special project teams, taskforces, and committees; and communities of practice and birds of a feather.

Dud: That is remarkably interesting, Pete. I can see you have your thumb on the pulse.

Pete: It is nothing, Dud. After so many decades, it has come down to second nature.

Dud: Cool! How can we be at the forefront without spilling blood?

Pete: Here is an anecdote for you, Dud. According to NASA, an estimated 400,000 people (about half the population of Delaware) were involved in making the Apollo 11 moon landing a reality. This included teams of scientists, engineers, strategists, and technicians, many of whom had not worked in the aerospace business before. It is said that to form a more cohesive team, the astronauts worked with many of these groups, creating the human connection that is the lifeblood of any team. And this was in 1969. Well before, scrum, Agile and Agile to Scale.

Dud: But did the moon landing really happen, Pete?

Pete: Yes, Dudley. It really happened.