View from the top
Business hierarchies have typically started with a top-down method, and data offices have been no different. The top-down method in data provides a large focus on data control and is heavily centralised with a small team of professionals running the show. It is usually the case with this model that governance, compliance and data modelling are the priorities, with data being disseminated across the business at a later stage.
From the get-go, the top-down method has a huge scalability problem and will frequently find itself overwhelmed with demands for data sets from all areas of the business when a certain level of maturity and literacy is reached. The data providers in this model are often combined with IT departments and there is a clear dividing line between the providers and the data consumers elsewhere in the organisation. This did not use to be as big of an issue as it is today as there was less data being consumed and data literacy levels were much lower outside of the data office, but with contemporary technological developments and easier access to training, the system can easily become clogged or face multiple bottlenecks. DataIQ members frequently highlight speed of data access and quality of data as their key bottlenecks and this framework, while effective at securing high quality data, can often be a slow process.
Looking up
On the flip side, the bottom-up approach to this type of framework provides far more agility for data management and data access. Usually, bottom-up approaches the framework by focusing on raw data over modelling and governance. From here, once the data is ingested, different forms of structures can be created and implemented to provide quality, security, compliance and a host of other actions.
Bottom-up found mainstream success with the advent of big data and the boom of data processes as it is more scalable than top-down which made it a quick win for large multinational organisations and introduced the concept of decentralised data.
As the data being ingested in a bottom-up structure can be entered by anyone and the governance is not implemented until further in the process, it does mean that it is harder to establish control, but with a high enough level of literacy and understanding of data culture the quality of the data being sourced can remain high. Of course, a lack (or perceived lack) of control can be a frightening thing for stakeholders to accept as issues such as data governance can become a regulatory risk. Bottom-up approaches often contribute to increasing data quality problems, meaning decision making can be less reliable. This can quickly spiral out of control in rapidly evolving environments without a robust governance framework, but with suitable data management, training and a well-understood data culture these concerns can be mitigated.
Utilising the mesh
Data mesh is the latest framework to undergo rigorous testing across multiple global organisations and it has shown some promising results for specific user cases with an extreme approach to decentralisation. It should be noted that data mesh still requires rigorous governance and quality frameworks, but these can differ from node to node as long as the interfacing is well understood and there is a strong level of connectivity and transparency. The main selling point of the data mesh framework is that it encourages thinking of data with a decentralised approach from the get-go, so there is no single data team to become a bottleneck, which also drastically increases the speed in which data can be used. Instead of specific teams holding and running data, data mesh focuses on decentralised ownership and treating data as a product. This approach makes data more reusable within the organisation and avoids the issue of teams having to source their own data for answering their needs, which means a reduction in shadow data departments.
However, data mesh is not a system that is suitable for smaller businesses that plan on remaining small, it is designed for larger organisations with multiple divisions, territories and business units. It is a good alternative for large businesses that are struggling with a central data team that cannot manage pace of growth the business desires. There have been businesses that have implemented the data mesh approach since inception and are now scaling successfully. Data mesh also provides businesses with the chance to embrace decentralised data ownership boundaries early on in a data journey which is key for a maturing data culture.
Which one works?
Simply put, there is no one perfect structure: the diversity of businesses, the ways in which data is used and the future ambitions of each business are as unique as fingerprints. Every business will need to consciously identify and implement a framework that is geared toward its specific needs. This will naturally involve investment in time and analysis, perhaps also financial investment too, but the return on investment for creating the most suitable framework will be rapid. In most cases, the framework typically involves a form of centralised strategic and governance framework with decentralised, federated operational hubs. One of the major benefits of these structures is that they can be moulded into a form that sits between the extremes of a fully centralised system and the decentralised nodes of a mesh to operate efficiently for specific business needs.
