It is estimated that over 180 zettabytes of data will be produced annually by the end of 2025, and that is a lot of data that needs to be utilised by individuals and teams. To put that in perspective, linguists calculated the storage requirements for all human speech ever spoken at 42 zettabytes if digitised as 16 kHz 16-bit audio.
Utilising and handling this colossal amount of data will only be possible with improved data efficiencies that begin within the data function and radiate out into the wider business with people being supported to use data accordingly.
When it comes to business, data efficiency is only possible and maximised when the data and data teams are running at full efficiency themselves. There are several ways to address data efficiency and her are a few suggestions.
Data efficiency base level
In a nutshell, data efficiency with a focus on data refers to the effectiveness of various processes applied to data, such as storage, access, filtering, and sharing, to achieve desired outcomes within resource constraints. It involves optimising how data is managed and used to ensure that it is easily accessible, manageable, and cost-effective.
Data leaders need to assess the existing level of data efficiency throughout the business, and not just from the data team, but for all departments. When a base level has been determined, this will provide a road map to examine different areas of improvement.
Improved data efficiency leads to:
- Higher quality analytics
- Improved productivity
- Reduced costs
When it comes to utilising data for the wider organisation, the clean and effective data provided by the data team is what drives the change. For example, efficient data processes make it easier for analysts to locate and retrieve relevant information for different functions, which enhances the quality of insights they can generate.
It should also be noted that, by choosing the right storage media based on how frequently data is accessed, organisations can reduce costs and improve overall efficiency. However, this does require a minimum level of data literacy and some form of data stewardship to ensure the effective use of the tools. It is possible to limit the access different teams have to these tools, but this can slow data democratisation and culture development.
Not just tools, but people too
It is very easy for stakeholders and decision makers who lack a data background to see a new shiny tool and think that will solve the problems. It is up to the data leaders to explain thoroughly that tools are not solutions, but a part of the journey to a solution. Another part of this journey is the effective training and support of team members.
Tools are wielded by people, and people must be trained to use tools.
Data leaders need to ensure that they are accurately gauging the data literacy and data culture of the organisation and providing suitable training opportunities to address any gaps that get identified. This can be a slow process but is one of the most important parts as there must always be a person in the system that can handle and protect the data being used.
Furthermore, businesses want to expand, and this requires new talent. It is pivotal that the new talent has the provisions and support to be trained to the ever-changing minimum level of data literacy and culture that has been achieved.
Organisation for large businesses
Outside of the data team, organisation – and specifically categorisation – is key to driving efficiencies. It is hard enough to drive a data-centric culture in a small business, but when the challenge involves a large, perhaps legacy, business, the challenge requires additional thought and consideration.
Data leaders and their teams, alongside data stewards and data owners, need to make sure data is accurately and consistently categorised across internal and external data centres, virtual environments, and cloud locations. This will also help drive success for businesses utilising a federated or hybrid approach to their data structure.
By accurately cataloguing and labelling data, businesses can reduce duplication. This is a particularly important hurdle faced by businesses that stretch over numerous geographies, especially those working across multiple languages. Duplicated data is time-consuming, costly to store, and slows down access to the prime data when it is needed.
Accurate access equals efficiency
It is frequently mentioned by data teams and data leaders that chunks of data are left unleveraged, and this can happen for multiple reasons. One of the easiest ways to address this is to introduce a storage solution that provides suitable and relevant access to those that need it when it is needed. Furthermore, this will help drive data democratisation and instil a stronger data culture.
The issue is that introducing a new storage media service to better reflect the volume of access is not a plug-and-play solution and requires careful consideration and preparation. However, it is a prime example of addressing the root cause issues which will alleviate bottlenecks for the future.
An organisation that has infrequently accessed archive data on high-performance and costly solid-state hard drives is a significant waste of resources and a drain on data efficiency. Additionally, end user productivity takes a hit when the data required must be retrieved from lower-performance storage data solutions.
Data leaders should consider introducing a conceptual model that follows a hot and cold labelling system. Hot storage tiers are those considered high performance, whereas cold tier refers to cheaper and often deeper, lower quality storage solutions. The benefit of this approach is that is helps with data storytelling and can identify when data items need to be moved from one solution to another for usage reasons or insufficient storage capabilities.
There is plenty to be done by data leaders to address and improve data efficiency in 2025, and it all needs to start today. It is an ongoing journey, but each year it will get easier and faster to implement new efficiencies and gain backing for data-centric improvements.
Attend an upcoming DataIQ roundtable or masterclass to get involved with ongoing industry discussions.
