Data preparation is the essential precursor to generating analytics and insights for your business. It involves transforming the information identified in the data discovery stage (read our blog on common data discovery challenges here) to make it fit for a purpose, such as visualisation, integration or migration. However, like in most businesses with many moving parts, you may experience a range of data preparation challenges that can jeopardise the quality and validity of your analytics, leading to flawed insights and poor business decisions.
Steve Bedford, principal consultant and data analyst at Optivia helps explain the most common data preparation challenges found in modern businesses and how they can be solved.
Common data preparation challenges and their solutions
1. Establishing clear business rules
Data transformation should always be based on an established set of business rules. These rules inform what data is required for a purpose and what form it should take. However, business rules aren’t always clearly defined, even for very basic metrics such as the number of sales in a given period.
“One of the big challenges we have is that an organisation might not have a clear definition for what a sale is, for example,” explains Steve.
“I’ve worked in organisations where there are three different definitions across the organisation for what actually constitutes a sale. So if that definition is not rock solid, then you are not able to build a metric that represents data uniformly across the organisation.”
Business rules must be clearly defined for data to be valuable. While data owners or subject matter experts (SMEs) should be responsible for business rules, when dealing with a situation where they haven’t been defined reliably or in advance, a strong business analyst (BA) comes in handy.
“The BA can reach out to anyone with a vested interest in the project, data or information processes and conduct a ‘requirements gathering’ exercise. With detailed information from each stakeholder, they can then form a business rule that can be signed off by the SME or data custodian,” says Steve.
2. Having a reliable data tool
Data is only useful if it’s reliable. Even if just one of the data sources informing your metric is inaccurate, then all of your insights are compromised and you won’t have the complete picture. This is where technology often comes into play.
“If the organisation doesn’t have a good technology stack, then this can present challenges and limitations,” says Steve.
“It’s not uncommon for businesses to have basic or outdated toolsets, which make the data preparation process more difficult and time-consuming. And if you don’t have the technology stack to support the end product you want, then your data becomes less powerful.”
The solution that most companies quickly jump to is to upgrade their tech stack. While this may be reasonable, it can raise its own challenges, particularly when building a foolproof business case for the additional investment. This is one of the reasons why it’s worth checking if someone else in your business has a tool you can use before you start investing significant money in a brand new platform.
“Big companies will often act in a very siloed way, with some divisions having more advanced technology than others. It’s worth reaching out to other areas of the business to see if they’ve already got a tool that would be appropriate for your purpose,” explains Steve.
If you aren’t willing to upgrade your tech stack, then you may have to lower your expectations for your end product. A good data consultant will be upfront with you about what can be achieved with your current tech stack. But when it comes down to it, sometimes you just need better technology to achieve your goals.
How can businesses minimise these challenges?
Introduce a data governance strategy
A properly formed data governance strategy should not only include appropriate, robust technology but also the right processes, policies and standards to ensure that information is used efficiently and effectively throughout your organisation. This includes solid, well-communicated procedures to improve and maintain data quality, which benefits your data preparation efforts.
“An effective data governance strategy will set your organisation on the right path by ensuring that you’ve got your data strategy under control. That way, you can hit the ground running on new data projects without having to do a lot of catch up,” says Steve.
As with many business frameworks, data governance depends on leadership to drive a strong data agenda and strategy, building the capabilities and motivation within business teams.
Ensure business rules are clearly defined
As part of your data governance strategy, business rules should be clearly defined through consultation with multiple stakeholders to ensure agreement and buy-in across departments. Once defined, the rules should be communicated across the business and maintained by the subject matter expert (SME) or the custodian of the data if no SME exists.
“If the rules are all clearly defined before a data project and the management team wholly understands them, there should never be any unwanted surprises about how you came to a set of numbers. Having this as part of your overall governance strategy for how data is managed is essential,” explains Steve.
Implement the right technology stack
The functionality and success of an organisation’s data preparation hinges on the quality of its technology stack. Upgrading limited platforms to more holistic, integrated systems will help eliminate data silos. Similar to choosing the best candidate for the job, organisations need the best technology for data programs to run smoothly and achieve optimal results. An agile, transparent and auditable toolset will enable organisations to meet constantly evolving business requirements.
With the right technology supported by clear-cut business rules and a cohesive data governance strategy, organisations can avoid the most common data preparation challenges and achieve actionable outcomes as efficiently as possible.
Want to know how to level up your organisation’s decision-making capabilities by tapping into the power of high-quality data? Download our free eBook to find out how your organisation can improve data quality and data governance to gain more value from their business intelligence.
Optivia’s data quality services empower reliable, informed decision-making. We help you strengthen your business with robust data governance principles designed to improve data accuracy and ensure you adhere to data privacy regulations. If you need a data quality framework that aligns with your strategic goals and helps you make rapid, meaningful decisions, contact us today.