The State of Data Quality – and How Developers Can Improve Data Efficacy
April 21, 2020

Rachel Roumeliotis
O'Reilly Media

Organizations are giving increasing attention to data quality. This doesn't come as a surprise — data is a valuable commodity that is quickly proving its worth in a growing number of cloud-native applications to make decisions about product development, how to market to customers, and more. It's not uncommon to see developers leverage insights stemming from customer data or use sensor data from a variety of connected IoT devices.

However, for data to be beneficial, it needs to be high quality. Without high quality data, organizations run the risk of making costly decisions, or missing opportunities that make them fall behind their competitors.

O'Reilly recently surveyed more than 1,900 practitioners who work with data and/or code and the people who directly manage them to take a look at the quality of the data organizations are using to power their analytics and decision-making. The research found that organizations are concerned about data quality — and at the same time are uncertain about how best to address those concerns.

Organizations' Top Data Quality Issue: Too Many Data Sources

By a wide margin, respondents cited the sheer preponderance of data sources as the single most common data quality issue. More than 60% said that they had to deal with too many data sources and inconsistent data, followed by 50% reporting disorganized data stores and lack of metadata and 47% reporting poor data quality controls at data entry.

There's good and bad in this. Unfortunately, reducing the number of data sources is hard. This has been an issue that has been prevalent since the 1990s and, with the rise of self-service data analysis tools, it's hard to say when — if at all — we'll be rid of multiple, redundant, and sometimes inconsistent copies of data sets.

On the flip side, technological progress has been made on front-end tools that generate metadata and capture provenance and lineage tracking — which are essential for diagnosing and resolving data quality issues. This, in addition to further education on data quality, data governance, and general data literacy, can help alleviate organizations' top concern — especially since only 20% of survey respondents reported that their organizations publish information about data provenance or data lineage.

Organizations aren't dealing with only one data quality issue, however. In addition to a deluge of data sources, a majority of respondents reported that they deal with either three or four data quality issues at the same time. Other common data quality issues include poor data quality from third-party sources, missing data, and too few resources to address data quality issues.

Improving Data Quality to Build Better Applications

While organizations have little control over third-party data — and missing data will always be something we'll need to grapple with — there are practical steps organizations and their developers can take as they embark on new projects to improve the quality of their data.

Obtain C-suite buy-in and support: Data quality has a tendency to be regarded as more of a people-and-process-laden problem than a technological one. Executives have a clear perspective on the impact data quality can have on business operations and strategy, and have the authority to spearhead data quality initiatives or help kick-start a data quality center of excellence.

Pick projects with clear business value: When it comes to commencing any project, it's important to pursue those that add significant business value to a new or existing business process. The same is true for data quality. Because data conditioning is not cheap, high costs should compel developers to take an ROI-based approach to how and where to deploy their data conditioning resources. This includes deciding what is not worth addressing.

Invest in AI: Artificial intelligence helps simplify and automate tasks, providing organizations with additional resources to address issues involved in discovering, profiling and indexing data. In fact, almost half (48%) of respondents reported that they use data analysis, machine learning, or artificial intelligence tools to address data quality issues.

While we might not be able to solve all of our data quality issues, it's important to implement practices that incorporate data quality as part of the development process. By placing importance on data quality, developers can make better choices about the applications they build — and ultimately set up their organizations for success.

Rachel Roumeliotis is VP of Content Strategy at O'Reilly Media
Share this

Industry News

May 21, 2020

Exadel announced the Grand Prize winner of the “Appery.io COVID-19 Virtual Hackathon.”

May 21, 2020

CloudBees announced significant advances for its Software Delivery Management (SDM) platform – integrations with additional continuous integration and continuous delivery (CI/CD) engines, including Google Cloud Build and Tekton, and extension of the availability of CloudBees’ SDM Preview Program.

May 21, 2020

OutSystems is announcing over 70 development accelerators that ensure web and mobile applications created on the OutSystems low-code development platform can comply with the highest accessibility standards and regulations.

May 20, 2020

Styra announced that Styra Declarative Authorization Service (DAS) now supports microservices and extends context-based authorization to the service mesh.

May 20, 2020

Optimizely announced that its free feature flagging plan for development teams, Rollouts, now also includes A/B testing and feature configuration.

May 20, 2020

StackRox announced new runtime security features in the latest release of the StackRox Kubernetes Security Platform.

May 19, 2020

Docker has partnered with Snyk to deliver the first, native vulnerability scanning of container images in Docker.

May 19, 2020

Rancher Labs announced the launch of Rancher Academy.

May 19, 2020

Datical, a provider of database release automation solutions, has rebranded and will be conducting operations under the name Liquibase.

May 18, 2020

D2iQ introduced the D2iQ Shortcut to Success promotion. The cost-effective package of technology, training, professional services and support ensures successful Day 2 production operations for Kubernetes at a price point that makes it affordable for companies of all sizes to get started.

May 18, 2020

Altran announced the release of a new tool available on GitHub that predicts the likelihood of bugs in source code created by developers early in the software development process.

May 18, 2020

DigitalOcean closed a $50 million Series C funding round led by Access Industries, with participation from Andreessen Horowitz.

May 14, 2020

Venafi announced a definitive agreement to acquire Jetstack, a provider of open source machine identity protection software for Kubernetes and cloud native ecosystems.

May 14, 2020

SonarSource announced the acquisition of RIPS Technologies, a German startup from Bochum known for its leading code security analyzers.

May 14, 2020

IT Revolution announced The Idealcast — a new podcast series to share insights and solutions for how organizations can thrive during the age of digital disruption.