As organizations deal with managing ever more data, the need to automate data management becomes clear.
Last week erwin issued its 2020 State of Data Governance and Automation (DGA) Report. The research from the survey suggests that companies are still grappling with the challenges of data governance — challenges that will only get worse as they collect more data.
One piece of the research that stuck with me is that 70% of respondents spend 10 or more hours per week on data-related activities. Searching for data was the biggest time-sinking culprit followed by managing, analyzing and preparing data. Protecting data came in last place.
In 2018, IDC predicted that the collective sum of the world’s data would grow from 33 zettabytes (ZB) to 175 ZB by 2025. That’s a lot of data to manage!
Here’s the thing: you do not need to waste precious time, energy and resources to search, manage, analyze, prepare or protect data manually. And unless your data is well-governed, downstream data analysts and data scientists will not be able to generate significant value from it. So, what should you do? The answer is clear. It’s time to automate data management. But how?
How to Automate Data Management
Here are our eight recommendations for how to transition from manual to automated data management:
- 1) Put Data Quality First:
Automating and matching business terms with data assets and documenting lineage down to the column level are critical to good decision making. - 2) Don’t Ignore Data Lineage Complexity:
It’s a risky endeavor to support data lineage using a manual approach, and businesses that attempt it that way will find that it’s not sustainable given data’s constant movement from one place to another via multiple routes- and doing it correctly down to the column level. - 3) Automate Code Generation:
Mapping data elements to their sources within a single repository to determine data lineage and harmonize data integration across platforms reduces the need for specialized, technical resources with knowledge of ETL and database procedural code. - 4) Use Integrated Impact Analysis to Automate Data Due Diligence:
This helps IT deliver operational intelligence to the business. Business users benefit from automating impact analysis to better examine value and prioritize individual data sets. - 5) Catalog Data:
Catalog data using a solution with a broad set of metadata connectors so all data sources can be leveraged. - 6) Stress Data Literacy Across the Organization:
There’s a clear connection to data literacy because of its foundation in business glossaries and socializing data so that all stakeholders can view and understand it within the context of their roles. - 7) Make Automation Standard Practice:
Too many companies are still living in a world where data governance is a high-level mandate and not a practically implemented one. - 8) Create a Solid Data Governance Strategy:
Craft your data governance strategy before making any investments. Gather multiple stakeholders—both business and IT—with multiple viewpoints to discover where their needs mesh and where they diverge and what represents the greatest pain points to the business.
The Benefits of Data Management Automation
With data management automation, data professionals can meet their organization’s data needs at a fraction of the cost of the traditional, manual way.
Some of the benefits of data management automation are:
- Centralized and standardized code management with all automation templates stored in a governed repository
- Better quality code and minimized rework
- Business-driven data movement and transformation specifications
- Superior data movement job designs based on best practices
- Greater agility and faster time-to-value in data preparation, deployment and governance
- Cross-platform support of scripting languages and data movement technologies
For a deeper dive on how to automate data management and to view the full research, download a copy of erwin’s 2020 State of Data Governance and Automation report.
