Tag: data automation

Categories
Data Governance erwin Expert Blog

Automating Data Governance

Automating Data Governance

Automating data governance is key to addressing the exponentially growing volume and variety of data.

erwin CMO Mariann McDonagh recounts erwin’s vision to automate everything from day 1 of erwin Insights 2020.

Data readiness is everything. Whether driving digital experiences, mapping customer journeys, enhancing digital operations, developing digital innovations, finding new ways to interact with customers, or building digital ecosystems or marketplaces – all of this digital transformation is powered by data.

In a COVID and post-COVID world, organizations need to radically change as we look to reimagine business models and reform the way we approach almost everything.

The State of Data Automation

Data readiness depends on automation to create the data pipeline. Earlier this year, erwin conducted a research project in partnership with Dataversity, the 2020 State of Data Governance and Automation.

We asked participants to “talk to us about data value chain bottlenecks.” They told us their number one challenge is documenting complete data lineage (62%), followed by understanding the quality of the data source (58%).

Two other significant bottlenecks are finding, identifying and harvesting data (55%) curating data assets with business content for context and semantics (52%). Every item mentioned here are recurring themes we hear from our customers in terms of what led them to erwin.

We also looked at data preparation, governance and intelligence to see where organizations might be getting stuck and spending lots of time. We found that project length, slow delivery time, is one of the biggest inhibitors. Data quality and accuracy are recurring themes as well.

Reliance on developers and technical resources is another barrier to productivity. Even with data scientists in the front office, the lack of people in the back office to harvest and prepare the data means  time to value is prolonged.

Last but not least, we looked at the amount of time spent on data activities. The great news is that most organizations spend more than 10 hours a week on data-related activities. But the problem is that not enough of that time is spent on analysis because of being stuck in data prep.

IDC talks about this reverse 80/20 rule: 80% of time and effort is spent on data preparation, with only 20% focused on data analysis. This means 80% of your time is left on the cutting-room floor and can’t be used to drive your business forward.

2020 Data Governance and Automation Report

Data Automation Adds Value

Automating data operations adds a lot of value by making a solution more effective and more powerful. Consider a smart home’s thermostat, smoke detectors, lights, doorbell, etc. You have centralized access and control – from anywhere.

At erwin, our goal is to automate the entire data governance journey, whether top down or bottom up. We’re on a mission to automate all the tasks data stewards typically perform so they spend less time building and populating the data governance framework and more time using the framework to realize value and ROI.

Automation also ensures that the data governance framework is always up to date and never stale. Because without current and accurate data, a data governance initiative will fall apart.

Here are some ways erwin adds value by automating the data governance journey:

  • Metadata ingestion into the erwin Data Intelligence Suite (erwin DI) through our standard data connectors. And you can schedule metadata scans to ensure it’s always refreshed and up to date.
  • erwin Smart Data Connectors address data in motion, how it travels and transforms across the enterprise. These custom software solutions document all the traversing and transformations of data and populate the erwin DI’s Metadata Manager with the technical metadata. erwin Smart Data Connectors also document ETL scripts work with the tool of your choice.
  • erwin Lineage Analyzer puts everything together in an easy-to-understand format, making it easy for both business and technical users to visualize how data is traversing the enterprise, how it is getting transformed and the different hops it is taking along the way.
  • erwin DM Connect for DI makes it easy for metadata to be ingested from erwin Data Modeler to erwin DI. erwin DM customers can take advantage of all the rich metadata created and stored in their erwin data models. With just a couple of clicks, some or all data models can be configured and pushed erwin DI’s Metadata Manager.

The automation and integration of erwin DM and erwin DI ensures that your data models are always updated and uploaded, providing a single source of truth for your data governance journey.

This is part one of a two-part series on how erwin is automating data governance. Learn more by watching this session from erwin Insights 2020, which now is available on demand.

erwin Insights 2020

Categories
erwin Expert Blog

8 Tips to Automate Data Management

As organizations deal with managing ever more data, the need to automate data management becomes clear.

Last week erwin issued its 2020 State of Data Governance and Automation (DGA) Report. The research from the survey suggests that companies are still grappling with the challenges of data governance — challenges that will only get worse as they collect more data.

One piece of the research that stuck with me is that 70% of respondents spend 10 or more hours per week on data-related activities. Searching for data was the biggest time-sinking culprit followed by managing, analyzing and preparing data. Protecting data came in last place.

In 2018, IDC predicted that the collective sum of the world’s data would grow from 33 zettabytes (ZB) to 175 ZB by 2025. That’s a lot of data to manage!

Here’s the thing: you do not need to waste precious time, energy and resources to search, manage, analyze, prepare or protect data manually. And unless your data is well-governed, downstream data analysts and data scientists will not be able to generate significant value from it. So, what should you do?  The answer is clear. It’s time to automate data management. But how?

Automate Data Management

How to Automate Data Management

Here are our eight recommendations for how to transition from manual to automated data management:

  • 1) Put Data Quality First:
    Automating and matching business terms with data assets and documenting lineage down to the column level are critical to good decision making.
  • 2) Don’t Ignore Data Lineage Complexity:
    It’s a risky endeavor to support data lineage using a manual approach, and businesses that attempt it that way will find that it’s not sustainable given data’s constant movement from one place to another via multiple routes- and doing it correctly down to the column level.
  • 3) Automate Code Generation:
    Mapping data elements to their sources within a single repository to determine data lineage and harmonize data integration across platforms reduces the need for specialized, technical resources with knowledge of ETL and database procedural code.
  • 4) Use Integrated Impact Analysis to Automate Data Due Diligence:
    This helps IT deliver operational intelligence to the business. Business users benefit from automating impact analysis to better examine value and prioritize individual data sets.
  • 5) Catalog Data:
    Catalog data using a solution with a broad set of metadata connectors so all data sources can be leveraged.
  • 6) Stress Data Literacy Across the Organization:
    There’s a clear connection to data literacy because of its foundation in business glossaries and socializing data so that all stakeholders can view and understand it within the context of their roles.
  • 7) Make Automation Standard Practice:
    Too many companies are still living in a world where data governance is a high-level mandate and not a practically implemented one.
  • 8) Create a Solid Data Governance Strategy:
    Craft your data governance strategy before making any investments. Gather multiple stakeholders—both business and IT—with multiple viewpoints to discover where their needs mesh and where they diverge and what represents the greatest pain points to the business. 

The Benefits of Data Management Automation

With data management automation, data professionals can meet their organization’s data needs at a fraction of the cost of the traditional, manual way.

Some of the benefits of data management automation are:

  • Centralized and standardized code management with all automation templates stored in a governed repository
  • Better quality code and minimized rework
  • Business-driven data movement and transformation specifications
  • Superior data movement job designs based on best practices
  • Greater agility and faster time-to-value in data preparation, deployment and governance
  • Cross-platform support of scripting languages and data movement technologies

For a deeper dive on how to automate data management and to view the full research, download a copy of erwin’s 2020 State of Data Governance and Automation report.

2020 Data Governance and Automation Report

Categories
erwin Expert Blog

How Metadata Makes Data Meaningful

Metadata is an important part of data governance, and as a result, most nascent data governance programs are rife with project plans for assessing and documenting metadata. But in many scenarios, it seems that the underlying driver of metadata collection projects is that it’s just something you do for data governance.

So most early-stage data governance managers kick off a series of projects to profile data, make inferences about data element structure and format, and store the presumptive metadata in some metadata repository. But are these rampant and often uncontrolled projects to collect metadata properly motivated?

There is rarely a clear directive about how metadata is used. Therefore prior to launching metadata collection tasks, it is important to specifically direct how the knowledge embedded within the corporate metadata should be used.

Managing metadata should not be a sub-goal of data governance. Today, metadata is the heart of enterprise data management and governance/ intelligence efforts and should have a clear strategy – rather than just something you do.

metadata data governance

What Is Metadata?

Quite simply, metadata is data about data. It’s generated every time data is captured at a source, accessed by users, moved through an organization, integrated or augmented with other data from other sources, profiled, cleansed and analyzed. Metadata is valuable because it provides information about the attributes of data elements that can be used to guide strategic and operational decision-making. It answers these important questions:

  • What data do we have?
  • Where did it come from?
  • Where is it now?
  • How has it changed since it was originally created or captured?
  • Who is authorized to use it and how?
  • Is it sensitive or are there any risks associated with it?

The Role of Metadata in Data Governance

Organizations don’t know what they don’t know, and this problem is only getting worse. As data continues to proliferate, so does the need for data and analytics initiatives to make sense of it all. Here are some benefits of metadata management for data governance use cases:

  • Better Data Quality: Data issues and inconsistencies within integrated data sources or targets are identified in real time to improve overall data quality by increasing time to insights and/or repair.
  • Quicker Project Delivery: Accelerate Big Data deployments, Data Vaults, data warehouse modernization, cloud migration, etc., by up to 70 percent.
  • Faster Speed to Insights: Reverse the current 80/20 rule that keeps high-paid knowledge workers too busy finding, understanding and resolving errors or inconsistencies to actually analyze source data.
  • Greater Productivity & Reduced Costs: Being able to rely on automated and repeatable metadata management processes results in greater productivity. Some erwin customers report productivity gains of 85+% for coding, 70+% for metadata discovery, up to 50% for data design, up to 70% for data conversion, and up to 80% for data mapping.
  • Regulatory Compliance: Regulations such as GDPR, HIPAA, PII, BCBS and CCPA have data privacy and security mandates, so sensitive data needs to be tagged, its lineage documented, and its flows depicted for traceability.
  • Digital Transformation: Knowing what data exists and its value potential promotes digital transformation by improving digital experiences, enhancing digital operations, driving digital innovation and building digital ecosystems.
  • Enterprise Collaboration: With the business driving alignment between data governance and strategic enterprise goals and IT handling the technical mechanics of data management, the door opens to finding, trusting and using data to effectively meet organizational objectives.

Giving Metadata Meaning

So how do you give metadata meaning? While this sounds like a deep philosophical question, the reality is the right tools can make all the difference.

erwin Data Intelligence (erwin DI) combines data management and data governance processes in an automated flow.

It’s unique in its ability to automatically harvest, transform and feed metadata from a wide array of data sources, operational processes, business applications and data models into a central data catalog and then make it accessible and understandable within the context of role-based views.

erwin DI sits on a common metamodel that is open, extensible and comes with a full set of APIs. A comprehensive list of erwin-owned standard data connectors are included for automated harvesting, refreshing and version-controlled metadata management. Optional erwin Smart Data Connectors reverse-engineer ETL code of all types and connect bi-directionally with reporting and other ecosystem tools. These connectors offer the fastest and most accurate path to data lineage, impact analysis and other detailed graphical relationships.

Additionally, erwin DI is part of the larger erwin EDGE platform that integrates data modeling, enterprise architecture, business process modeling, data cataloging and data literacy. We know our customers need an active metadata-driven approach to:

  • Understand their business, technology and data architectures and the relationships between them
  • Create an automate a curated enterprise data catalog, complete with physical assets, data models, data movement, data quality and on-demand lineage
  • Activate their metadata to drive agile and well-governed data preparation with integrated business glossaries and data dictionaries that provide business context for stakeholder data literacy

erwin was named a Leader in Gartner’s “2019 Magic Quadrant for Metadata Management Solutions.”

Click here to get a free copy of the report.

Click here to request a demo of erwin DI.

Gartner Magic Quadrant Metadata Management

 

Categories
erwin Expert Blog

Metadata Management, Data Governance and Automation

Can the 80/20 Rule Be Reversed?

erwin released its State of Data Governance Report in February 2018, just a few months before the General Data Protection Regulation (GDPR) took effect.

This research showed that the majority of responding organizations weren’t actually prepared for GDPR, nor did they have the understanding, executive support and budget for data governance – although they recognized the importance of it.

Of course, data governance has evolved with astonishing speed, both in response to data privacy and security regulations and because organizations see the potential for using it to accomplish other organizational objectives.

But many of the world’s top brands still seem to be challenged in implementing and sustaining effective data governance programs (hello, Facebook).

We wonder why.

Too Much Time, Too Few Insights

According to IDC’s “Data Intelligence in Context” Technology Spotlight sponsored by erwin, “professionals who work with data spend 80 percent of their time looking for and preparing data and only 20 percent of their time on analytics.”

Specifically, 80 percent of data professionals’ time is spent on data discovery, preparation and protection, and only 20 percent on analysis leading to insights.

In most companies, an incredible amount of data flows from multiple sources in a variety of formats and is constantly being moved and federated across a changing system landscape.

Often these enterprises are heavily regulated, so they need a well-defined data integration model that will help avoid data discrepancies and remove barriers to enterprise business intelligence and other meaningful use.

IT teams need the ability to smoothly generate hundreds of mappings and ETL jobs. They need their data mappings to fall under governance and audit controls, with instant access to dynamic impact analysis and data lineage.

But most organizations, especially those competing in the digital economy, don’t have enough time or money for data management using manual processes. Outsourcing is also expensive, with inevitable delays because these vendors are dependent on manual processes too.

The Role of Data Automation

Data governance maturity includes the ability to rely on automated and repeatable processes.

For example, automatically importing mappings from developers’ Excel sheets, flat files, Access and ETL tools into a comprehensive mappings inventory, complete with automatically generated and meaningful documentation of the mappings, is a powerful way to support governance while providing real insight into data movement — for data lineage and impact analysis — without interrupting system developers’ normal work methods.

GDPR compliance, for instance, requires a business to discover source-to-target mappings with all accompanying transactions, such as what business rules in the repository are applied to it, to comply with audits.

When data movement has been tracked and version-controlled, it’s possible to conduct data archeology — that is, reverse-engineering code from existing XML within the ETL layer — to uncover what has happened in the past and incorporating it into a mapping manager for fast and accurate recovery.

With automation, data professionals can meet the above needs at a fraction of the cost of the traditional, manual way. To summarize, just some of the benefits of data automation are:

• Centralized and standardized code management with all automation templates stored in a governed repository
• Better quality code and minimized rework
• Business-driven data movement and transformation specifications
• Superior data movement job designs based on best practices
• Greater agility and faster time-to-value in data preparation, deployment and governance
• Cross-platform support of scripting languages and data movement technologies

One global pharmaceutical giant reduced costs by 70 percent and generated 95 percent of production code with “zero touch.” With automation, the company improved the time to business value and significantly reduced the costly re-work associated with error-prone manual processes.

Gartner Magic Quadrant Metadata Management

Help Us Help You by Taking a Brief Survey

With 2020 just around the corner and another data regulation about to take effect, the California Consumer Privacy Act (CCPA), we’re working with Dataversity on another research project.

And this time, you guessed it – we’re focusing on data automation and how it could impact metadata management and data governance.

We would appreciate your input and will release the findings in January 2020.

Click here to take the brief survey

Categories
erwin Expert Blog

Four Use Cases Proving the Benefits of Metadata-Driven Automation

Organization’s cannot hope to make the most out of a data-driven strategy, without at least some degree of metadata-driven automation.

The volume and variety of data has snowballed, and so has its velocity. As such, traditional – and mostly manual – processes associated with data management and data governance have broken down. They are time-consuming and prone to human error, making compliance, innovation and transformation initiatives more complicated, which is less than ideal in the information age.

So it’s safe to say that organizations can’t reap the rewards of their data without automation.

Data scientists and other data professionals can spend up to 80 percent of their time bogged down trying to understand source data or addressing errors and inconsistencies.

That’s time needed and better used for data analysis.

By implementing metadata-driven automation, organizations across industry can unleash the talents of their highly skilled, well paid data pros to focus on finding the goods: actionable insights that will fuel the business.

Metadata-Driven Automation

Metadata-Driven Automation in the BFSI Industry

The banking, financial services and insurance industry typically deals with higher data velocity and tighter regulations than most. This bureaucracy is rife with data management bottlenecks.

These bottlenecks are only made worse when organizations attempt to get by with systems and tools that are not purpose-built.

For example, manually managing data mappings for the enterprise data warehouse via MS Excel spreadsheets had become cumbersome and unsustainable for one BSFI company.

After embracing metadata-driven automation and custom code automation templates, it saved hundreds of thousands of dollars in code generation and development costs and achieved more work in less time with fewer resources. ROI on the automation solutions was realized within the first year.

Metadata-Driven Automation in the Pharmaceutical Industry

Despite its shortcomings, the Excel spreadsheet method for managing data mappings is common within many industries.

But with the amount of data organizations need to process in today’s business climate, this manual approach makes change management and determining end-to-end lineage a significant and time-consuming challenge.

One global pharmaceutical giant headquartered in the United States experienced such issues until it adopted metadata-driven automation. Then the pharma company was able to scan in all source and target system metadata and maintain it within a single repository. Users now view end-to-end data lineage from the source layer to the reporting layer within seconds.

On the whole, the implementation resulted in extraordinary time savings and a total cost reduction of 60 percent.

Metadata-Driven Automation in the Insurance Industry

Insurance is another industry that has to cope with high data velocity and stringent data regulations. Plus many organizations in this sector find that they’ve outgrown their systems.

For example, an insurance company using a CDMA product to centralize data mappings is probably missing certain critical features, such as versioning, impact analysis and lineage, which adds to costs, times to market and errors.

By adopting metadata-driven automation, organizations can standardize the pre-ETL data mapping process and better manage data integration through the change and release process. As a result, both internal data mapping and cross functional teams now have easy and fast web-based access to data mappings and valuable information like impact analysis and lineage.

Here is the story of a business that adopted such an approach and achieved operational excellence and a delivery time reduction by 80 percent, as well as achieving ROI within 12 months.

Metadata-Driven Automation for a Non-Profit

Another common issue cited by organizations using manual data mapping is ballooning complexity and subsequent confusion.

Any organization expanding its data-driven focus without sufficiently maturing data management initiative(s) will experience this at some point.

One of the world’s largest humanitarian organizations, with millions of members and volunteers operating all over the world, was confronted with this exact issue.

It recognized the need for a solution to standardize the pre-ETL data mapping process to make data integration more efficient and cost-effective.

With metadata-driven automation, the organization would be able to scan and store metadata and data dictionaries in a central repository, as well as manage the business definitions and data dictionary for legacy systems contributing data to the enterprise data warehouse.

By adopting such an approach, the organization realized time savings across all IT development and cross-functional testing teams. Additionally, they were able to more easily manage mappings, code sets, reference data and data validation rules.

Again, ROI was achieved within a year.

A Universal Solution for Metadata-Driven Automation

Metadata-driven automation is a capability any organization can benefit from – regardless of industry, as demonstrated by the various real-world use cases chronicled here.

The erwin Automation Framework is a key component of the erwin EDGE platform for comprehensive data management and data governance.

With it, data professionals realize these industry-agnostic benefits:

  • Centralized and standardized code management with all automation templates stored in a governed repository
  • Better quality code and minimized rework
  • Business-driven data movement and transformation specifications
  • Superior data movement job designs based on best practices
  • Greater agility and faster time-to-value in data preparation, deployment and governance
  • Cross-platform support of scripting languages and data movement technologies

Learn more about metadata-driven automation as it relates to data preparation and enterprise data mapping.

Join one our weekly erwin Mapping Manager demos.

Automate Data Mapping