Tag: automation

Categories
Data Governance erwin Expert Blog

Are Data Governance Bottlenecks Holding You Back?

Better decision-making has now topped compliance as the primary driver of data governance. However, organizations still encounter a number of bottlenecks that may hold them back from fully realizing the value of their data in producing timely and relevant business insights.

While acknowledging that data governance is about more than risk management and regulatory compliance may indicate that companies are more confident in their data, the data governance practice is nonetheless growing in complexity because of more:

  • Data to handle, much of it unstructured
  • Sources, like IoT
  • Points of integration
  • Regulations

Without an accurate, high-quality, real-time enterprise data pipeline, it will be difficult to uncover the necessary intelligence to make optimal business decisions.

So what’s holding organizations back from fully using their data to make better, smarter business decisions?

Data Governance Bottlenecks

erwin’s 2020 State of Data Governance and Automation report, based on a survey of business and technology professionals at organizations of various sizes and across numerous industries, examined the role of automation in  data governance and intelligence  efforts.  It uncovered a number of obstacles that organizations have to overcome to improve their data operations.

The No.1 bottleneck, according to 62 percent of respondents, was documenting complete data lineage. Understanding the quality of source data is the next most serious bottleneck (58 percent); followed by finding, identifying, and harvesting data (55 percent); and curating assets with business context (52 percent).

The report revealed that all but two of the possible bottlenecks were marked by more than 50 percent of respondents. Clearly, there’s a massive need for a data governance framework to keep these obstacles from stymying enterprise innovation.

As we zeroed in on the bottlenecks of day-to-day operations, 25 percent of respondents said length of project/delivery time was the most significant challenge, followed by data quality/accuracy is next at 24 percent, time to value at 16 percent, and reliance on developer and other technical resources at 13 percent.

Are Data Governance Bottlenecks Holding You Back?

Overcoming Data Governance Bottlenecks

The 80/20 rule describes the unfortunate reality for many data stewards: they spend 80 percent of their time finding, cleaning and reorganizing huge amounts of data and only 20 percent on actual data analysis.

In fact, we found that close to 70 percent of our survey respondents spent an average of 10 or more hours per week on data-related activities, most of it searching for and preparing data.

What can you do to reverse the 80/20 rule and subsequently overcome data governance bottlenecks?

1. Don’t ignore the complexity of data lineage: It’s a risky endeavor to support data lineage using a manual approach, and businesses that attempt it that way will find it’s not sustainable given data’s constant movement from one place to another via multiple routes – and doing it correctly down to the column level. Adopting automated end-to-end lineage makes it possible to view data movement from the source to reporting structures, providing a comprehensive and detailed view of data in motion.

2. Automate code generation: Alleviate the need for developers to hand code connections from data sources to target schema. Mapping data elements to their sources within a single repository to determine data lineage and harmonize data integration across platforms reduces the need for specialized, technical resources with knowledge of ETL and database procedural code. It also makes it easier for business analysts, data architects, ETL developers, testers and project managers to collaborate for faster decision-making.

3. Use an integrated impact analysis solution: By automating data due diligence for IT you can deliver operational intelligence to the business. Business users benefit from automating impact analysis to better examine value and prioritize individual data sets. Impact analysis has equal importance to IT for automatically tracking changes and understanding how data from one system feeds other systems and reports. This is an aspect of data lineage, created from technical metadata, ensuring nothing “breaks” along the change train.

4. Put data quality first: Users must have confidence in the data they use for analytics. Automating and matching business terms with data assets and documenting lineage down to the column level are critical to good decision-making. If this approach hasn’t been the case to date, enterprises should take a few steps back to review data quality measures before jumping into automating data analytics.

5. Catalog data using a solution with a broad set of metadata connectors: All data sources will be leveraged, including big data, ETL platforms, BI reports, modeling tools, mainframe, and relational data as well as data from many other types of systems. Don’t settle for a data catalog from an emerging vendor that only supports a narrow swath of newer technologies, and don’t rely on a catalog from a legacy provider that may supply only connectors for standard, more mature data sources.

6. Stress data literacy: You want to ensure that data assets are used strategically. Automation expedites the benefits of data cataloging. Curated internal and external datasets for a range of content authors doubles business benefits and ensures effective management and monetization of data assets in the long-term if linked to broader data governance, data quality and metadata management initiatives. There’s a clear connection to data literacy here because of its foundation in business glossaries and socializing data so all stakeholders can view and understand it within the context of their roles.

7. Make automation the norm across all data governance processes: Too many companies still live in a world where data governance is a high-level mandate, not practically implemented. To fully realize the advantages of data governance and the power of data intelligence, data operations must be automated across the board. Without automated data management, the governance housekeeping load on the business will be so great that data quality will inevitably suffer. Being able to account for all enterprise data and resolve disparity in data sources and silos using manual approaches is wishful thinking.

8. Craft your data governance strategy before making any investments: Gather multiple stakeholders—both business and IT— with multiple viewpoints to discover where their needs mesh and where they diverge and what represents the greatest pain points to the business. Solve for these first, but build buy-in by creating a layered, comprehensive strategy that ultimately will address most issues. From there, it’s on to matching your needs to an automated data governance solution that squares with business and IT – both for immediate requirements and future plans.

Register now for the first of a new, six-part webinar series on the practice of data governance and how to proactively deal with the complexities. “The What & Why of Data Governance” webinar on Tuesday, Feb. 23rd at 3 pm GMT/10 am ET.

Categories
Data Governance erwin Expert Blog

There’s More to erwin Data Governance Automation Than Meets the AI

Prashant Parikh, erwin’s Senior Vice President of Software Engineering, talks about erwin’s vision to automate every aspect of the data governance journey to increase speed to insights. The clear benefit is that data stewards spend less time building and populating the data governance framework and more time realizing value and ROI from it. 

Industry analysts and other people who write about data governance and automation define it narrowly, with an emphasis on artificial intelligence (AI) and machine learning (ML). Although AI and ML are massive fields with tremendous value, erwin’s approach to data governance automation is much broader.

Automation adds a lot of value by making processes more effective and efficient. For data governance, automation ensures the framework is always accurate and up to date; otherwise the data governance initiative itself falls apart.

From our perspective, the key to data governance success is meeting the needs of both IT and business users in the discovery and application of enterprise “data truths.” We do this through an open, configurable and flexible metamodel across data catalog, business glossary, and self-service data discovery capabilities with built-in automation.

To better explain our vision for automating data governance, let’s look at some of the different aspects of how the erwin Data Intelligence Suite (erwin DI) incorporates automation.

Metadata Harvesting and Ingestion: Automatically harvest, transform and feed metadata from virtually any source to any target to activate it within the erwin Data Catalog (erwin DC). erwin provides this metadata-driven automation through two types of data connectors: 1) erwin Standard Data Connectors for data at rest or JDBC-compliant data sources and 2) optional erwin Smart Data Connectors for data in motion or a broad variety of code types and industry-standard languages, including ELT/ETL platforms, business intelligence reports, database procedural code, testing automation tools, ecosystem utilities and ERP environments.

Data Cataloging: Catalog and sync metadata with data management and governance artifacts according to business requirements in real time. erwin DC helps organizations learn what data they have and where it’s located, including data at rest and in motion. It’s an inventory of the entire metadata universe, able to tell you the data and metadata available for a certain topic so those particular sources and assets can be found quickly for analysis and decision-making.

Data Mapping: erwin DI’s Mapping Manager provides an integrated development environment for creating and maintaining source-to-target mapping and transformation specifications to centrally version control data movement, integration and transformation. Import existing Excel or CSV files, use the drag-and-drop feature to extract the mappings from your ETL scripts, or manually populate the inventory to then be visualized with the lineage analyzer.

Code Generation: Generate ETL/ELT, Data Vault and code for other data integration components with plug-in SDKs to accelerate project delivery and reduce rework.

Data Lineage: Document and visualize how data moves and transforms across your enterprise. erwin DC generates end-to-end data lineage, down to the column level, between repositories and shows data flows from source systems to reporting layers, including intermediate transformation and business logic. Whether you’re a business user or a technical user, you can understand how data travels and transforms from point A to point B.

Data Profiling: Easily assess the contents and quality of registered data sets and associate these metrics with harvested metadata as part of ongoing data curation. Find hidden inconsistencies and highlight other potential problems using intelligent statistical algorithms and provides robust validation scores to help correct errors.

Business Glossary Management: Curate, associate and govern data assets so all stakeholders can find data relevant to their roles and understand it within a business context. erwin DI’s Business Glossary Manager is a central repository for all terms, policies and rules with out-of-the-box, industry-specific business glossaries with best-practice taxonomies and ontologies.

Semantic and Metadata Associations: erwin AIMatch automatically discovers and suggests relationships and associations between business terms and technical metadata to accelerate the creation and maintenance of governance frameworks.

Sensitive Data Discovery + Mind Mapping: Identify, document and prioritize sensitive data elements, flagging sensitive information to accelerate compliance efforts and reduce data-related risks. For example, we ship out-of-the-box General Data Protection Regulation (GDPR) policies and critical data elements that make up the GDPR policy. 

Additionally, the mind map automatically connects technical and business objects so both sets of stakeholders can easily visualize the organization’s most valuable data assets. It provides a current, holistic and enterprise-wide view of risks, enabling compliance and regulatory managers to quickly update the classifications at one level or at higher levels, if necessary. The mind map also shows you the sensitivity indicator and it allows you to propagate the sensitivity across your related objects to ensure compliance.

Self-Service Data Discovery: With an easy-to-use UI and flexible search mechanisms, business users can look up information and then perform the required analysis for quick and accurate decision-making. It further enables data socialization and collaboration between data functions within the organization.

Data Modeling Integration: By automatically harvesting your models from erwin Data Modeler and all the associated metadata for ingestion into a data catalog you ensure a single source of truth.  Then you can associate metadata with physical assets, develop a business glossary with model-driven naming standards, and socialize data models with a wider range of stakeholders. This integration also helps the business stewards because if your data model has your naming standard convention filled in, we also help them by populating the business glossary.

Enterprise Architecture Integration: erwin DI Harvester for Evolve systemically harvests data assets via smart data connectors for a wide range of data sources, both data at rest and data in motion. The harvested metadata integrates with enterprise architecture providing an accurate picture of the processes, applications and data within an organization.

Why Automating Everything Matters

The bottom line is you do not need to waste precious time, energy and resources to search, manage, analyze, prepare or protect data manually. And unless your data is well-governed, downstream data analysts and data scientists will not be able to generate significant value from it.

erwin DI provides you with the ability to populate your system with the metadata from your enterprise. We help you every step with the built in, out-of-the-box solutions and automation for every aspect of your data governance journey.

By ensuring your environment always stays controlled, you are always on top of your compliance, your tagging of sensitive data, and satisfying your unique governance needs with flexibility built into the product, and automation guiding you each step of the way.

erwin DI also enables and encourages collaboration and democratization of the data that is collected in the system; letting business users mine the data sets, because that is the ultimate value of your data governance solution.

With software-based automation and guidance from humans, the information in your data governance framework will never be outdated or out of sync with your IT and business functions. Stale data can’t fuel a successful data governance program.

Learn more about erwin automation, including what’s on the technology roadmap, by watching “Our Vision to Automate Everything” from the first day of erwin Insights 2020.

Or you can request your own demo of erwin DI.

erwin Insights 2020 on demand

Categories
Data Governance erwin Expert Blog

Automating Data Governance

Automating Data Governance

Automating data governance is key to addressing the exponentially growing volume and variety of data.

erwin CMO Mariann McDonagh recounts erwin’s vision to automate everything from day 1 of erwin Insights 2020.

Data readiness is everything. Whether driving digital experiences, mapping customer journeys, enhancing digital operations, developing digital innovations, finding new ways to interact with customers, or building digital ecosystems or marketplaces – all of this digital transformation is powered by data.

In a COVID and post-COVID world, organizations need to radically change as we look to reimagine business models and reform the way we approach almost everything.

The State of Data Automation

Data readiness depends on automation to create the data pipeline. Earlier this year, erwin conducted a research project in partnership with Dataversity, the 2020 State of Data Governance and Automation.

We asked participants to “talk to us about data value chain bottlenecks.” They told us their number one challenge is documenting complete data lineage (62%), followed by understanding the quality of the data source (58%).

Two other significant bottlenecks are finding, identifying and harvesting data (55%) curating data assets with business content for context and semantics (52%). Every item mentioned here are recurring themes we hear from our customers in terms of what led them to erwin.

We also looked at data preparation, governance and intelligence to see where organizations might be getting stuck and spending lots of time. We found that project length, slow delivery time, is one of the biggest inhibitors. Data quality and accuracy are recurring themes as well.

Reliance on developers and technical resources is another barrier to productivity. Even with data scientists in the front office, the lack of people in the back office to harvest and prepare the data means  time to value is prolonged.

Last but not least, we looked at the amount of time spent on data activities. The great news is that most organizations spend more than 10 hours a week on data-related activities. But the problem is that not enough of that time is spent on analysis because of being stuck in data prep.

IDC talks about this reverse 80/20 rule: 80% of time and effort is spent on data preparation, with only 20% focused on data analysis. This means 80% of your time is left on the cutting-room floor and can’t be used to drive your business forward.

2020 Data Governance and Automation Report

Data Automation Adds Value

Automating data operations adds a lot of value by making a solution more effective and more powerful. Consider a smart home’s thermostat, smoke detectors, lights, doorbell, etc. You have centralized access and control – from anywhere.

At erwin, our goal is to automate the entire data governance journey, whether top down or bottom up. We’re on a mission to automate all the tasks data stewards typically perform so they spend less time building and populating the data governance framework and more time using the framework to realize value and ROI.

Automation also ensures that the data governance framework is always up to date and never stale. Because without current and accurate data, a data governance initiative will fall apart.

Here are some ways erwin adds value by automating the data governance journey:

  • Metadata ingestion into the erwin Data Intelligence Suite (erwin DI) through our standard data connectors. And you can schedule metadata scans to ensure it’s always refreshed and up to date.
  • erwin Smart Data Connectors address data in motion, how it travels and transforms across the enterprise. These custom software solutions document all the traversing and transformations of data and populate the erwin DI’s Metadata Manager with the technical metadata. erwin Smart Data Connectors also document ETL scripts work with the tool of your choice.
  • erwin Lineage Analyzer puts everything together in an easy-to-understand format, making it easy for both business and technical users to visualize how data is traversing the enterprise, how it is getting transformed and the different hops it is taking along the way.
  • erwin DM Connect for DI makes it easy for metadata to be ingested from erwin Data Modeler to erwin DI. erwin DM customers can take advantage of all the rich metadata created and stored in their erwin data models. With just a couple of clicks, some or all data models can be configured and pushed erwin DI’s Metadata Manager.

The automation and integration of erwin DM and erwin DI ensures that your data models are always updated and uploaded, providing a single source of truth for your data governance journey.

This is part one of a two-part series on how erwin is automating data governance. Learn more by watching this session from erwin Insights 2020, which now is available on demand.

erwin Insights 2020

Categories
Data Governance Data Intelligence erwin Expert Blog

Doing Cloud Migration and Data Governance Right the First Time

More and more companies are looking at cloud migration.

Migrating legacy data to public, private or hybrid clouds provide creative and sustainable ways for organizations to increase their speed to insights for digital transformation, modernize and scale their processing and storage capabilities, better manage and reduce costs, encourage remote collaboration, and enhance security, support and disaster recovery.

But let’s be honest – no one likes to move. So if you’re going to move from your data from on-premise legacy data stores and warehouse systems to the cloud, you should do it right the first time. And as you make this transition, you need to understand what data you have, know where it is located, and govern it along the way.

cloud migration

Automated Cloud Migration

Historically, moving legacy data to the cloud hasn’t been easy or fast.

As organizations look to migrate their data from legacy on-prem systems to cloud platforms, they want to do so quickly and precisely while ensuring the quality and overall governance of that data.

The first step in this process is converting the physical table structures themselves. Then you must bulk load the legacy data. No less daunting, your next step is to re-point or even re-platform your data movement processes.

Without automation, this is a time-consuming and expensive undertaking. And you can’t risk false starts or delayed ROI that reduces the confidence of the business and taint this transformational initiative.

By using automated and repeatable capabilities, you can quickly and safely migrate data to the cloud and govern it along the way.

But transforming and migrating enterprise data to the cloud is only half the story – once there, it needs to be governed for completeness and compliance. That means your cloud data assets must be available for use by the right people for the right purposes to maximize their security, quality and value.

Why You Need Cloud Data Governance

Companies everywhere are building innovative business applications to support their customers, partners and employees and are increasingly migrating from legacy to cloud environments. But even with the “need for speed” to market, new applications must be modeled and documented for compliance, transparency and stakeholder literacy.

The desire to modernize technology, over time, leads to acquiring many different systems with various data entry points and transformation rules for data as it moves into and across the organization.

These tools range from enterprise service bus (ESB) products, data integration tools; extract, transform and load (ETL) tools, procedural code, application program interfaces (APIs), file transfer protocol (FTP) processes, and even business intelligence (BI) reports that further aggregate and transform data.

With all these diverse metadata sources, it is difficult to understand the complicated web they form much less get a simple visual flow of data lineage and impact analysis.

Regulatory compliance is also a major driver of data governance (e.g., GDPR, CCPA, HIPAA, SOX, PIC DSS). While progress has been made, enterprises are still grappling with the challenges of deploying comprehensive and sustainable data governance, including reliance on mostly manual processes for data mapping, data cataloging and data lineage.

Introducing erwin Cloud Catalyst

erwin just announced the release of erwin Cloud Catalyst, a suite of automated cloud migration and data governance software and services. It helps organizations quickly and precisely migrate their data from legacy, on-premise databases to the cloud and then govern those data assets throughout their lifecycle.

Only erwin provides software and services that automate the complete cloud migration and data governance lifecycle – from the reverse-engineering and transformation of legacy systems and ETL/ELT code to moving bulk data to cataloging and auto generating lineage. The metadata-driven suite automatically finds, models, ingests, catalogs and governs cloud data assets.

erwin Cloud Catalyst is comprised of erwin Data Modeler (erwin DM), erwin Data Intelligence (erwin DI) and erwin Smart Data Connectors, working together to simplify and accelerate cloud migration by removing barriers, reducing risks and decreasing time to value for your investments in these modern systems, such Snowflake, Microsoft Azure and Google Cloud.

We start with an assessment of your cloud migration strategy to determine what automation and optimization opportunities exist. Then we deliver an automation roadmap and design the appropriate smart data connectors to help your IT services team achieve your future-state cloud architecture, including accelerating data ingestion and ETL conversion.

Once your data reaches the cloud, you’ll have deep and detailed metadata management with full data governance, data lineage and impact analysis. With erwin Cloud Catalyst, you automate these data governance steps:

  • Harvest and catalog cloud data: erwin DM and erwin DI’s Metadata Manager natively scans RDBMS sources to catalog/document data assets.
  • Model cloud data structures: erwin DM converts, modifies and models the new cloud data structures.
  • Map data movement: erwin DI’s Mapping Manager defines data movement and transformation requirements via drag-and-drop functionality.
  • Generate source code: erwin DI’s automation framework generates data migration source code for any ETL/ELT SDK.
  • Test migrated data: erwin DI’s automation framework generates test cases and validation source code to test migrated data.
  • Govern cloud data: erwin DI gives cloud data assets business context and meaning through the Business Glossary Manager, as well as policies and rules for use.
  • Distribute cloud data: erwin DI’s Business User Portal provides self-service access to cloud data asset discovery and reporting tools.

Request an erwin Cloud Catalyst assessment.

And don’t forget to register for erwin Insights 2020 on October 13-14, with sessions on Snowflake, Microsoft and data lake initiatives powered by erwin Cloud Catalyst.

erwin Data Intelligence

Subscribe to the erwin Expert Blog

Once you submit the trial request form, an erwin representative will be in touch to verify your request and help you start data modeling.

Categories
Data Intelligence erwin Expert Blog

Top 6 Benefits of Automating End-to-End Data Lineage

Replace manual and recurring tasks for fast, reliable data lineage and overall data governance

Benefits of Data Lineage

It’s paramount that organizations understand the benefits of automating end-to-end data lineage. Critically, it makes it easier to get a clear view of how information is created and flows into, across and outside an enterprise.

The importance of end-to-end data lineage is widely understood and ignoring it is risky business. But it’s also important to understand why and how automation plays a critical role.

Benjamin Franklin said, “Lost time is never found again.” According to erwin’s “2020 State of Data Governance and Automation” report, close to 70 percent of data professional respondents say they spend an average of 10 or more hours per week on data-related activities, and most of that time is spent searching for and preparing data.

Data automation reduces the loss of time in collecting, processing and storing large chunks of data because it replaces manual processes (and human errors) with intelligent processes, software and artificial intelligence (AI).

Automating end-to-end data lineage helps organizations further focus their available resources on more important and strategic tasks, which ultimately provides greater value.

For example, automatically importing mappings from developers’ Excel sheets, flat files, Access and ETL tools into a comprehensive mappings inventory, complete with auto generated and meaningful documentation of the mappings, is a powerful way to support overall data governance.

According to the erwin report, documenting complete data lineage is currently the data operation with the largest percentage spread between its current level of automation (25%) and being seen as the most valuable operation to automate (65%).

Doing Data Lineage Right

Eliminating manual tasks is not the only reason to adopt automated data lineage. Replacing recurring tasks that don’t rely on human intelligence for completion is where automation makes an even bigger difference. Here are six benefits of automating end-to-end data lineage:

  1. Reduced Errors and Operational Costs

Data quality is crucial to every organization. Automated data capture can significantly reduce errors when compared to manual entry. Company documents can be filled out, stored, retrieved, and used more accurately and this, in turn, can save organizations a significant amount of money.

The 1-10-100 rule, commonly used in business circles, states that preventing an error will cost an organization $1, correcting an error already made will cost $10, and allowing an error to stand will cost $100.

Ratios will vary depending on the magnitude of the mistake and the company involved, of course, but the point remains that adopting the most reliable means of preventing a mistake, is the best approach to take in the long run.

  1. Faster Business Turnaround

Speed and faster time to market is a driving force behind most organizations’ efforts with data lineage automation. More work can be done when you are not waiting on someone to manually process data or forms.

For example, when everything can be scanned using RFID technology, it can be documented and confirmed instantaneously, cutting hours of work down to seconds.

This opens opportunities for employees to train for more profitable roles, allowing organizations to reinvest in their employees. With complex data architectures and systems within so many organizations, tracking data in motion and data at rest is daunting to say the least.

Harvesting the data through automation seamlessly removes ambiguity and speeds up the processing time-to-market capabilities.

  1. Compliance and Auditability

Regulatory compliance places greater transparency demands on firms when it comes to tracing and auditing data.

For example, capital markets trading firms must implement data lineage to support risk management, data governance and reporting for various regulations such as the Basel Committee on Banking Supervision’s standard number 239 (BCBS 239) and Markets in Financial Instruments Directive (MiFID II).

Business terms and data policies should be implemented through standardized and documented business rules. Compliance with these business rules can be tracked through data lineage, incorporating auditability and validation controls across data transformations and pipelines to generate alerts when there are non-compliant data instances.

Also, different organizational stakeholders (customers, employees and auditors) need to understand and trust reported data. Automated data lineage ensures captured data is accurate and consistent across its trajectory.

  1. Consistency, Clarity and Greater Efficiency

Data lineage automation can help improve efficiency and ensure accuracy. The more streamlined your processes, the more efficient your business. The more efficient your business, the more money you save on daily operations.

For example, backing up your data effectively and routinely is important. Data is one of the most important assets for any business.

However, different types of data need to be treated differently. Some data needs to be backed up daily while some types of data demand weekly or monthly backups.

With automation in place, you just need to develop backup strategies for your data with a consistent scheduling process. The actual job of backing things up will be managed by the system processes you set up for consistency and clarity.

  1. Improved Customer and Employee Satisfaction

Customer disengagement is a more severe problem than you might think. A recent study has shown that it costs U.S. businesses around $300 billion annually, nearly equal to the U.S. defense budget. When the employees are disengaged, they consistently give you their time but do not put the best of their efforts.

With data lineage automation, employers can automate such tasks and free up time for high-value work. According to a smartsheet report, 69% of employees thought that automation would reduce wasting time during their workday and 59% thought that they would have more than six spare hours per week if repetitive jobs were automated.

  1. Governance Enforcement

Data lineage automation is a great way to implement governance in any business. Any task that an automated process completes is always documented and has traceability.

For every task, you get clear logs that tell you what was done, who did it and when it was done. As stated before, automation plays a major role in reducing human errors and speeds up tasks that need to be performed repeatedly.

If you have not made the jump to digital yet, you are probably wading through high volumes of resources and manual processes daily. There is no denying the fact that automating business processes contributes immensely to an organization’s success. 

Automated Data Lineage in Action

Automated data lineage tools document the flow of data into and out of an organization’s systems. They capture end-to-end lineage and ensure proper impact analysis can be performed in the event of problems or changes to data assets as they move across pipelines.

erwin Data Intelligence (erwin DI) helps bind business terms to technical data assets with a complete data lineage of scanned metadata assets. Automating data capture frees up resources to focus on more strategic and useful tasks.

It automatically generates end-to-end data lineage, down to the column level and between repositories. You can view data flows from source systems to the reporting layers, including intermediate transformation and business logic.

Request your own demo of erwin DI to see metadata-driven, automated data lineage in action.

erwin Data Intelligence

Categories
Data Intelligence Data Governance erwin Expert Blog

Overcoming the 80/20 Rule – Finding More Time with Data Intelligence

The 80/20 rule is well known. It describes an unfortunate reality for many data stewards, who spend 80 percent of their time finding, cleaning and reorganizing huge amounts of data, and only 20 percent of their time on actual data analysis.

That’s a lot wasted of time.

Earlier this year, erwin released its 2020 State of Data Governance and Automation (DGA) report. About 70 percent of the DGA report respondents – a combination of roles from data architects to executive managers – say they spend an average of 10 or more hours per week on data-related activities.

COVID-19 has changed the way we work – essentially overnight – and may change how companies work moving forward. Companies like Twitter, Shopify and Box have announced that they are moving to a permanent work-from-home status as their new normal.

For much of our time as data stewards, collecting, revising and building consensus around our metadata has meant that we need to balance find time on multiple calendars against multiple competing priorities so that we can pull the appropriate data stakeholders into a room to discuss term definitions, the rules for measuring “clean” data, and identifying processes and applications that use the data.

Overcoming the 80/20 Rule - Analyzing Data

This style of data governance most often presents us with eight one-hour opportunities per day (40 one-hour opportunities per week) to meet.

As the 80/20 rule suggests, getting through hundreds, or perhaps thousands of individual business terms using this one-hour meeting model can take … a … long … time.

Now that pulling stakeholders into a room has been disrupted …  what if we could use this as 40 opportunities to update the metadata PER DAY?

What if we could buck the trend, and overcome the 80/20 rule?

Overcoming the 80/20 Rule with Micro Governance for Metadata

Micro governance is a strategy that leverages the native functionality around workflows.

erwin Data Intelligence (DI) offers Workflow Manager that creates a persistent, reusable role-based workflow such that edits to the metadata for any term can move from, for example, draft to under review to approved to published.

Using a defined workflow, it can eliminate the need for hour-long meetings with multiple stakeholders in a room. Now users can suggest edits, review changes, and approve changes on their own schedule! Using micro governance these steps should take less than 10 minutes per term:

  • Log on the DI Suite
  • Open your work queue to see items requiring your attention
  • Review and/or approve changes
  • Log out

That’s it!

And as a bonus, where stakeholders may need to discuss the edits to achieve consensus, the Collaboration Center within the Business Glossary Manager facilitates conversations between stakeholders that persistent and attached directly to the business term. No more searching through months of email conversations or forgetting to cc a key stakeholder.

Using the DI Suite Workflow Manager and the Collaboration Center, and assuming an 8-hour workday, we should each have 48 opportunities for 10 minutes of micro-governance stewardship each day.

A Culture of Micro Governance

In these days when we are all working at home, and face-to-face meetings are all but impossible, we should see this time as an opportunity to develop a culture of micro governance around our metadata.

This new way of thinking and acting will help us continuously improve our transparency and semantic understanding of our data while staying connected and collaborating with each other.

When we finally get back into the office, the micro governance ethos we’ve built while at home will help make our data governance programs more flexible, responsive and agile. And ultimately, we’ll take up less of our colleagues’ precious time.

Request a free demo of erwin DI.

Data Intelligence for Data Automation

Categories
erwin Expert Blog

Automation Gives DevOps More Horsepower

Almost 70 percent of CEOs say they expect their companies to change their business models in the next three years, and 62 percent report they have management initiatives or transformation programs underway to make their businesses more digital, according to Gartner.

Wouldn’t it be advantageous for these organizations to accelerate these digital transformation efforts? They have that option with automation, shifting DevOps away from dependence on manual processes. Just like with cars, more horsepower in DevOps translates to greater speed.

DevOps Automation

Doing More with Less

We have clients looking to do more with existing resources, and others looking to reduce full-time employee count on their DevOps teams. With metadata-driven automation, many DevOps processes can be automated, adding more “horsepower” to increase their speed and accuracy. For example:

Auto-documentation of data mappings and lineage: By using data harvesting templates, organizations can eliminate time spent updating and maintaining data mappings, creating them directly from code written by the ETL staff. Such automation can save close to 100 percent of the time usually spent on this type of documentation.

  • Data lineage and impact analysis views for ‘data in motion’ also stay up to date with no additional effort.
  • Human errors are eliminated, leading to higher quality documentation and output.

Automatic updates/changes reflected throughout each release cycle: Updates can be picked up and the ETL job/package generated with 100-percent accuracy. An ETL developer is not required to ‘hand code’ mappings from a spreadsheet – greatly reducing the time spent on the ETL process, and perhaps the total number of resources required to manage that process month over month.

  • ETL skills are still necessary for validation and to compile and execute the automated jobs, but the overall quality of these jobs (machine-generated code) will be much higher, also eliminating churn and rework.

Auto-scanning of source and target data assets with synchronized mappings: This automation eliminates the need for a resource or several resources dealing with manual updates to the design mappings, creating additional time savings and cost reductions associated with data preparation.

  • A change in the source-column header may impact 1,500 design mappings. Managed manually, this process – opening the mapping document, making the change, saving the file with a new version, and placing it into a shared folder for development – could take an analyst several days. But synchronization instantly updates the mappings, correctly versioned, and can be picked up and packaged into an ETL job/package within the same hour. Whether using agile or classic waterfall development, these processes will see exponential improvement and time reduction. 

Data Intelligence: Speed and Quality Without Compromise

Our clients often understand that incredible DevOps improvements are possible, but they fear the “work” it will take to get there.

It really comes down to deciding to embrace change a la automation or continue down the same path. But isn’t the definition of insanity doing the same thing over and over, expecting but never realizing different results?

With traditional means, you may improve speed but sacrifice quality. On the flipside, you may improve quality but sacrifice speed.

However, erwin’s technology shifts this paradigm. You can have both speed and quality.

The erwin Data Intelligence Suite (erwin DI) combines the capabilities of erwin Data Catalog with erwin Data Literacy to fuel an automated, real-time, high-quality data pipeline.

Then all enterprise stakeholders – data scientists, data stewards, ETL developers, enterprise architects, business analysts, compliance officers, CDOs and CEOs – can access data relevant to their roles for insights they can put into action.

It creates the fastest path to value, with an automation framework and metadata connectors configured by our team to deliver the data harvesting and preparation features that make capturing enterprise data assets fast and accurate.

Click here to request a free demo of erwin DI.

erwin Data Intelligence

Categories
erwin Expert Blog

Data Governance and Metadata Management: You Can’t Have One Without the Other

When an organization’s data governance and metadata management programs work in harmony, then everything is easier.

Data governance is a complex but critical practice. There’s always more data to handle, much of it unstructured; more data sources, like IoT, more points of integration, and more regulatory compliance requirements.

Creating and sustaining an enterprise-wide view of and easy access to underlying metadata is also a tall order.

The numerous data types and data sources that exist today weren’t designed to work together, and data infrastructures have been cobbled together over time with disparate technologies, poor documentation and little thought for downstream integration.

Therefore, most enterprises have encountered difficulty trying to master data governance and metadata management, but they need a solid data infrastructure on which to build their applications and initiatives.

Without it, they risk faulty analyses and insights that effect not only revenue generation but regulatory compliance and any number of other organizational objectives.

Data Governance Predictions

Data Governance Attitudes Are Shifting

The 2020 State of Data Governance and Automation (DGA) shows that attitudes about data governance and the drivers behind it are changing – arguably for the better.

Regulatory compliance was the biggest driver for data governance implementation, according to the 2018 report. That’s not surprising given the General Data Protection Regulation (GDPR) was going into effect just six months after the survey.

Now better decision-making is the primary reason to implement data governance, cited by 60 percent of survey participants. This shift suggests organizations are using data to improve their overall performance, rather than just trying to tick off a compliance checkbox.

We’re pleased to see this because we’ve always believed that IT-siloed data governance has limited value. Instead, data governance has to be an enterprise initiative with IT and the wider business collaborating to limit data-related risks and determine where greater potential and value can be unleashed.

Metadata Management Takes Time

About 70 percent of DGA report respondents – a combination of roles from data architects to executive managers – say they spend an average of 10 or more hours per week on data-related activities.

Most of that time is spent on data analysis – but only after searching for and preparing data.

A separate study by IDC indicates data professionals actually spend 80 percent of their time on data discovery, preparation and protection and only 20 percent on analysis.

Why such a heavy lift? Finding metadata, “the data about the data,” isn’t easy.

When asked about the most significant bottlenecks in the data value chain, documenting complete data lineage leads with 62 percent followed by understanding the quality of the source data (58 percent), discovery, identification and harvesting (55 percent), and curating data assets with business context (52%.)

So it make sense that the data operations deemed most valuable in terms of automation are:

  • Data Lineage (65%)
  • Data Cataloging (61%)
  • Data Mapping (53%)
  • Impact Analysis (48%)
  • Data Harvesting (38%)
  • Code Generation (21%)

But as suspected, most data operations are still manual and largely dependent on technical resources. They aren’t taking advantage of repeatable, sustainable practices – also known as automation.

The Benefits of Automating Data Governance and Metadata Management Processes

Availability, quality, consistency, usability and reduced latency are requirements at the heart of successful data governance.

And with a solid framework for automation, organizations can generate metadata every time data is captured at a source, accessed by users, moved through an organization, integrated or augmented with other data from other sources, profiled, cleansed and analyzed.

Other benefits of automating data governance and metadata management processes include:

  • Better Data Quality – Identification and repair of data issues and inconsistencies within integrated data sources in real time
  • Quicker Project Delivery – Acceleration of Big Data deployments, Data Vaults, data warehouse modernization, cloud migration, etc.
  • Faster Speed to Insights – Reversing the 80/20 rule that keeps high-paid knowledge workers too busy finding, understanding and resolving errors or inconsistencies to actually analyze source data
  • Greater Productivity & Reduced Costs – Use of automated, repeatable processes to for metadata discovery, data design, data conversion, data mapping and code generation
  • Digital Transformation – Better understanding of what data exists and its potential value to improve digital experiences, enhance digital operations, drive digital innovation and build digital ecosystems
  • Enterprise Collaboration – The ability for IT and the wider business to find, trust and use data to effectively meet organizational objectives

To learn more about the information we’ve covered in today’s blog, please join us for our webinar with Dataversity on Feb. 18.

Data Governance Webinar

Categories
erwin Expert Blog

Business Architecture and Process Modeling for Digital Transformation

At a fundamental level, digital transformation is about further synthesizing an organization’s operations and technology, so involving business architecture and process modeling is a best practice organizations cannot ignore.

This post outlines how business architecture and process modeling come together to facilitate efficient and successful digital transformation efforts.

Business Process Modeling: The First Step to Giving Customers What They Expect

Salesforce recently released the State of the Connected Customer report, with 75 percent of customers saying they expect companies to use new technologies to create better experiences. So the business and digital transformation playbook has to be updated.

These efforts must be carried out with continuous improvement in mind. Today’s constantly evolving business environment totally reinforces the old adage that change is the only constant.

Even historically reluctant-to-change banks now realize they need to innovate, adopting digital transformation to acquire and retain customers. Innovate or die is another adage that holds truer than ever before.

Fidelity International is an example of a successful digital transformation adopter and innovator. The company realized that different generations want different information and have distinct communication preferences.

For instance, millennials are adept at using digital channels, and they are the fastest-growing customer base for financial services companies. Fidelity knew it needed to understand customer needs and adapt its processes around key customer touch points and build centers of excellence to support them.

Business architecture and process modeling

Business Architecture and Process Modeling

Planning and working toward a flexible, responsive and adaptable future is no longer enough – the modern organization must be able to visualize not only the end state (the infamous and so-elusive “to-be”) but also perform detailed and comprehensive impact analysis on each scenario, often in real time. This analysis also needs to span multiple departments, extending beyond business and process architecture to IT, compliance and even HR and legal.

The ability of process owners to provide this information to management is central to ensuring the success of any transformation initiative. And new requirements and initiatives need to be managed in new ways. Digital and business transformation is about being able to do three things at the same time, all working toward the same goals:

  • Collect, document and analyze requirements
  • Establish all information layers impacted by the requirements
  • Develop and test the impact of multiple alternative scenarios

Comprehensive business process modeling underpins all of the above, providing the central information axis around which initiatives are scoped, evaluated, planned, implemented and ultimately managed.

Because of its central role, business process modeling must expand to modeling information from other layers within the organization, including:

  • System and application usage information
  • Supporting and reference documentation
  • Compliance, project and initiative information
  • Data usage

All these information layers must be captured and modeled at the appropriate levels, then connected to form a comprehensive information ecosystem that enables parts of the organization running transformation and other initiatives to instantly access and leverage it for decision-making, simulation and scenario evaluation, and planning, management and maintenance.

No Longer a Necessary Evil

Traditionally, digital and business transformation initiatives relied almost exclusively on human knowledge and experience regarding processes, procedures, how things worked, and how they fit together to provide a comprehensive and accurate framework. Today, technology can aggregate and manage all this information – and more – in a structured, organized and easily accessible way.

Business architecture extends beyond simple modeling; it also incorporates automation to reduce manual effort, remove potential for error, and guarantee effective data governance – with visibility from strategy all the way down to data entry and the ability to trace and manage data lineage. It requires robotics to cross-reference mass amounts of information, never before integrated to support effective decision-making.

The above are not options that are “nice to have,” but rather necessary gateways to taking business process management into the future. And the only way to leverage them is through systemic, organized and comprehensive business architecture modeling and analysis.

Therefore, business architecture and process modeling are no longer a necessary evil. They are critical success factors to any digital or business transformation journey.

A Competitive Weapon

Experts confirm the need to rethink and revise business processes to incorporate more digital automation. Forrester notes in its report, The Growing Importance of Process to Digital Transformation, that the changes in how business is conducted are driving the push “to reframe organizational operational processes around digital transformation efforts.” In a dramatic illustration of the need to move in this direction, the research firm writes that “business leaders are looking to use process as a competitive weapon.”

If a company hasn’t done a good job of documenting its processes, it can’t realize a future in which digital transformation is part of everyday operations. It’s never too late to start, though. In a fast-moving and pressure cooker business environment, companies need to implement business process models that make it possible to visually and analytically represent the steps that will add value to the company – either around internal operations or external ones, such as product or service delivery.

erwin BP, part of the erwin EDGE Platform, enables effective business architecture and process modeling. With it, any transformation initiative becomes a simple, streamlined exercise to support distributed information capture and management, object-oriented modeling, simulation and collaboration.

To find out about how erwin can help in empowering your transformation initiatives, please click here.

data-driven business transformation