Category: erwin Expert Blog

Categories
erwin Expert Blog Data Governance

erwin Positioned as a Leader in Gartner’s 2020 Magic Quadrant for Metadata Management Solutions for Second Year in a Row

erwin has once again been positioned as a Leader in the Gartner “2020 Magic Quadrant for Metadata Management Solutions.”

This year, erwin had the largest move of any player on the Quadrant and moved up significantly in terms of “Ability to Execute” and also in “Vision.”

This recognition affirms our efforts in developing an integrated platform for enterprise modeling and data intelligence to support data governance, digital transformation and any other effort that relies on data for favorable outcomes.

erwin’s metadata management offering, the erwin Data Intelligence Suite (erwin DI), includes data catalog, data literacy and automation capabilities for greater awareness of and access to data assets, guidance on their use, and guardrails to ensure data policies and best practices are followed.

With erwin DI’s automated, metadata-driven framework, organizations have visibility and control over their disparate data streams – from harvesting to aggregation and integration, including transformation with complete upstream and downstream lineage and all the associated documentation.

We’re super proud of this achievement and the value erwin DI provides.

We invite you to download the report and quadrant graphic.

Categories
erwin Expert Blog Enterprise Architecture

The Future of Enterprise Architecture

The business challenges facing organizations today emphasize the value of enterprise architecture (EA), so the future of EA is closer than you think. Are you ready for it?

COVID-19 has forced organizations around the globe to re-examine or reimagine themselves. However, even in “normal times,” business leaders need to understand how to grow, bring new products to market through organic growth or acquisition, identify new trends and opportunities, determine if new opportunities provide a return on investment, etc. Organizations that can identify these opportunities and respond to them have a distinct edge over their competitors.

Of course, enterprise architecture plays an important role in helping to confront and/or capitalize on these use cases:

  • COVID-19 Global Response Plans
  • Digital Transformation
  • Data Security & Risk Management
  • Compliance/Legislation
  • Innovation Management
  • Artificial Intelligence
  • Knowledge Improvement and Retention
  • Data Center Consolidation
  • Mergers and Acquisitions
  • Cloud Migration
  • Application Portfolio Management
  • Data Governance (knowing what data you have and where it is)

Let’s dig into the first two and then look at the role of enterprise architects and how to ensure your EA tools are up to the tasks ahead.

change management enterprise architecture

Chaos Creates Opportunity

COVID-19 is not just accelerating digital progress, it’s also driving a radical change in thinking, as organizations reset. The most significant COVID business takeaway has been the readiness of organizations and their employees to challenge rules, break conventions, and cut through red tape to stay in business.

In the first phases of the pandemic, organizations had to navigate business continuity and how they were going to survive. As we move into recovery mode, organizations are assessing the processes, systems and technologies that will help them assimilate to the new normal and thrive post-pandemic.

EA provides a way to drive change through every phase of recovery by providing an understanding of technology assets with business needs. For example, a COVID response plan will use EA to document if employees work from home, what their roles are, the projects on which they’re working, and what their schedules are.

Enterprise architecture has been critical to helping businesses navigate the pandemic to ensure business continuity, reimagine their business and operating models, and identify the tools to survive and ultimately thrive in a post-COVID world.

Digital Transformation

The key driver of modern EA is the demand for digital transformation. Data-driven business models and information-fueled business ecosystems provide the basis for new, innovative products and services.

The need for digital transformation has led enterprise architects to think about EA based on insights and outcomes throughout the architectural products. Architectural products are configurations of business capabilities to facilitate customer journeys, value chains, products and customer lifecycles, thus bringing the enterprise architecture much closer to the roles of product managers, customer success managers, digital platform and marketing experts.

However, digital transformation presents many challenges so it’s important not to focus on technology simply for technology’s sake. To realize successful transformation, an organization should establish new business models and the underlying supporting operating models. For example, an enterprise should start by developing a target operating model, which includes:

  • Key performance indicators (including goals, performance and benefits realization)
  • Technology (addressing business and operation systems, the assets, resources and the business)
  • Process (including the product life cycle, the development, the quality, the management and the assurance processes)
  • People management (addressing leadership, ways of working, skills and competencies and capabilities)

Enterprise Architects Become More Valuable

The importance of the enterprise architect role is recognized widely in successful businesses. An enterprise architect is now required to understand improved value through many different aspects of the business, including profits and loss, share value, risk, sales, customers and products, to name a few.

According to Gartner’s Vice President Analyst Marcus Blosch,”By 2021, 40% of organizations will use enterprise architects to help ideate new business innovations made possible by emerging technologies. EA and technology innovation leaders must use the latest business and technology ideas to create new revenue streams, services and customer experiences.”

Many organizations see business architecture as a starting point for EA, incorporating business processes and organizational design with the ability to connect to IT programs and goals.

Traditional EA is not forgotten and EAs continue to support IT governance, assurance, architecture standards and architecture review boards; however, there’s more focus on agility than command and control as has been traditionally the case with EA.

A good enterprise architect understands and tracks technology trends and appreciates how to apply these to business to enable good business outcomes.

According to Gartner, today’s enterprise architects play a transformational role in their organization. They lead and define business operating models and often have a seat at the table to advise executives and other important decision-makers. In this sense, they are trusted advisors and act in a consultancy capacity to the rest of the organization.

As with most professions, enterprise architect salaries tend to increase with years of experience and are healthy. It’s also noticeable that enterprise architects who add EA certifications to their resumes report higher earnings.

Future-Proofing EA Tools

Over the years, customer relationship management (CRM) and enterprise resource planning (ERP) tools have expanded to cover new business use cases based on customer and financial information.

It’s now important for organizations to develop an ecosystem with EA at the heart of it, with connections to ERP financials, risk systems, employees, etc. This will determine the value of EA tools and their ability to meet the needs of organizations and their EA use cases for today as well as over the next five or so years.

With that said, tools need to retain the traditional EA approaches to inspire architecture. They need to have the functionality to be built on and work up the underlying business operating models to deliver outcomes and to demonstrate real value.

Organizations need to ingest data as inputs from disparate sources within the organization — allowing the contents of the models with the help of artificial intelligence and analytics — to drive decision-making.

Business-driven applications also will be deployed through the EA repositories, which contain a wealth of information, such as strategies, processes, peoples and skills, locations, working practices, metadata, applications and technologies.

An EA tool also should offer technology trend tracking and be designed to showcase new innovation and how it can affect the organization’s goals at speed.

You can learn more by watching “The Future of Enterprise Architecture Is Closer Than You Think” from erwin Insights 2020.

And if you’re ready to get started with an EA tool that can evolve with you and your organization’s needs, then I invite you to try erwin Evolve.

future of enterprise architecture, martin owen

Categories
erwin Expert Blog Data Governance

There’s More to erwin Data Governance Automation Than Meets the AI

Prashant Parikh, erwin’s Senior Vice President of Software Engineering, talks about erwin’s vision to automate every aspect of the data governance journey to increase speed to insights. The clear benefit is that data stewards spend less time building and populating the data governance framework and more time realizing value and ROI from it. 

Industry analysts and other people who write about data governance and automation define it narrowly, with an emphasis on artificial intelligence (AI) and machine learning (ML). Although AI and ML are massive fields with tremendous value, erwin’s approach to data governance automation is much broader.

Automation adds a lot of value by making processes more effective and efficient. For data governance, automation ensures the framework is always accurate and up to date; otherwise the data governance initiative itself falls apart.

From our perspective, the key to data governance success is meeting the needs of both IT and business users in the discovery and application of enterprise “data truths.” We do this through an open, configurable and flexible metamodel across data catalog, business glossary, and self-service data discovery capabilities with built-in automation.

To better explain our vision for automating data governance, let’s look at some of the different aspects of how the erwin Data Intelligence Suite (erwin DI) incorporates automation.

Metadata Harvesting and Ingestion: Automatically harvest, transform and feed metadata from virtually any source to any target to activate it within the erwin Data Catalog (erwin DC). erwin provides this metadata-driven automation through two types of data connectors: 1) erwin Standard Data Connectors for data at rest or JDBC-compliant data sources and 2) optional erwin Smart Data Connectors for data in motion or a broad variety of code types and industry-standard languages, including ELT/ETL platforms, business intelligence reports, database procedural code, testing automation tools, ecosystem utilities and ERP environments.

Data Cataloging: Catalog and sync metadata with data management and governance artifacts according to business requirements in real time. erwin DC helps organizations learn what data they have and where it’s located, including data at rest and in motion. It’s an inventory of the entire metadata universe, able to tell you the data and metadata available for a certain topic so those particular sources and assets can be found quickly for analysis and decision-making.

Data Mapping: erwin DI’s Mapping Manager provides an integrated development environment for creating and maintaining source-to-target mapping and transformation specifications to centrally version control data movement, integration and transformation. Import existing Excel or CSV files, use the drag-and-drop feature to extract the mappings from your ETL scripts, or manually populate the inventory to then be visualized with the lineage analyzer.

Code Generation: Generate ETL/ELT, Data Vault and code for other data integration components with plug-in SDKs to accelerate project delivery and reduce rework.

Data Lineage: Document and visualize how data moves and transforms across your enterprise. erwin DC generates end-to-end data lineage, down to the column level, between repositories and shows data flows from source systems to reporting layers, including intermediate transformation and business logic. Whether you’re a business user or a technical user, you can understand how data travels and transforms from point A to point B.

Data Profiling: Easily assess the contents and quality of registered data sets and associate these metrics with harvested metadata as part of ongoing data curation. Find hidden inconsistencies and highlight other potential problems using intelligent statistical algorithms and provides robust validation scores to help correct errors.

Business Glossary Management: Curate, associate and govern data assets so all stakeholders can find data relevant to their roles and understand it within a business context. erwin DI’s Business Glossary Manager is a central repository for all terms, policies and rules with out-of-the-box, industry-specific business glossaries with best-practice taxonomies and ontologies.

Semantic and Metadata Associations: erwin AIMatch automatically discovers and suggests relationships and associations between business terms and technical metadata to accelerate the creation and maintenance of governance frameworks.

Sensitive Data Discovery + Mind Mapping: Identify, document and prioritize sensitive data elements, flagging sensitive information to accelerate compliance efforts and reduce data-related risks. For example, we ship out-of-the-box General Data Protection Regulation (GDPR) policies and critical data elements that make up the GDPR policy. 

Additionally, the mind map automatically connects technical and business objects so both sets of stakeholders can easily visualize the organization’s most valuable data assets. It provides a current, holistic and enterprise-wide view of risks, enabling compliance and regulatory managers to quickly update the classifications at one level or at higher levels, if necessary. The mind map also shows you the sensitivity indicator and it allows you to propagate the sensitivity across your related objects to ensure compliance.

Self-Service Data Discovery: With an easy-to-use UI and flexible search mechanisms, business users can look up information and then perform the required analysis for quick and accurate decision-making. It further enables data socialization and collaboration between data functions within the organization.

Data Modeling Integration: By automatically harvesting your models from erwin Data Modeler and all the associated metadata for ingestion into a data catalog you ensure a single source of truth.  Then you can associate metadata with physical assets, develop a business glossary with model-driven naming standards, and socialize data models with a wider range of stakeholders. This integration also helps the business stewards because if your data model has your naming standard convention filled in, we also help them by populating the business glossary.

Enterprise Architecture Integration: erwin DI Harvester for Evolve systemically harvests data assets via smart data connectors for a wide range of data sources, both data at rest and data in motion. The harvested metadata integrates with enterprise architecture providing an accurate picture of the processes, applications and data within an organization.

Why Automating Everything Matters

The bottom line is you do not need to waste precious time, energy and resources to search, manage, analyze, prepare or protect data manually. And unless your data is well-governed, downstream data analysts and data scientists will not be able to generate significant value from it.

erwin DI provides you with the ability to populate your system with the metadata from your enterprise. We help you every step with the built in, out-of-the-box solutions and automation for every aspect of your data governance journey.

By ensuring your environment always stays controlled, you are always on top of your compliance, your tagging of sensitive data, and satisfying your unique governance needs with flexibility built into the product, and automation guiding you each step of the way.

erwin DI also enables and encourages collaboration and democratization of the data that is collected in the system; letting business users mine the data sets, because that is the ultimate value of your data governance solution.

With software-based automation and guidance from humans, the information in your data governance framework will never be outdated or out of sync with your IT and business functions. Stale data can’t fuel a successful data governance program.

Learn more about erwin automation, including what’s on the technology roadmap, by watching “Our Vision to Automate Everything” from the first day of erwin Insights 2020.

Or you can request your own demo of erwin DI.

erwin Insights 2020 on demand

Categories
erwin Expert Blog Data Governance

Automating Data Governance

Automating Data Governance

Automating data governance is key to addressing the exponentially growing volume and variety of data.

erwin CMO Mariann McDonagh recounts erwin’s vision to automate everything from day 1 of erwin Insights 2020.

Data readiness is everything. Whether driving digital experiences, mapping customer journeys, enhancing digital operations, developing digital innovations, finding new ways to interact with customers, or building digital ecosystems or marketplaces – all of this digital transformation is powered by data.

In a COVID and post-COVID world, organizations need to radically change as we look to reimagine business models and reform the way we approach almost everything.

The State of Data Automation

Data readiness depends on automation to create the data pipeline. Earlier this year, erwin conducted a research project in partnership with Dataversity, the 2020 State of Data Governance and Automation.

We asked participants to “talk to us about data value chain bottlenecks.” They told us their number one challenge is documenting complete data lineage (62%), followed by understanding the quality of the data source (58%).

Two other significant bottlenecks are finding, identifying and harvesting data (55%) curating data assets with business content for context and semantics (52%). Every item mentioned here are recurring themes we hear from our customers in terms of what led them to erwin.

We also looked at data preparation, governance and intelligence to see where organizations might be getting stuck and spending lots of time. We found that project length, slow delivery time, is one of the biggest inhibitors. Data quality and accuracy are recurring themes as well.

Reliance on developers and technical resources is another barrier to productivity. Even with data scientists in the front office, the lack of people in the back office to harvest and prepare the data means  time to value is prolonged.

Last but not least, we looked at the amount of time spent on data activities. The great news is that most organizations spend more than 10 hours a week on data-related activities. But the problem is that not enough of that time is spent on analysis because of being stuck in data prep.

IDC talks about this reverse 80/20 rule: 80% of time and effort is spent on data preparation, with only 20% focused on data analysis. This means 80% of your time is left on the cutting-room floor and can’t be used to drive your business forward.

2020 Data Governance and Automation Report

Data Automation Adds Value

Automating data operations adds a lot of value by making a solution more effective and more powerful. Consider a smart home’s thermostat, smoke detectors, lights, doorbell, etc. You have centralized access and control – from anywhere.

At erwin, our goal is to automate the entire data governance journey, whether top down or bottom up. We’re on a mission to automate all the tasks data stewards typically perform so they spend less time building and populating the data governance framework and more time using the framework to realize value and ROI.

Automation also ensures that the data governance framework is always up to date and never stale. Because without current and accurate data, a data governance initiative will fall apart.

Here are some ways erwin adds value by automating the data governance journey:

  • Metadata ingestion into the erwin Data Intelligence Suite (erwin DI) through our standard data connectors. And you can schedule metadata scans to ensure it’s always refreshed and up to date.
  • erwin Smart Data Connectors address data in motion, how it travels and transforms across the enterprise. These custom software solutions document all the traversing and transformations of data and populate the erwin DI’s Metadata Manager with the technical metadata. erwin Smart Data Connectors also document ETL scripts work with the tool of your choice.
  • erwin Lineage Analyzer puts everything together in an easy-to-understand format, making it easy for both business and technical users to visualize how data is traversing the enterprise, how it is getting transformed and the different hops it is taking along the way.
  • erwin DM Connect for DI makes it easy for metadata to be ingested from erwin Data Modeler to erwin DI. erwin DM customers can take advantage of all the rich metadata created and stored in their erwin data models. With just a couple of clicks, some or all data models can be configured and pushed erwin DI’s Metadata Manager.

The automation and integration of erwin DM and erwin DI ensures that your data models are always updated and uploaded, providing a single source of truth for your data governance journey.

This is part one of a two-part series on how erwin is automating data governance. Learn more by watching this session from erwin Insights 2020, which now is available on demand.

erwin Insights 2020

Categories
erwin Expert Blog

Is Climbing the Corporate Ladder Still a Thing?

Thoughts on erwin Insights Day No. 2 Keynote

If you didn’t watch New York Times Best-Selling Author Keith Ferrazzi’s keynote from erwin Insights 2020, what are you waiting for?

I was blown away by Keith’s perspective on “Leading Without Authority” and it got me thinking about my own career, our employees here at erwin, work as we knew it, and work as we’ll know it in a post-COVID world.

Here are my takeaways from Keith’s session from erwin Insights Day No. 2 called “Leadership in Times of Radical Change”:

erwin Insights Leadership in Challenging Times - free access

  1. Don’t Ask for Permission

Keith asked, “How do you become transformational … how do you find the courage to say what’s possible to be that tipping point your organization needs?” The idea of having a vision within the organization and not waiting for the organization to tell you what to do really resonated with me.

So many people see things that need to be done but just sit there waiting – waiting for someone to tell them what they should do. Don’t wait. If needed, ask for forgiveness not permission to do the things that need to be done to create the type of organization you want work for.

  1. Be Your Authentic Self

We all know people who have two personalities – their work face/persona and the moment they leave the office (or log off Zoom) they become their true self. There’s an old-school notion of keeping your personal and professional lives separate. Fortunately, those days have changed.

Today, you need to bring your whole self to work. As employers, we need to give employees the space to be who are they are. And as leaders, it’s ok to be vulnerable – it’s what makes people want to follow you. Our personal lives make us who we are as people and employees, the good and the bad. It’s ok to be vulnerable at work – it builds empathy and trust in the workplace.

  1. Servant Leadership

Being a leader is multifaceted. Servant leadership flips the organizational hierarchy on its head. It puts the employees first – or at the top of the pyramid – and the executives at the bottom.

A book called “The Customer Comes Second” by Hal Rosenbluth focuses on this idea of servant leadership and the principle holds true (if not truer) today – if you put your employees first it will improve morale, performance and ultimately your bottom line.

  1. You’re the CEO of Your Own Career

As Keith points out, leadership has nothing to do with titles. Your career is in your hands, so ask yourself: Am I working with the right people? Is this job good for me right now?

Once you answer those two questions, you can make decisions that best suit you. Additionally, you must be able to cocreate/collaborate. I’ve always paid attention to the people asking the smart questions – and sought out like-minded people to create the type of organization that I envisioned and wanted to be a part of.

Going Forward (Not Back) to Work

What will work life look like in a post-COVID world? Keith says that we shouldn’t think about “going back to work” but rather “going forward to work.”

I love this concept, and I think it’s fair to say that everything has changed and will continue to evolve for quite some time. As more employees work from home or re-evaluate their careers, I believe organizations will become flatter. Therefore, climbing the corporate ladder will become a thing of the past.

Keith’s latest book is “Leading Without Authority: How the New Power of Co-Elevation Can Break Down Silos, Transform Teams, and Reinvent Collaboration.”

You can watch Keith’s presentation as well as all other erwin Insights 2020 sessions on demand.

erwin Insights 2020

Categories
Data Intelligence erwin Expert Blog

Surviving Radical Disruption with Data Intelligence

It’s certainly no secret that data has been growing in volume, variety and velocity, and most companies are overwhelmed by managing it, let alone harnessing it to put it to work.

We’re now generating 2.5 quintillion bytes of data every day, and 90% of the world’s data volume has been created in the past two years alone. With this absolute data explosion, it’s nearly impossible to filter out the time-sensitive data, the information that has immediate relevance and impact on your business.

And this time sensitivity is a massive issue, as taking a proactive and data-driven approach can literally mean life or death to your business or to your customers. And that’s where data analytics can play a huge role.

By leveraging the power of the cloud, harnessing data from the Internet of Things (IoT) and other events, and processing this data in near-real time, analytics helps to effectively process the relentless incoming data feed.

Without automation and the development of a governed data pipeline, you’ll never have enough data scientists in the front office to put the data to work. The benefits of fast time to insights is clear, regardless of the industry you’re in.

Think about these examples: a communications agency that needs to get out in front of a difficult message, a retailer driving sales based on real-time customer behavior, a logistics and delivery company needing to understand road conditions, stoppages and up-to-the-minute weather, or a hospital that needs to tailor patient care based on the latest public health findings.

Your data needs to fuel rapid decisions that make your organization more effective, customer-centric and competitive. This was true before the world changed.

COVID-19 Changed Everything

COVID changed everything. It’s a radical disruptor the likes of which we’ve never seen.

As a CEO, a husband and a father, I’ve made decisions during the past seven months that I never dreamed possible, and I’m sure this is true for you and your family – and business – as well.

Now to survive and thrive in the face of radical disruption requires radical transformation and new business models. Reimagining business, like moving fitness centers outdoors, or developing new products and services, such as restaurants packaging fruits and vegetables to sell as food bundles, or market expansion, like traditional grocers that are becoming online shopping hubs.

The companies that come out of this historic period of global uncertainty and change are those who’ve taken intelligent and data-driven approaches to their businesses.

What holds most companies back from faster time to insights and leveraging radical transformation? I think those answers can be found by asking these core questions:

  1. What data do I have?
  2. Where is the data?
  3. What people and systems are using that data and for what purposes?
  4. What processes should governance use?
  5. How is this data relevant and accessible to the business?  

Data Intelligence Provides an EDGE

There’s a common denominator in what they’re all missing, and that is data intelligence.

IDC defines data intelligence as business, technical, relational, and operational metadata that provides transparency of data profiles, classification, quality, location, context, and lineage, providing people, processes, and technology with trustworthy, reliable data.

In a new IDC Solution Brief, “The Value of Robust Data Intelligence to Enable Data Governance with erwin,” its authors state:

Data is the lifeblood of the digital economy — it is what is driving new business models, better customer experiences, better decision-making, and artificially intelligent automation. The global pandemic in 2020 has accelerated digital transformation and amplified the value of data in what will become the next normal as the global economy struggles through recovery. In a world where market conditions, supply chains, work locations, and communication methods are constantly changing, data is a constant source that can be used to inform decisions from crisis to recovery. To use data effectively, it needs to be trusted, understood, and used appropriately, and herein lies many problems that organizations face in the digital economy.

The IDC authors also interviewed erwin customers who described the erwin Data Intelligence Suite, part of the erwin EDGE platform, as a fundamental component of their efforts to generate more value from data while minimizing data-related risk.

The erwin EDGE helps organizations unlock their potential by maximizing the security, quality and value of their data assets, and it operationalizes these steps by connecting enterprise architecture, business process and data modeling with data intelligence software.

The result is an automated, real-time, high-quality data pipeline from which accurate insights can be derived.

The erwin EDGE enables organizations to see how data flows through and impacts all their business, technology and data architectures. Then all stakeholders within a company, those in IT as well as the larger enterprise, can collaborate to make better decisions based upon data truth, not just gut instinct.

Parts of this blog are excerpted from my keynote on day No. 1 of erwin Insights 2020, our virtual conference on enterprise modeling and data governance/intelligence.

You can view the entire keynote and all other sessions of the conference by registering here.

erwin Insights 2020

Subscribe to the erwin Expert Blog

Once you submit the trial request form, an erwin representative will be in touch to verify your request and help you start data modeling.

Categories
erwin Expert Blog Data Governance Data Intelligence

Doing Cloud Migration and Data Governance Right the First Time

More and more companies are looking at cloud migration.

Migrating legacy data to public, private or hybrid clouds provide creative and sustainable ways for organizations to increase their speed to insights for digital transformation, modernize and scale their processing and storage capabilities, better manage and reduce costs, encourage remote collaboration, and enhance security, support and disaster recovery.

But let’s be honest – no one likes to move. So if you’re going to move from your data from on-premise legacy data stores and warehouse systems to the cloud, you should do it right the first time. And as you make this transition, you need to understand what data you have, know where it is located, and govern it along the way.

cloud migration

Automated Cloud Migration

Historically, moving legacy data to the cloud hasn’t been easy or fast.

As organizations look to migrate their data from legacy on-prem systems to cloud platforms, they want to do so quickly and precisely while ensuring the quality and overall governance of that data.

The first step in this process is converting the physical table structures themselves. Then you must bulk load the legacy data. No less daunting, your next step is to re-point or even re-platform your data movement processes.

Without automation, this is a time-consuming and expensive undertaking. And you can’t risk false starts or delayed ROI that reduces the confidence of the business and taint this transformational initiative.

By using automated and repeatable capabilities, you can quickly and safely migrate data to the cloud and govern it along the way.

But transforming and migrating enterprise data to the cloud is only half the story – once there, it needs to be governed for completeness and compliance. That means your cloud data assets must be available for use by the right people for the right purposes to maximize their security, quality and value.

Why You Need Cloud Data Governance

Companies everywhere are building innovative business applications to support their customers, partners and employees and are increasingly migrating from legacy to cloud environments. But even with the “need for speed” to market, new applications must be modeled and documented for compliance, transparency and stakeholder literacy.

The desire to modernize technology, over time, leads to acquiring many different systems with various data entry points and transformation rules for data as it moves into and across the organization.

These tools range from enterprise service bus (ESB) products, data integration tools; extract, transform and load (ETL) tools, procedural code, application program interfaces (APIs), file transfer protocol (FTP) processes, and even business intelligence (BI) reports that further aggregate and transform data.

With all these diverse metadata sources, it is difficult to understand the complicated web they form much less get a simple visual flow of data lineage and impact analysis.

Regulatory compliance is also a major driver of data governance (e.g., GDPR, CCPA, HIPAA, SOX, PIC DSS). While progress has been made, enterprises are still grappling with the challenges of deploying comprehensive and sustainable data governance, including reliance on mostly manual processes for data mapping, data cataloging and data lineage.

Introducing erwin Cloud Catalyst

erwin just announced the release of erwin Cloud Catalyst, a suite of automated cloud migration and data governance software and services. It helps organizations quickly and precisely migrate their data from legacy, on-premise databases to the cloud and then govern those data assets throughout their lifecycle.

Only erwin provides software and services that automate the complete cloud migration and data governance lifecycle – from the reverse-engineering and transformation of legacy systems and ETL/ELT code to moving bulk data to cataloging and auto generating lineage. The metadata-driven suite automatically finds, models, ingests, catalogs and governs cloud data assets.

erwin Cloud Catalyst is comprised of erwin Data Modeler (erwin DM), erwin Data Intelligence (erwin DI) and erwin Smart Data Connectors, working together to simplify and accelerate cloud migration by removing barriers, reducing risks and decreasing time to value for your investments in these modern systems, such Snowflake, Microsoft Azure and Google Cloud.

We start with an assessment of your cloud migration strategy to determine what automation and optimization opportunities exist. Then we deliver an automation roadmap and design the appropriate smart data connectors to help your IT services team achieve your future-state cloud architecture, including accelerating data ingestion and ETL conversion.

Once your data reaches the cloud, you’ll have deep and detailed metadata management with full data governance, data lineage and impact analysis. With erwin Cloud Catalyst, you automate these data governance steps:

  • Harvest and catalog cloud data: erwin DM and erwin DI’s Metadata Manager natively scans RDBMS sources to catalog/document data assets.
  • Model cloud data structures: erwin DM converts, modifies and models the new cloud data structures.
  • Map data movement: erwin DI’s Mapping Manager defines data movement and transformation requirements via drag-and-drop functionality.
  • Generate source code: erwin DI’s automation framework generates data migration source code for any ETL/ELT SDK.
  • Test migrated data: erwin DI’s automation framework generates test cases and validation source code to test migrated data.
  • Govern cloud data: erwin DI gives cloud data assets business context and meaning through the Business Glossary Manager, as well as policies and rules for use.
  • Distribute cloud data: erwin DI’s Business User Portal provides self-service access to cloud data asset discovery and reporting tools.

Request an erwin Cloud Catalyst assessment.

And don’t forget to register for erwin Insights 2020 on October 13-14, with sessions on Snowflake, Microsoft and data lake initiatives powered by erwin Cloud Catalyst.

erwin Data Intelligence

Subscribe to the erwin Expert Blog

Once you submit the trial request form, an erwin representative will be in touch to verify your request and help you start data modeling.

Categories
erwin Expert Blog Data Modeling

The Top Six Benefits of Data Modeling – What Is Data Modeling?

Understanding the benefits of data modeling is more important than ever.

Data modeling is the process of creating a data model to communicate data requirements, documenting data structures and entity types.

It serves as a visual guide in designing and deploying databases with high-quality data sources as part of application development.

Data modeling has been used for decades to help organizations define and categorize their data, establishing standards and rules so it can be consumed and then used by information systems. Today, data modeling is a cost-effective and efficient way to manage and govern massive volumes of data, aligning data assets with the business functions they serve.

You can automatically generate data models and database designs to increase efficiency and reduce errors to make the lives or your data modelers – and other stakeholders – much more productive.

Continue reading “The Top Six Benefits of Data Modeling – What Is Data Modeling?”
Categories
erwin Expert Blog Data Intelligence

Top 6 Benefits of Automating End-to-End Data Lineage

Replace manual and recurring tasks for fast, reliable data lineage and overall data governance

Benefits of Data Lineage

It’s paramount that organizations understand the benefits of automating end-to-end data lineage. Critically, it makes it easier to get a clear view of how information is created and flows into, across and outside an enterprise.

The importance of end-to-end data lineage is widely understood and ignoring it is risky business. But it’s also important to understand why and how automation plays a critical role.

Benjamin Franklin said, “Lost time is never found again.” According to erwin’s “2020 State of Data Governance and Automation” report, close to 70 percent of data professional respondents say they spend an average of 10 or more hours per week on data-related activities, and most of that time is spent searching for and preparing data.

Data automation reduces the loss of time in collecting, processing and storing large chunks of data because it replaces manual processes (and human errors) with intelligent processes, software and artificial intelligence (AI).

Automating end-to-end data lineage helps organizations further focus their available resources on more important and strategic tasks, which ultimately provides greater value.

For example, automatically importing mappings from developers’ Excel sheets, flat files, Access and ETL tools into a comprehensive mappings inventory, complete with auto generated and meaningful documentation of the mappings, is a powerful way to support overall data governance.

According to the erwin report, documenting complete data lineage is currently the data operation with the largest percentage spread between its current level of automation (25%) and being seen as the most valuable operation to automate (65%).

Doing Data Lineage Right

Eliminating manual tasks is not the only reason to adopt automated data lineage. Replacing recurring tasks that don’t rely on human intelligence for completion is where automation makes an even bigger difference. Here are six benefits of automating end-to-end data lineage:

  1. Reduced Errors and Operational Costs

Data quality is crucial to every organization. Automated data capture can significantly reduce errors when compared to manual entry. Company documents can be filled out, stored, retrieved, and used more accurately and this, in turn, can save organizations a significant amount of money.

The 1-10-100 rule, commonly used in business circles, states that preventing an error will cost an organization $1, correcting an error already made will cost $10, and allowing an error to stand will cost $100.

Ratios will vary depending on the magnitude of the mistake and the company involved, of course, but the point remains that adopting the most reliable means of preventing a mistake, is the best approach to take in the long run.

  1. Faster Business Turnaround

Speed and faster time to market is a driving force behind most organizations’ efforts with data lineage automation. More work can be done when you are not waiting on someone to manually process data or forms.

For example, when everything can be scanned using RFID technology, it can be documented and confirmed instantaneously, cutting hours of work down to seconds.

This opens opportunities for employees to train for more profitable roles, allowing organizations to reinvest in their employees. With complex data architectures and systems within so many organizations, tracking data in motion and data at rest is daunting to say the least.

Harvesting the data through automation seamlessly removes ambiguity and speeds up the processing time-to-market capabilities.

  1. Compliance and Auditability

Regulatory compliance places greater transparency demands on firms when it comes to tracing and auditing data.

For example, capital markets trading firms must implement data lineage to support risk management, data governance and reporting for various regulations such as the Basel Committee on Banking Supervision’s standard number 239 (BCBS 239) and Markets in Financial Instruments Directive (MiFID II).

Business terms and data policies should be implemented through standardized and documented business rules. Compliance with these business rules can be tracked through data lineage, incorporating auditability and validation controls across data transformations and pipelines to generate alerts when there are non-compliant data instances.

Also, different organizational stakeholders (customers, employees and auditors) need to understand and trust reported data. Automated data lineage ensures captured data is accurate and consistent across its trajectory.

  1. Consistency, Clarity and Greater Efficiency

Data lineage automation can help improve efficiency and ensure accuracy. The more streamlined your processes, the more efficient your business. The more efficient your business, the more money you save on daily operations.

For example, backing up your data effectively and routinely is important. Data is one of the most important assets for any business.

However, different types of data need to be treated differently. Some data needs to be backed up daily while some types of data demand weekly or monthly backups.

With automation in place, you just need to develop backup strategies for your data with a consistent scheduling process. The actual job of backing things up will be managed by the system processes you set up for consistency and clarity.

  1. Improved Customer and Employee Satisfaction

Customer disengagement is a more severe problem than you might think. A recent study has shown that it costs U.S. businesses around $300 billion annually, nearly equal to the U.S. defense budget. When the employees are disengaged, they consistently give you their time but do not put the best of their efforts.

With data lineage automation, employers can automate such tasks and free up time for high-value work. According to a smartsheet report, 69% of employees thought that automation would reduce wasting time during their workday and 59% thought that they would have more than six spare hours per week if repetitive jobs were automated.

  1. Governance Enforcement

Data lineage automation is a great way to implement governance in any business. Any task that an automated process completes is always documented and has traceability.

For every task, you get clear logs that tell you what was done, who did it and when it was done. As stated before, automation plays a major role in reducing human errors and speeds up tasks that need to be performed repeatedly.

If you have not made the jump to digital yet, you are probably wading through high volumes of resources and manual processes daily. There is no denying the fact that automating business processes contributes immensely to an organization’s success. 

Automated Data Lineage in Action

Automated data lineage tools document the flow of data into and out of an organization’s systems. They capture end-to-end lineage and ensure proper impact analysis can be performed in the event of problems or changes to data assets as they move across pipelines.

erwin Data Intelligence (erwin DI) helps bind business terms to technical data assets with a complete data lineage of scanned metadata assets. Automating data capture frees up resources to focus on more strategic and useful tasks.

It automatically generates end-to-end data lineage, down to the column level and between repositories. You can view data flows from source systems to the reporting layers, including intermediate transformation and business logic.

Request your own demo of erwin DI to see metadata-driven, automated data lineage in action.

erwin Data Intelligence

Categories
Data Intelligence erwin Expert Blog

Why You Need End-to-End Data Lineage

Not Documenting End-to-End Data Lineage Is Risky Business – Understanding your data’s origins is key to successful data governance.

Not everyone understands what end-to-end data lineage is or why it is important. In a previous blog, I explained that data lineage is basically the history of data, including a data set’s origin, characteristics, quality and movement over time.

This information is critical to regulatory compliance, change management and data governance not to mention delivering an optimal customer experience. But given the volume, velocity and variety of data (the three Vs of data) we generate today, producing and keeping up with end-to-end data linage is complex and time-consuming.

Yet given this era of digital transformation and fierce competition, understanding what data you have, where it came from, how it’s changed since creation or acquisition, and whether it poses any risks is paramount to optimizing its value. Furthermore, faulty decision-making based on inconsistent analytics and inaccurate reporting can cost millions.

Data Lineage

Data Lineage Tells an Important Origin Story

End-to-end data lineage explains how information flows into, across and outside an organization. And knowing how information was created, its origin and quality may have greater value than a given data set’s current state.

For example, data lineage provides a way to determine which downstream applications and processes are affected by a change in data expectations and helps in planning for application updates.

As I mentioned above, the three Vs of data and the integration of systems makes it difficult to understand the resulting data web much less capture a simple visual of that flow. Yet a consistent view of data and how it flows is paramount to the success of enterprise data governance and any data-driven initiative.

Whether you need to drill down for a granular view of a particular data set or create a high-level summary to describe a particular system and the data it relies on, end-to-end data lineage must be documented and tracked, with an emphasis on the dynamics of data processing and movement as opposed to data structures. Data lineage helps answer questions about the origin of data in key performance indicator (KPI) reports, including:

  • How are the report tables and columns defined in the metadata?
  • Who are the data owners?
  • What are the transformation rules?

Five Consequences of Ignoring Data Lineage

Why do so many organizations struggle with end-to-end data lineage?

The struggle is real for a number of reasons. At the top of the list, organizations are dealing with more data than ever before using systems that weren’t designed to communicate effectively with one another.

Next, their IT and business stakeholders have a difficult time collaborating. And, for a lot of organizations, they’ve relied mostly on manual processes – if data lineage documentation has been attempted at all.

The risks of ignoring end-to-end data lineage are just too great. Let’s look at some of those consequences:

  1. Derailed Projects

Effectively managing business operations is a key factor to success– especially for organizations that are in the midst of digital transformation. Failures in business processes attributed to errors can be a big problem.

For example, in a typical business scenario where an incorrect data set is discovered within a report, the length of time (on average) that it takes a team to find the source of the error can take days or sometimes weeks – derailing the project and costing time and money.

  1. Policy Bloat and Unruly Rules

The business glossary environment must represent the actual environment, e.g., be refreshed and synched, otherwise it becomes obsolete. You need real collaboration.

Data dictionaries, glossaries and policies can’t live in different formats and in different places. It is common for these to be expressed in different ways, depending on the database and underlying storage technology, but this causes policy bloat and rules that no organization, team or employee will understand, let alone realistically manage.

Effective data governance requires that business glossaries, data dictionaries and data privacy policies live in one central location, so they can be easily tracked, monitored and updated over time.

  1. Major Inefficiencies

Successful data migration and upgrades rely on seamless integration of tools and processes with coordinated efforts of people/resources. A passive approach frequently relies on creating new copies of data, usually with sensitive identifiers removed or obscured.

Not only does this passive approach create inefficiencies between determining what data to copy, how to copy it, and where to store the copy, it also creates new volumes of data that become harder to track over time. Yet again, a passive approach to data cannot scale. Direct access to the same live data across the organization is required.

  1. Not Knowing Where Your Data Is

Metadata management and manual mapping are a challenge to most organizations. Data comes in all shapes, sizes and formats, and there is no way to know what type of data a project will need – or even where that data will sit.

Some data might be in the cloud, some on premise, and sometimes projects will require a hybrid approach. All data must be governed, regardless of where it is located.

  1. Privacy and Compliance Challenges

Privacy and compliance personnel know the rules that must be applied to data, but may not necessarily know the technology. Instead, automated data governance requires that anyone, with any level of expertise, can understand what rules (e.g. privacy policies) are applied to enterprise data.

Organizations with established data governance must empower both those with technical skill sets and those with privacy and compliance knowledge, so all teams can play a meaningful role controlling how data is used.

For more information on data lineage, get the free white paper, Tech Brief: Data Lineage.

End-to-End Data Lineage