Category: erwin Expert Blog

Categories
erwin Expert Blog

Data Governance Stock Check: Using Data Governance to Take Stock of Your Data Assets

For regulatory compliance (e.g., GDPR) and to ensure peak business performance, organizations often bring consultants on board to help take stock of their data assets. This sort of data governance “stock check” is important but can be arduous without the right approach and technology. That’s where data governance comes in …

While most companies hold the lion’s share of operational data within relational databases, it also can live in many other places and various other formats. Therefore, organizations need the ability to manage any data from anywhere, what we call our “any-squared” (Any2) approach to data governance.

Any2 first requires an understanding of the ‘3Vs’ of data – volume, variety and velocity – especially in context of the data lifecycle, as well as knowing how to leverage the key  capabilities of data governance – data cataloging, data literacy, business process, enterprise architecture and data modeling – that enable data to be leveraged at different stages for optimum security, quality and value.

Following are two examples that illustrate the data governance stock check, including the Any2 approach in action, based on real consulting engagements.

Data Governance Stock Check

Data Governance “Stock Check” Case 1: The Data Broker

This client trades in information. Therefore, the organization needed to catalog the data it acquires from suppliers, ensure its quality, classify it, and then sell it to customers. The company wanted to assemble the data in a data warehouse and then provide controlled access to it.

The first step in helping this client involved taking stock of its existing data. We set up a portal so data assets could be registered via a form with basic questions, and then a central team received the registrations, reviewed and prioritized them. Entitlement attributes also were set up to identify and profile high-priority assets.

A number of best practices and technology solutions were used to establish the data required for managing the registration and classification of data feeds:

1. The underlying metadata is harvested followed by an initial quality check. Then the metadata is classified against a semantic model held in a business glossary.

2. After this classification, a second data quality check is performed based on the best-practice rules associated with the semantic model.

3. Profiled assets are loaded into a historical data store within the warehouse, with data governance tools generating its structure and data movement operations for data loading.

4. We developed a change management program to make all staff aware of the information brokerage portal and the importance of using it. It uses a catalog of data assets, all classified against a semantic model with data quality metrics to easily understand where data assets are located within the data warehouse.

5. Adopting this portal, where data is registered and classified against an ontology, enables the client’s customers to shop for data by asset or by meaning (e.g., “what data do you have on X topic?”) and then drill down through the taxonomy or across an ontology. Next, they raise a request to purchase the desired data.

This consulting engagement and technology implementation increased data accessibility and capitalization. Information is registered within a central portal through an approved workflow, and then customers shop for data either from a list of physical assets or by information content, with purchase requests also going through an approval workflow. This, among other safeguards, ensures data quality.

Benefits of Data Governance

Data Governance “Stock Check” Case 2: Tracking Rogue Data

This client has a geographically-dispersed organization that stored many of its key processes in Microsoft Excel TM spreadsheets. They were planning to move to Office 365TM and were concerned about regulatory compliance, including GDPR mandates.

Knowing that electronic documents are heavily used in key business processes and distributed across the organization, this company needed to replace risky manual processes with centralized, automated systems.

A key part of the consulting engagement was to understand what data assets were in circulation and how they were used by the organization. Then process chains could be prioritized to automate and outline specifications for the system to replace them.

This organization also adopted a central portal that allowed employees to register data assets. The associated change management program raised awareness of data governance across the organization and the importance of data registration.

For each asset, information was captured and reviewed as part of a workflow. Prioritized assets were then chosen for profiling, enabling metadata to be reverse-engineered before being classified against the business glossary.

Additionally, assets that were part of a process chain were gathered and modeled with enterprise architecture (EA) and business process (BP) modeling tools for impact analysis.

High-level requirements for new systems then could be defined again in the EA/BP tools and prioritized on a project list. For the others, decisions could be made on whether they could safely be placed in the cloud and whether macros would be required.

In this case, the adoption of purpose-built data governance solutions helped build an understanding of the data assets in play, including information about their usage and content to aid in decision-making.

This client then had a good handle of the “what” and “where” in terms of sensitive data stored in their systems. They also better understood how this sensitive data was being used and by whom, helping reduce regulatory risks like those associated with GDPR.

In both scenarios, we cataloged data assets and mapped them to a business glossary. It acts as a classification scheme to help govern data and located data, making it both more accessible and valuable. This governance framework reduces risk and protects its most valuable or sensitive data assets.

Focused on producing meaningful business outcomes, the erwin EDGE platform was pivotal in achieving these two clients’ data governance goals – including the infrastructure to undertake a data governance stock check. They used it to create an “enterprise data governance experience” not just for cataloging data and other foundational tasks, but also for a competitive “EDGE” in maximizing the value of their data while reducing data-related risks.

To learn more about the erwin EDGE data governance platform and how it aids in undertaking a data governance stock check, register for our free, 30-minute demonstration here.

Categories
erwin Expert Blog

Digital Transformation in Municipal Government: The Hidden Force Powering Smart Cities

Smart cities are changing the world.

When you think of real-time, data-driven experiences and modern applications to accomplish tasks faster and easier, your local town or city government probably doesn’t come to mind. But municipal government is starting to embrace digital transformation and therefore data governance.

Municipal government has never been an area in which to look for tech innovation. Perpetually strapped for resources and budget, often relying on legacy applications and infrastructure, and perfectly happy being available during regular business hours (save for emergency responders), most municipal governments lacked the ability and motivation to (as they say in the private sector) digitally transform. Then an odd thing happened – the rest of the world started transforming.

If you shop at a retailer that doesn’t deliver a modern, personalized experience, thousands more retailers are just a click away. But people rarely pick up and move to a new city because the new city offers a better website or mobile app. The motivation for municipal governments to transform simply isn’t there in the same way it is for the private sector.

But there are some things many city residents care about deeply: public safety, quality of life, how their tax dollars are spent, and the ability to do business with their local government when they want, not when it’s convenient for the municipality. And much like the private sector, better decisions around all of these concerns can be made when accurate, timely data is available to help inform them.

Digital transformation in municipal government is taking place in two main areas today: constituent services and the “smart cities” movement.

Digital Transformation in Municipal Government: Being “Smart” About It

The ability to serve constituents easily and efficiently is of increasing importance and a key objective of digital transformation in municipal government. It’s a direct result of the data-driven customer experiences that are increasingly the norm in the private sector.

Residents want the ability to pay their taxes online, report a pothole from their phone, and generally make it easier to interact with their local officials and services. This can be accomplished with dashboards and constituent portals.

The smart cities movement refers to the broad effort of municipal governments to incorporate sensors, data collection and analysis to improve responses to everything from rush-hour traffic to air quality to crime prevention. When the McKinsey Global Institute examined smart technologies that could be deployed by cities, it found that the public sector would be the natural owner of 70 percent of the applications it reviewed.

“Cities are getting in on the data game,” says Danny Sandwell, product marketing director at erwin, Inc. And with information serving as the lifeblood of many of these projects, the effectiveness of the services offered, the return on the investments in hardware and software, and the happiness of the users all depend on timely, accurate and effective data.

These initiatives present a pretty radical departure from the way cities have traditionally been managed.

A constituent portal, for example, requires that users can be identified, authenticated and then have access to information that resides in various departments, such as the tax collector to view and pay taxes, the building department to view a building permit, and the parking authority to manage public parking permits.

For many municipalities, this is uncharted territory.

Smart Cities

Data Governance: The Force Powering Smart Cities

The efficiencies offered by smart city technologies only exist if the data leads to a proper allocation of resources.

If you can identify an increase in crime in a certain neighborhood, for example, you can increase police patrols in response. But if the data is inaccurate, those patrols are wasted while other neighborhoods experience a rise in crime.

Now that they’re in the data game, it’s time for municipal governments to understand data governance – the driving force behind any successful data-driven operation. When you have the ability to understand all of the information related to a piece of data, you have more confidence in how it is analyzed, used and protected.

Data governance doesn’t take place at a single application or in the data warehouse. It needs to be woven into the enterprise architecture and processes of the municipality to ensure data is accurate, timely and accessible to those who need it (and inaccessible to everyone else).

When this all comes together – good data, solid analytics and improved services for residents – the results can be quite striking. New efficiencies will make municipal governments better stewards of tax dollars. An improved quality of life can lift tax revenue by making the city more appealing to citizens and developers.

There’s a lot for cities to gain if they get in the data game. And truly smart cities will make sure they play the game right with effective data governance.

Benefits of Data Governance

Categories
erwin Expert Blog

Digital Transformation In Retail: The Retail Apocalypse

Much like the hospitality industry, digital transformation in retail has been a huge driver of change.

One important fact is getting lost among all of the talk of “the retail apocalypse” and myriad stories about increasingly empty shopping malls: there’s a lot of money to be made in retail. In fact, the retail market was expected to grow by more than 3 percent in 2018, unemployment is low, and wages are at least stable.

In short, there’s money to be spent. Now, where are shoppers spending it?

Coming into 2019, consumers are in control when it comes to retail. Choices are abundant. According to Deloitte’s 2018 Retail, Wholesale and Distribution Industry Trends Outlook, “consumers have been conditioned to expect fast, convenient and effortless consumption.”

This is arguably the result of the degree of digital transformation in retail that we’ve seen in recent years.

If you want to survive in retail today, you need to make it easy on your customers. That means meeting their needs across channels, fulfilling orders quickly and accurately, offering competitive prices, and not sacrificing quality in the process.

Even in a world where Amazon has changed the retail game, Walmart just announced that it had its best holiday season in years. According to a recent Fortune article, “Walmart’s e-commerce sales rose 43 percent during the quarter, belying another myth: e-commerce and store sales are in competition with each other.”

Retail has always been a very fickle industry, with the right product mix and the right appeal to the right customers being crucial to success. But digital transformation in retail has seen the map change. You’re no longer competing with the store across the street; you’re competing with the store across the globe.

Digital Transformation In Retail

Retailers are putting every aspect of their businesses under scrutiny to help them remain relevant. Four areas in particular are getting a great deal of attention:

Customer experience: In today’s need-it-fast, need-it-now, need-it-right world, customers expect the ability to make purchases where they are, not where you are. That means via the Web, mobile devices or in a store. And all of the information about those orders needs to be tied together, so that if there is a problem, it can be resolved quickly via any channel.

Competitive differentiation: Appealing to retail customers used to mean appealing to all of your customers as one group or like-minded block. But customers are individuals, and today they can be targeted with personalized messaging and products that are likely to appeal to them, not to everyone.

Supply chain: Having the right products in the right place at the right time is part of the supply chain strategy. But moving them efficiently and cost effectively from any number of suppliers to warehouses and stores can make or break margins.

Partnerships: Among the smaller players in the retail space, partnerships with industry giants like Amazon can help reach a global audience that simply isn’t otherwise available and also reduce complexity. Larger players also recognize that partnerships can be mutually beneficial in the retail space.

Enabling each of these strategies is data – and lots of it. Data is the key to recognizing customers, personalizing experiences, making helpful recommendations, ensuring items are in stock, tracking deliveries and more. At its core, this is what digital transformation in retail seeks to achieve.

Digital Transformation in Retail – What’s the Risk?

But if data is the great enabler in retail, it’s also a huge risk – risk that the data is wrong, that it is old, and that it ends up in the hands of some person or entity that isn’t supposed to have it.

Danny Sandwell, director of product marketing for erwin, Inc., says retailers need to achieve a level of what he calls “data intelligence.” A little like business intelligence, Sandwell uses the term to mean that when someone in retail uses data to make a decision or power an experience or send a recommendation, they have the ability to find out anything they need about that data, including its source, age, who can access it, which applications use it, and more.

Given all of the data that flows into the modern retailer, this level of data intelligence requires a holistic, mature and well-planned data governance strategy. Data governance doesn’t just sit in the data warehouse, it’s woven into business processes and enterprise architecture to provide data visibility for fast, accurate decision-making, help keep data secure, identify problems early, and alert users to things that are working.

How important is clean, accurate, timely data in retail? Apply it to the four areas discussed above:

Customer experience:  If your data shows a lot of abandoned carts from mobile app users, then that’s an area to investigate, and good data will identify it.

Competitive differentiation: Are personalized offers increasing sales and creating customer loyalty? This is an important data point for marketing strategy.

Supply chain: Can a problem with quality be related to items shipping from a certain warehouse? Data will zero in on the location of the problem.

Partnerships: Are your partnerships helping grow other parts of your business and creating new customers? Or are your existing customers using partners in place of visiting your store? Data can tell you.

Try drawing these conclusions without data. You can’t. And even worse, try drawing them with inaccurate data and see what happens when a partnership that was creating customers is ended or mobile app purchases plummet after an ill-advised change to the experience.

If you want to focus on margins in retail, don’t forget this one: there is no margin for error.

Over the next few weeks, we’ll be looking closely at digital transformation examples in other sectors, including hospitality and government. Subscribe to to stay in the loop.

Data Management and Data Governance: Solving the Enterprise Data Dilemma

Categories
erwin Expert Blog

Digital Transformation Examples: How Data Is Transforming the Hospitality Industry

The rate at which organizations have adopted data-driven strategies means there are a wealth of digital transformation examples for organizations to draw from.

By now, you probably recognize this recurring pattern in the discussions about digital transformation:

  • An industry set in its ways slowly moves toward using information technology to create efficiencies, automate processes or help identify new customer or product opportunities.
  • All is going fine until a new kid on the block, born in the age of IT and the internet, quickly starts to create buzz and redefine what customers expect from the industry.
  • To keep pace, the industry stalwarts rush into catch-up mode but make inevitably mistakes. ROI doesn’t meet expectations, the customer experience isn’t quite right, and data gets exposed or mishandled.

There’s one industry we’re all familiar with that welcomes billions of global customers every year; that’s in the midst of a strong economic run; is dealing with high-profile disruptors; and suffered a very public data breach to one of its storied brands in 2018 that raised eyebrows around the world.

Welcome to the hospitality industry.

The hotel and hospitality industry was expected to see 5 to 6 percent growth in 2018, part of an impressive run of performance fueled by steady demand, improved midmarket offerings, and a new supply of travelers from developing regions.

All this despite challenges from upstarts like AirB2B, HomeAway and Couchsurfing plus a data breach at Marriott/Starwood that exposed the data of 500 million customers.

Digital Transformation Examples: Data & the Hospitality Industry

Online start-ups such as Airbnb, HomeAway and Couchsurfing are some of the most clear cut digital transformation examples in the hospitality industry.

Digital Transformation Examples: Hospitality – Data, Data Everywhere

As with other industries, digital transformation examples in the hospitality industry are abundant – and in turn, those businesses are awash in data with sources that include:

  • Data generated by reservations and payments
  • The data hotels collect to drive their loyalty programs
  • Data used to enhance the customer experience
  • Data shared as part of the billions of handoffs between hotel chains and the various booking sites and agencies that travelers use to plan trips

But all of this data, which now permeates the industry, is relatively new.

“IT wasn’t always a massive priority for [the hospitality industry],” says Danny Sandwell, director of product marketing for erwin, Inc. “So now there’s a lot of data, but these organizations often have a weak backend.”

The combination of data and analytics carries a great deal of potential for companies in the hospitality industry. Today’s demanding customers want experiences, not just a bed to sleep in; they want to do business with brands that understand their likes and dislikes; and that send offers relevant to their interests and desired destinations.

All of this is possible when a business collects and analyzes data on the scale that many hotel brands do. However, all of this can fail loudly if there is a problem with that data.

Getting a return on their investments in analytics and marketing technology requires hospitality companies to thoroughly understand the source of their data, the quality of the data, and the relevance of the data. This is where data governance comes into play.

When hospitality businesses are confident in their data, they can use it a number of ways, including:

  • Customer Experience: Quality data can be used to power a best-in-class experience for hotels in a number of areas, including the Web experience, mobile experience, and the in-person guest experience. This is similar to the multi-channel strategy of retailers hoping to deliver memorable and helpful experiences based on what they know about customers, including the ability to make predictions and deliver cross-sell and up-sell opportunities. 
  • Mergers and Acquisitions: Hospitality industry disruptors have some industry players thinking about boosting their businesses via mergers and acquisitions. Good data can identify the best targets and help discover the regions or price points where M&A makes the most sense and will deliver the most value. Accurate data can also help pinpoint the true cost of M&A activity.
  • Security: Marriott’s data breach, which actually began as a breach at Starwood before Marriott acquired it, highlights the importance of data security in the hospitality industry. Strong data governance can help prevent breaches, as well as help control breaches so organizations more quickly identify the scope and action behind a breach, an important part of limiting damage.
  • Partnerships: The hospitality industry is increasingly connected, not just because of booking sites working with dozens of hotel brands but also because of tour operators turning a hotel stay into an experience and transportation companies arranging travel for guests. Providing a room is no longer enough.

Data governance is not an application or a tool. It is a strategy. When it is done correctly and it is deployed in a holistic manner, data governance becomes woven into an organization’s business processes and enterprise architecture.

It then improves the organization’s ability to understand where its data is, where it came from, its value, its quality, and how the data is accessed and used by people and applications.

It’s this level of data maturity that provides comfort to employees – from IT staff to the front desk and everyone in between – that the data they are working with is accurate and helping them better perform their jobs and improve the way they serve customers.

Over the next few weeks, we’ll be looking closely at digital transformation examples in other sectors, including retail and government. Subscribe to to stay in the loop.

GDPR White Paper

Categories
erwin Expert Blog

Four Use Cases Proving the Benefits of Metadata-Driven Automation

Organization’s cannot hope to make the most out of a data-driven strategy, without at least some degree of metadata-driven automation.

The volume and variety of data has snowballed, and so has its velocity. As such, traditional – and mostly manual – processes associated with data management and data governance have broken down. They are time-consuming and prone to human error, making compliance, innovation and transformation initiatives more complicated, which is less than ideal in the information age.

So it’s safe to say that organizations can’t reap the rewards of their data without automation.

Data scientists and other data professionals can spend up to 80 percent of their time bogged down trying to understand source data or addressing errors and inconsistencies.

That’s time needed and better used for data analysis.

By implementing metadata-driven automation, organizations across industry can unleash the talents of their highly skilled, well paid data pros to focus on finding the goods: actionable insights that will fuel the business.

Metadata-Driven Automation

Metadata-Driven Automation in the BFSI Industry

The banking, financial services and insurance industry typically deals with higher data velocity and tighter regulations than most. This bureaucracy is rife with data management bottlenecks.

These bottlenecks are only made worse when organizations attempt to get by with systems and tools that are not purpose-built.

For example, manually managing data mappings for the enterprise data warehouse via MS Excel spreadsheets had become cumbersome and unsustainable for one BSFI company.

After embracing metadata-driven automation and custom code automation templates, it saved hundreds of thousands of dollars in code generation and development costs and achieved more work in less time with fewer resources. ROI on the automation solutions was realized within the first year.

Metadata-Driven Automation in the Pharmaceutical Industry

Despite its shortcomings, the Excel spreadsheet method for managing data mappings is common within many industries.

But with the amount of data organizations need to process in today’s business climate, this manual approach makes change management and determining end-to-end lineage a significant and time-consuming challenge.

One global pharmaceutical giant headquartered in the United States experienced such issues until it adopted metadata-driven automation. Then the pharma company was able to scan in all source and target system metadata and maintain it within a single repository. Users now view end-to-end data lineage from the source layer to the reporting layer within seconds.

On the whole, the implementation resulted in extraordinary time savings and a total cost reduction of 60 percent.

Metadata-Driven Automation in the Insurance Industry

Insurance is another industry that has to cope with high data velocity and stringent data regulations. Plus many organizations in this sector find that they’ve outgrown their systems.

For example, an insurance company using a CDMA product to centralize data mappings is probably missing certain critical features, such as versioning, impact analysis and lineage, which adds to costs, times to market and errors.

By adopting metadata-driven automation, organizations can standardize the pre-ETL data mapping process and better manage data integration through the change and release process. As a result, both internal data mapping and cross functional teams now have easy and fast web-based access to data mappings and valuable information like impact analysis and lineage.

Here is the story of a business that adopted such an approach and achieved operational excellence and a delivery time reduction by 80 percent, as well as achieving ROI within 12 months.

Metadata-Driven Automation for a Non-Profit

Another common issue cited by organizations using manual data mapping is ballooning complexity and subsequent confusion.

Any organization expanding its data-driven focus without sufficiently maturing data management initiative(s) will experience this at some point.

One of the world’s largest humanitarian organizations, with millions of members and volunteers operating all over the world, was confronted with this exact issue.

It recognized the need for a solution to standardize the pre-ETL data mapping process to make data integration more efficient and cost-effective.

With metadata-driven automation, the organization would be able to scan and store metadata and data dictionaries in a central repository, as well as manage the business definitions and data dictionary for legacy systems contributing data to the enterprise data warehouse.

By adopting such an approach, the organization realized time savings across all IT development and cross-functional testing teams. Additionally, they were able to more easily manage mappings, code sets, reference data and data validation rules.

Again, ROI was achieved within a year.

A Universal Solution for Metadata-Driven Automation

Metadata-driven automation is a capability any organization can benefit from – regardless of industry, as demonstrated by the various real-world use cases chronicled here.

The erwin Automation Framework is a key component of the erwin EDGE platform for comprehensive data management and data governance.

With it, data professionals realize these industry-agnostic benefits:

  • Centralized and standardized code management with all automation templates stored in a governed repository
  • Better quality code and minimized rework
  • Business-driven data movement and transformation specifications
  • Superior data movement job designs based on best practices
  • Greater agility and faster time-to-value in data preparation, deployment and governance
  • Cross-platform support of scripting languages and data movement technologies

Learn more about metadata-driven automation as it relates to data preparation and enterprise data mapping.

Join one our weekly erwin Mapping Manager demos.

Automate Data Mapping

Categories
erwin Expert Blog

Google’s Record GDPR Fine: Avoiding This Fate with Data Governance

The General Data Protection Regulation (GDPR) made its first real impact as Google’s record GDPR fine dominated news cycles.

Historically, fines had peaked at six figures with the U.K.’s Information Commissioner’s Office (ICO) fines of 500,000 pounds ($650,000 USD) against both Facebook and Equifax for their data protection breaches.

Experts predicted an uptick in GDPR enforcement in 2019, and Google’s recent record GDPR fine has brought that to fruition. France’s data privacy enforcement agency hit the tech giant with a $57 million penalty – more than 80 times the steepest ICO fine.

If it can happen to Google, no organization is safe. Many in fact still lag in the GDPR compliance department. Cisco’s 2019 Data Privacy Benchmark Study reveals that only 59 percent of organizations are meeting “all or most” of GDPR’s requirements.

So many more GDPR violations are likely to come to light. And even organizations that are currently compliant can’t afford to let their data governance standards slip.

Data Governance for GDPR

Google’s record GDPR fine makes the rationale for better data governance clear enough. However, the Cisco report offers even more insight into the value of achieving and maintaining compliance.

Organizations with GDPR-compliant security measures are not only less likely to suffer a breach (74 percent vs. 89 percent), but the breaches suffered are less costly too, with fewer records affected.

However, applying such GDPR-compliant provisions can’t be done on a whim; organizations must expand their data governance practices to include compliance.

GDPR White Paper

A robust data governance initiative provides a comprehensive picture of an organization’s systems and the units of data contained or used within them. This understanding encompasses not only the original instance of a data unit but also its lineage and how it has been handled and processed across an organization’s ecosystem.

With this information, organizations can apply the relevant degrees of security where necessary, ensuring expansive and efficient protection from external (i.e., breaches) and internal (i.e., mismanaged permissions) data security threats.

Although data security cannot be wholly guaranteed, these measures can help identify and contain breaches to minimize the fallout.

Looking at Google’s Record GDPR Fine as An Opportunity

The tertiary benefits of GDPR compliance include greater agility and innovation and better data discovery and management. So arguably, the “tertiary” benefits of data governance should take center stage.

While once exploited by such innovators as Amazon and Netflix, data optimization and governance is now on everyone’s radar.

So organization’s need another competitive differentiator.

An enterprise data governance experience (EDGE) provides just that.

THE REGULATORY RATIONALE FOR INTEGRATING DATA MANAGEMENT & DATA GOVERNANCE

This approach unifies data management and data governance, ensuring that the data landscape, policies, procedures and metrics stem from a central source of truth so data can be trusted at any point throughout its enterprise journey.

With an EDGE, the Any2 (any data from anywhere) data management philosophy applies – whether structured or unstructured, in the cloud or on premise. An organization’s data preparation (data mapping), enterprise modeling (business, enterprise and data) and data governance practices all draw from a single metadata repository.

In fact, metadata from a multitude of enterprise systems can be harvested and cataloged automatically. And with intelligent data discovery, sensitive data can be tagged and governed automatically as well – think GDPR as well as HIPAA, BCBS and CCPA.

Organizations without an EDGE can still achieve regulatory compliance, but data silos and the associated bottlenecks are unavoidable without integration and automation – not to mention longer timeframes and higher costs.

To get an “edge” on your competition, consider the erwin EDGE platform for greater control over and value from your data assets.

Data preparation/mapping is a great starting point and a key component of the software portfolio. Join us for a weekly demo.

Automate Data Mapping

Categories
erwin Expert Blog

Five Benefits of an Automation Framework for Data Governance

Organizations are responsible for governing more data than ever before, making a strong automation framework a necessity. But what exactly is an automation framework and why does it matter?

In most companies, an incredible amount of data flows from multiple sources in a variety of formats and is constantly being moved and federated across a changing system landscape.

Often these enterprises are heavily regulated, so they need a well-defined data integration model that helps avoid data discrepancies and removes barriers to enterprise business intelligence and other meaningful use.

IT teams need the ability to smoothly generate hundreds of mappings and ETL jobs. They need their data mappings to fall under governance and audit controls, with instant access to dynamic impact analysis and lineage.

With an automation framework, data professionals can meet these needs at a fraction of the cost of the traditional manual way.

In data governance terms, an automation framework refers to a metadata-driven universal code generator that works hand in hand with enterprise data mapping for:

  • Pre-ETL enterprise data mapping
  • Governing metadata
  • Governing and versioning source-to-target mappings throughout the lifecycle
  • Data lineage, impact analysis and business rules repositories
  • Automated code generation

Such automation enables organizations to bypass bottlenecks, including human error and the time required to complete these tasks manually.

In fact, being able to rely on automated and repeatable processes can result in up to 50 percent in design savings, up to 70 percent conversion savings and up to 70 percent acceleration in total project delivery.

So without further ado, here are the five key benefits of an automation framework for data governance.

Automation Framework

Benefits of an Automation Framework for Data Governance

  1. Creates simplicity, reliability, consistency and customization for the integrated development environment.

Code automation templates (CATs) can be created – for virtually any process and any tech platform – using the SDK scripting language or the solution’s published libraries to completely automate common, manual data integration tasks.

CATs are designed and developed by senior automation experts to ensure they are compliant with industry or corporate standards as well as with an organization’s best practice and design standards.

The 100-percent metadata-driven approach is critical to creating reliable and consistent CATs.

It is possible to scan, pull in and configure metadata sources and targets using standard or custom adapters and connectors for databases, ERP, cloud environments, files, data modeling, BI reports and Big Data to document data catalogs, data mappings, ETL (XML code) and even SQL procedures of any type.

  1. Provides blueprints anyone in the organization can use.

Stage DDL from source metadata for the target DBMS; profile and test SQL for test automation of data integration projects; generate source-to-target mappings and ETL jobs for leading ETL tools, among other capabilities.

It also can populate and maintain Big Data sets by generating PIG, Scoop, MapReduce, Spark, Python scripts and more.

  1. Incorporates data governance into the system development process.

An organization can achieve a more comprehensive and sustainable data governance initiative than it ever could with a homegrown solution.

An automation framework’s ability to automatically create, version, manage and document source-to-target mappings greatly matters both to data governance maturity and a shorter-time-to-value.

This eliminates duplication that occurs when project teams are siloed, as well as prevents the loss of knowledge capital due to employee attrition.

Another value capability is coordination between data governance and SDLC, including automated metadata harvesting and cataloging from a wide array of sources for real-time metadata synchronization with core data governance capabilities and artifacts.

  1. Proves the value of data lineage and impact analysis for governance and risk assessment.

Automated reverse-engineering of ETL code into natural language enables a more intuitive lineage view for data governance.

With end-to-end lineage, it is possible to view data movement from source to stage, stage to EDW, and on to a federation of marts and reporting structures, providing a comprehensive and detailed view of data in motion.

The process includes leveraging existing mapping documentation and auto-documented mappings to quickly render graphical source-to-target lineage views including transformation logic that can be shared across the enterprise.

Similarly, impact analysis – which involves data mapping and lineage across tables, columns, systems, business rules, projects, mappings and ETL processes – provides insight into potential data risks and enables fast and thorough remediation when needed.

Impact analysis across the organization while meeting regulatory compliance with industry regulators requires detailed data mapping and lineage.

THE REGULATORY RATIONALE FOR INTEGRATING DATA MANAGEMENT & DATA GOVERNANCE

  1. Supports a wide spectrum of business needs.

Intelligent automation delivers enhanced capability, increased efficiency and effective collaboration to every stakeholder in the data value chain: data stewards, architects, scientists, analysts; business intelligence developers, IT professionals and business consumers.

It makes it easier for them to handle jobs such as data warehousing by leveraging source-to-target mapping and ETL code generation and job standardization.

It’s easier to map, move and test data for regular maintenance of existing structures, movement from legacy systems to new systems during a merger or acquisition, or a modernization effort.

erwin’s Approach to Automation for Data Governance: The erwin Automation Framework

Mature and sustainable data governance requires collaboration from both IT and the business, backed by a technology platform that accelerates the time to data intelligence.

Part of the erwin EDGE portfolio for an “enterprise data governance experience,” the erwin Automation Framework transforms enterprise data into accurate and actionable insights by connecting all the pieces of the data management and data governance lifecycle.

 As with all erwin solutions, it embraces any data from anywhere (Any2) with automation for relational, unstructured, on-premise and cloud-based data assets and data movement specifications harvested and coupled with CATs.

If your organization would like to realize all the benefits explained above – and gain an “edge” in how it approaches data governance, you can start by joining one of our weekly demos for erwin Mapping Manager.

Automate Data Mapping

Categories
erwin Expert Blog

erwin Automation Framework: Achieving Faster Time-to-Value in Data Preparation, Deployment and Governance

Data governance is more important to the enterprise than ever before. It ensures everyone in the organization can discover and analyze high-quality data to quickly deliver business value.

It assists in successfully meeting increasingly strict compliance requirements, such as those in the General Data Protection Regulation (GDPR). And it provides a clear gauge on business performance.

A mature and sustainable data governance initiative must include data integration.

This often requires reconciling two groups of individuals within the organization: 1) those who care about governance and the meaningful use of data and 2) those who care about getting and transforming the data from source to target for actionable insights.

Both ends of the data value chain are covered when governance is coupled programmatically with IT’s integration practices.

The tools and processes for this should automatically generate “pre-ETL” source-to-target mapping to minimize human errors that can occur while manually compiling and interpreting a multitude of Excel-based data mappings that exist across the organization.

In addition to reducing errors and improving data quality, the efficiencies gained through automation, including minimizing rework, can help cut system development lifecycle costs in half.

In fact, being able to rely on automated and repeatable processes can result in up to 50 percent in design savings, up to 70 percent conversion savings, and up to 70 percent acceleration in total project delivery.

Data Governance and the System Development Lifecycle

Boosting data governance maturity starts with a central metadata repository (data dictionary) for version-controlling metadata imported from a broad array of file and database types to inform data mappings. It can be used to automatically generate governed design mappings and code in the design phase of the system development lifecycle.

The right toolset – one that supports a unifying and underlying metadata model – will be a design and code-generation platform that introduces efficiency, visibility and governance principles while reducing the opportunity for human error.

Automatically generating ETL/ELT jobs for leading ETL tools based on best design practices accommodates those principles; it functions according to approved corporate and industry standards.

Automatically importing mappings from developers’ Excel sheets, flat files, access and ETL tools into a comprehensive mappings inventory, complete with automatically generated and meaningful documentation of the mappings, is a powerful way to support governance while providing real insight into data movement – for lineage and impact analysis – without interrupting system developers’ normal work methods.

GDPR compliance, for example, requires a business to discover source-to-target mappings with all accompanying transactions, such as what business rules in the repository are applied to it, to comply with audits.

THE REGULATORY RATIONALE FOR INTEGRATING DATA MANAGEMENT & DATA GOVERNANCE

When data movement has been tracked and version-controlled, it’s possible to conduct data archeology – that is, reverse-engineering code from existing XML within the ETL layer – to uncover what has happened in the past and incorporating it into a mapping manager for fast and accurate recovery.

This is one example of how to meet data governance demands with more agility and accuracy at high speed.

Faster Time-to-Value with the erwin Automation Framework

The erwin Automation Framework is a metadata-driven universal code generator that works hand in hand with erwin Mapping Manager (MM) for:

  • Pre-ETL enterprise data mapping
  • Governing metadata
  • Governing and versioning source-to-target mappings throughout the lifecycle
  • Data lineage, impact analysis and business rules repositories
  • Automated code generation

If you’d like to save time and money in preparing, deploying and governing you organization’s data, please join us for a demo of erwin MM.

Automate Data Mapping

Categories
erwin Expert Blog

Data Preparation and Data Mapping: The Glue Between Data Management and Data Governance to Accelerate Insights and Reduce Risks

Organizations have spent a lot of time and money trying to harmonize data across diverse platforms, including cleansing, uploading metadata, converting code, defining business glossaries, tracking data transformations and so on. But the attempts to standardize data across the entire enterprise haven’t produced the desired results.

A company can’t effectively implement data governance – documenting and applying business rules and processes, analyzing the impact of changes and conducting audits – if it fails at data management.

The problem usually starts by relying on manual integration methods for data preparation and mapping. It’s only when companies take their first stab at manually cataloging and documenting operational systems, processes and the associated data, both at rest and in motion, that they realize how time-consuming the entire data prepping and mapping effort is, and why that work is sure to be compounded by human error and data quality issues.

To effectively promote business transformation, as well as fulfil regulatory and compliance mandates, there can’t be any mishaps.

It’s obvious that the manual road is very challenging to discover and synthesize data that resides in different formats in thousands of unharvested, undocumented databases, applications, ETL processes and procedural code.

Consider the problematic issue of manually mapping source system fields (typically source files or database tables) to target system fields (such as different tables in target data warehouses or data marts).

These source mappings generally are documented across a slew of unwieldy spreadsheets in their “pre-ETL” stage as the input for ETL development and testing. However, the ETL design process often suffers as it evolves because spreadsheet mapping data isn’t updated or may be incorrectly updated thanks to human error. So questions linger about whether transformed data can be trusted.

Data Quality Obstacles

The sad truth is that high-paid knowledge workers like data scientists spend up to 80 percent of their time finding and understanding source data and resolving errors or inconsistencies, rather than analyzing it for real value.

Statistics are similar when looking at major data integration projects, such as data warehousing and master data management with data stewards challenged to identify and document data lineage and sensitive data elements.

So how can businesses produce value from their data when errors are introduced through manual integration processes? How can enterprise stakeholders gain accurate and actionable insights when data can’t be easily and correctly translated into business-friendly terms?

How can organizations master seamless data discovery, movement, transformation and IT and business collaboration to reverse the ratio of preparation to value delivered.

What’s needed to overcome these obstacles is establishing an automated, real-time, high-quality and metadata- driven pipeline useful for everyone, from data scientists to enterprise architects to business analysts to C-level execs.

Doing so will require a hearty data management strategy and technology for automating the timely delivery of quality data that measures up to business demands.

From there, they need a sturdy data governance strategy and technology to automatically link and sync well-managed data with core capabilities for auditing, statutory reporting and compliance requirements as well as to drive business insights.

Creating a High-Quality Data Pipeline

Working hand-in-hand, data management and data governance provide a real-time, accurate picture of the data landscape, including “data at rest” in databases, data lakes and data warehouses and “data in motion” as it is integrated with and used by key applications. And there’s control of that landscape to facilitate insight and collaboration and limit risk.

With a metadata-driven, automated, real-time, high-quality data pipeline, all stakeholders can access data that they now are able to understand and trust and which they are authorized to use. At last they can base strategic decisions on what is a full inventory of reliable information.

The integration of data management and governance also supports industry needs to fulfill regulatory and compliance mandates, ensuring that audits are not compromised by the inability to discover key data or by failing to tag sensitive data as part of integration processes.

Data-driven insights, agile innovation, business transformation and regulatory compliance are the fruits of data preparation/mapping and enterprise modeling (business process, enterprise architecture and data modeling) that revolves around a data governance hub.

erwin Mapping Manager (MM) combines data management and data governance processes in an automated flow through the integration lifecycle from data mapping for harmonization and aggregation to generating the physical embodiment of data lineage – that is the creation, movement and transformation of transactional and operational data.

Its hallmark is a consistent approach to data delivery (business glossaries connect physical metadata to specific business terms and definitions) and metadata management (via data mappings).

Automate Data Mapping

Categories
erwin Expert Blog

Who to Follow in 2019 for Big Data, Data Governance and GDPR Advice

Experts are predicting a surge in GDPR enforcement in 2019 as regulators begin to crackdown on organizations still lagging behind compliance standards.

With this in mind, the erwin team has compiled a list of the most valuable data governance, GDPR and Big data blogs and news sources for data management and data governance best practice advice from around the web.

From regulatory compliance (GDPR, HIPPA, etc.,) to driving revenue through proactive data governance initiatives and Big Data strategies, these accounts cover it all.

Top 7 Data Governance, GDPR and Big Data Blogs and News Sources from Around the Web

Honorable Mention: @BigDataBatman

The Twitter account data professionals deserve, but probably not the one you need right now.

This quirky Twitter bot trawls the web for big data tweets and news stories, and substitutes “big data” for “Batman”. If data is the Bane of your existence, this account will serve up some light relief.

 

1. The erwin Expert Network

Twitter| LinkedIn | Facebook | Blog

For anything data management and data governance related, the erwin Experts should be your first point of call.

The team behind the most connected data management and data governance solutions on the market regularly share best practice advice in guide, whitepaper, blog and social media update form.

 

2. GDPR For Online Entrepreneurs (UK, US, CA, AU)

This community-driven Facebook group is a consistent source of insightful information for data-driven businesses.

In addition to sharing data and GDPR-focused articles from around the web, GDPR For Online Entrepreneurs encourages members to seek GDPR advice from its community’s members.

 

3. GDPR General Data Protection Regulation Technology

LinkedIn also has its own community-driven GDPR advice groups. The most active of these is the, “GDPR General Data Protection Regulation Technology”.

The group aims to be an information hub for anybody responsible for company data, including company leaders, GDPR specialists and consultants, business analysts and process experts. 

Data governance, GDPR, big data blogs

 

 

4. DBTA

Twitter | LinkedIn | Facebook

Database Trends and Applications is a publication that should be on every data professionals’ radar. Alongside news and editorials covering big data, database management, data integrations and more, DBTA is also a great source of advice for professionals looking to research buying options.

Their yearly “Trend-Setting Products in Data and Information Management” list and Product Spotlight featurettes can help data professionals put together proposals, and help give decision makers piece of mind.

 

5. Dataversity

Twitter | LinkedIn

Dataversity is another excellent source for data management and data governance related best practices and think-pieces.

In addition to hosting and sponsoring a number of live events throughout the year, the platform is a regular provider of data leadership webinars and training with a library full of webinars available on-demand.

 

6. WIRED

Twitter | LinkedIn | Facebook

Wired is a physical and digital tech magazine that covers all the bases.

For data professionals that are after the latest news and editorials pertaining to data security and a little extra – from innovations in transport to the applications of Blockchain – Wired is a great publication to keep on your radar.

 

7. TDAN

Twitter | LinkedIn | Facebook

For those looking for something a little more focused, check out TDAN. A subsidiary of Dataversity, TDAN regularly publish new editorial content covering data governance, data management, data modeling and Big Data.