Tag: enterprise architecture

Why Data Governance and Business Process Management Must Be Linked

Post author By Marios Forsos
Post date July 19, 2018
No Comments on Why Data Governance and Business Process Management Must Be Linked

Data governance and business process management must be linked.

Following the boom in data-driven business data governance (DG) has taken the modern enterprise by storm, garnering the attention of both the business and technical realms with an explosion of methodologies, targeted systems and training courses. That’s because a major gap needs to be addressed.

But despite all the admonitions and cautionary tales, little attention has focused on what can literally make or break any data governance initiative, turning it from a springboard for competitive advantage to a recipe for waste, anger and ultimately failure. The two key pivot points on which success hinges are business process management (BPM) and enterprise architecture. This article focuses on the critical connections between data governance and business process management.

Based on a True Story: Data Governance Without Process Is Not Data Governance

The following is based on a true story about a global pharmaceutical company implementing a cloud-based, enterprise-wide CRM system with a third-party provider.

Given the system’s nature, the data it would process, and the scope of the deployment, data security and governance was front and center. There were countless meetings – some with more than 50 participants – with protocols sent, reviewed, adjusted and so on. In fact, more than half a dozen outside security companies and advisors (and yes, data governance experts) came in to help design the perfect data protection system around which the CRM system would be implemented.

The framework was truly mind-boggling: hundreds of security measures, dozens of different file management protocols, data security software appearing every step of the way. Looking at it as an external observer, it appeared to be an ironclad net of absolute safety and effective governance.

But as the CRM implementation progressed, holes began to appear. They were small at first but quickly grew to the size of trucks, effectively rendering months of preparatory work pointless.

Detailed data transfer protocols were subverted daily by consultants and company employees who thought speed was more important than safety. Software locks and systems were overridden with passwords freely communicated through emails and even written on Post-It Notes. And a two-factor authentication principle was reduced to one person entering half a password, with a piece of paper taped over half the computer screen, while another person entered the other half of the password before a third person read the entire password and pressed enter.

While these examples of security holes might seem funny – in a sad way – when you read them here, they represent a $500,000 failure that potentially could lead to a multi-billion-dollar security breach.

Why? Because there were no simple, effective and clearly defined processes to govern the immense investment in security protocols and software to ensure employees would follow them and management could audit and control them. Furthermore, the organization failed to realize how complex this implementation was and that process changes would be paramount.

Both such failures could have been avoided if the organization had a simple system of managing, adjusting and monitoring its processes. More to the point, the implementation of the entire security and governance framework would have cost less and been completed in half the time. Furthermore, if a failure or breach were discovered, it would be easy to trace and correct.

Data Governance Starts with BPM

In a rush to implement a data governance methodology and system, you can forget that a system must serve a process – and be governed/controlled by one.

To choose the correct system and implement it effectively and efficiently, you must know – in every detail – all the processes it will impact, how it will impact them, who needs to be involved and when. Do these questions sound familiar? They should because they are the same ones we ask in data governance. They involve impact analysis, ownership and accountability, control and traceability – all of which effectively documented and managed business processes enable.

Data sets are not important in and of themselves. Data sets become important in terms of how they are used, who uses them and what their use is – and all this information is described in the processes that generate, manipulate and use them. So, unless we know what those processes are, how can any data governance implementation be complete or successful?

Consider this scenario: We’ve perfectly captured our data lineage, so we know what our data sets mean, how they’re connected, and who’s responsible for them – not a simple task but a massive win for any organization. Now a breach occurs. Will any of the above information tell us why it happened? Or where? No! It will tell us what else is affected and who can manage the data layer(s), but unless we find and address the process failure that led to the breach, it is guaranteed to happen again.

By knowing where data is used – the processes that use and manage it – we can quickly, even instantly, identify where a failure occurs. Starting with data lineage (meaning our forensic analysis starts from our data governance system), we can identify the source and destination processes and the associated impacts throughout the organization. We can know which processes need to change and how. We can anticipate the pending disruptions to our operations and, more to the point, the costs involved in mitigating and/or addressing them.

But knowing all the above requires that our processes – our essential and operational business architecture – be accurately captured and modelled. Instituting data governance without processes is like building a castle on sand.

Rethinking Business Process Management

Modern organizations need a simple and easy-to-use BPM system with easy access to all the operational layers across the organization – from high-level business architecture all the way down to data. Sure, most organizations already have various solutions here and there, some with claims of being able to provide a comprehensive picture. But chances are they don’t, so you probably need to rethink your approach.

Modern BPM ecosystems are flexible, adjustable, easy-to-use and can support multiple layers simultaneously, allowing users to start in their comfort zones and mature as they work toward the organization’s goals.

Processes need to be open and shared in a concise, consistent way so all parts of the organization can investigate, ask questions, and then add their feedback and information layers. In other words, processes need to be alive and central to the organization because only then will the use of data and data governance be truly effective.

Are you willing to think outside the traditional boxes or silos that your organization’s processes and data live in?

The erwin EDGE is one of the most comprehensive software platforms for managing an organization’s data governance and business process initiatives, as well as the whole data architecture. It allows natural, organic growth throughout the organization and the assimilation of data governance and business process management under the same platform provides a unique data governance experience because of its integrated, collaborative approach.

To learn more about erwin EDGE, and how data governance underpins and ensures data quality throughout the wider data management-suite, download our resource: Data Governance Is Everyone’s Business.

Tags enterprise architecture, business process, data management, data governance, data quality, data governance and business process, BPM, pharmaceutical data management, CRM data, security breach, poor data practices

erwin Expert Blog

The Role of An Effective Data Governance Initiative in Customer Purchase Decisions

Post author By Michael Pastore
Post date May 31, 2018
No Comments on The Role of An Effective Data Governance Initiative in Customer Purchase Decisions

A data governance initiative will maximize the security, quality and value of data, all of which build customer trust.

Without data, modern business would cease to function. Data helps guide decisions about products and services, makes it easier to identify customers, and serves as the foundation for everything businesses do today. The problem for many organizations is that data enters from any number of angles and gets stored in different places by different people and different applications.

Getting the most out of your data requires that you know what you have, where you have it, and that you understand its quality and value to the organization. This is where data governance comes into play. You can’t optimize your data if it’s scattered across different silos and lurking in various applications.

For about 150 years, manufacturers relied on their machinery and its ability to run reliably, properly and safely, to keep customers happy and revenue flowing. A data governance initiative has a similar role today, except its aim is to maximize the security, quality and value of data instead of machinery.

Customers are increasingly concerned about the safety and privacy of their data. According to a survey by Research+Data Insights, 85 percent of respondents worry about technology compromising their personal privacy. In a survey of 2,000 U.S. adults in 2016, researchers from Vanson Bourne found that 76 percent of respondents said they would move away from companies with a high record of data breaches.

For years, buying decisions were driven mainly by cost and quality, says Danny Sandwell, director of product marketing at erwin, Inc. But today’s businesses must consider their reputations in terms of both cost/quality and how well they protect their customers’ data when trying to win business.

Once the reputation is tarnished because of a breach or misuse of data, customers will question those relationships.

Unfortunately for consumers, examples of companies failing to properly govern their data aren’t difficult to find. Look no further than Under Armour, which announced this spring that 150 million accounts at its MyFitnessPal diet and exercise tracking app were breached, and Facebook, where the data of millions of users was harvested by third parties hoping to influence the 2016 presidential election in the United States.

Customers Hate Breaches, But They Love Data

While consumers are quick to report concerns about data privacy, customers also yearn for (and increasingly expect) efficient, personalized and relevant experiences when they interact with businesses. These experiences are, of course, built on data.

In this area, customers and businesses are on the same page. Businesses want to collect data that helps them build the omnichannel, 360-degree customer views that make their customers happy.

These experiences allow businesses to connect with their customers and demonstrate how well they understand them and know their preferences, like and dislikes – essentially taking the personalized service of the neighborhood market to the internet.

The only way to manage that effectively at scale is to properly govern your data.

Delivering personalized service is also valuable to businesses because it helps turn customers into brand ambassadors, and it’s a fact that it’s much easier to build on existing customer relationships than to find new customers.

Here’s the upshot: If your organization is doing data governance right, it’s helping create happy, loyal customers, while at the same time avoiding the bad press and financial penalties associated with poor data practices.

Putting A Data Governance Initiative Into Action

The good news is that 76 percent of respondents to a November 2017 survey we conducted with UBM said understanding and governing the data assets in the organization was either important or very important to the executives in their organization. Nearly half (49 percent) of respondents said that customer trust/satisfaction was driving their data governance initiatives.

What stops organizations from creating an effective data governance initiative? At some businesses, it’s a cultural issue. Both the business and IT sides of the organization play important roles in data, with the IT side storing and protecting it, and the business side consuming data and analyzing it.

For years, however, data governance was the volleyball passed back and forth over the net between IT and the business, with neither side truly owning it. Our study found signs this is changing. More than half (57 percent) of the respondents said both and IT and the business/corporate teams were responsible for data in their organization.

Once an organization understands that IT and the business are both responsible for data, it still needs to develop a comprehensive, holistic strategy for data governance that is capable of:

Reaching every stakeholder in the process
Providing a platform for understanding and governing trusted data assets
Delivering the greatest benefit from data wherever it lives, while minimizing risk
Helping users understand the impact of changes made to a specific data element across the enterprise.

To accomplish this, a modern data governance initiative needs to be interdisciplinary. It should include not only data governance, which is ongoing because organizations are constantly changing and transforming, but other disciples as well.

Enterprise architecture is important because it aligns IT and the business, mapping a company’s applications and the associated technologies and data to the business functions they enable.

By integrating data governance with enterprise architecture, businesses can define application capabilities and interdependencies within the context of their connection to enterprise strategy to prioritize technology investments so they align with business goals and strategies to produce the desired outcomes.

A business process and analysis component is also vital to modern data governance. It defines how the business operates and ensures employees understand and are accountable for carrying out the processes for which they are responsible.

Enterprises can clearly define, map and analyze workflows and build models to drive process improvement, as well as identify business practices susceptible to the greatest security, compliance or other risks and where controls are most needed to mitigate exposures.

Finally, data modeling remains the best way to design and deploy new relational databases with high-quality data sources and support application development.

Being able to cost-effectively and efficiently discover, visualize and analyze “any data” from “anywhere” underpins large-scale data integration, master data management, Big Data and business intelligence/analytics with the ability to synthesize, standardize and store data sources from a single design, as well as reuse artifacts across projects.

Michael Pastore is the Director, Content Services at QuinStreet B2B Tech. This content originally appeared as a sponsored post on http://www.eweek.com/.

Read the previous post on how compliance concerns and the EU’s GDPR are driving businesses to implement data governance.

Determine how effective your current data governance initiative is by taking our DG RediChek.

erwin Expert Blog

An Agile Data Governance Foundation for Building the Data-Driven Enterprise

Post author By Bunny Tharpe
Post date December 21, 2017
2 Comments on An Agile Data Governance Foundation for Building the Data-Driven Enterprise

The data-driven enterprise is the cornerstone of modern business, and good data governance is a key enabler.

In recent years, we’ve seen startups leverage data to catapult themselves ahead of legacy competitors. Companies such as Airbnb, Netflix and Uber have become household names. Although the service each offers differs vastly, all three identify as ‘technology’ organizations because data is integral to their operations.

As with any standard-setting revolution, businesses across the spectrum are now following these examples. But what these organizations need to understand is that simply deciding to be data-driven, or to “do Big Data,” isn’t enough.

As with any strategy or business model, it’s advisable to apply best practices to ensure the endeavor is worthwhile and that it operates as efficiently as possible. In fact, it’s especially important with data, as poorly governed data will lead to slower times to market and oversights in security. Additionally, poorly managed data fosters inaccurate analysis and poor decision-making, further hampering times to market due to inaccuracy in the planning stages, false starts and wasted cycles.

Essentially garbage in, garbage out – so it’s important for businesses to get their foundations right. To build something, you need to know exactly what you’re building and why to understand the best way to progress.

Data Governance 2.0 Is the Underlying Factor

Good data governance (DG) enables every relevant stakeholder – from executives to frontline employees – to discover, understand, govern and socialize data. Then the right people have access to the right data, so the right decisions are easier to make.

Traditionally, DG encompassed governance goals such as maintaining a business glossary of data terms, a data dictionary and catalog. It also enabled lineage mapping and policy authoring.

However, Data Governance 1.0 was siloed with IT left to handle it. Often there were gaps in context, the chain of accountability and the analysis itself.

Data Governance 2.0 remedies this by taking into account the fact that data now permeates all levels of a business. And it allows for greater collaboration.

It gives people interacting with data the required context to make good decisions, and documents the data’s journey, ensuring accountability and compliance with existing and upcoming data regulations.

But beyond the greater collaboration it fosters between people, it also allows for better collaboration between departments and integration with other technology.

By integrating data governance with data modeling (DM), enterprise architecture (EA) and business process (BP), organizations can break down inter-departmental and technical silos for greater visibility and control across domains.

By leveraging a common metadata repository and intuitive role-based and highly configurable user interfaces, organizations can guarantee everyone is singing off the same sheet of music.

Data Governance Enables Better Data Management

The collaborative nature of Data Governance 2.0 is a key enabler for strong data management. Without it, the differing data management initiatives can and often do pull in different directions.

These silos are usually born out of the use of disparate tools that don’t enable collaboration between the relevant roles responsible for the individual data management initiative. This stifles the potential of data analysis, something organizations can’t afford given today’s market conditions.

Businesses operating in highly competitive markets need every advantage: growth, innovation and differentiation. Organizations also need a complete data platform as the rise of data’s involvement in business and subsequent frequent tech advancements mean market landscapes are changing faster than ever before.

By integrating DM, EA and BP, organizations ensure all three initiatives are in sync. Then historically common issues born of siloed data management initiatives don’t arise.

A unified approach, with Data Governance 2.0 at its core, allows organizations to:

Enable data fluency and accountability across diverse stakeholders
Standardize and harmonize diverse data management platforms and technologies
Satisfy compliance and legislative requirements
Reduce risks associated with data-driven business transformation
Enable enterprise agility and efficiency in data usage.

Tags data modeling, enterprise architecture, business process, data-driven, business platform, data platform, collaboration, data governance, agile data governance, agile data management

erwin Expert Blog

Digital Trust: Enterprise Architecture and the Farm Analogy

Post author By Bunny Tharpe
Post date November 9, 2017
1 Comment on Digital Trust: Enterprise Architecture and the Farm Analogy

With the General Data Protection Regulation (GDPR) taking effect soon, organizations can use it as a catalyst in developing digital trust.

Data breaches are increasing in scope and frequency, creating PR nightmares for the organizations affected. The more data breaches, the more news coverage that stays on consumers’ minds.

The Equifax breach and subsequent stock price fall was well documented and should serve as a warning to businesses and how they manage their data. Large or small, organizations have lessons to learn when it comes to building and maintaining digital trust, especially with GDPR looming ever closer.

Previously, we discussed the importance of fostering a relationship of trust between business and consumer. Here, we focus more specifically on data keepers and the public.

Digital Trust and The Farm Analogy

Any approach to mitigating the risks associated with data management needs to consider the ‘three Vs’: variety, velocity and volume.

In describing best practices for handling data, let’s imagine data as an asset on a farm. The typical farm’s wide span makes constant surveillance impossible, similar in principle to data security.

With a farm, you can’t just put a fence around the perimeter and then leave it alone. The same is true of data because you need a security approach that makes dealing with volume and variety easier.

On a farm, that means separating crops and different types of animals. For data, segregation serves to stop those without permissions from accessing sensitive information.

And as with a farm and its seeds, livestock and other assets, data doesn’t just come in to the farm. You also must manage what goes out.

A farm has several gates allowing people, animals and equipment to pass through, pending approval. With data, gates need to make sure only the intended information filters out and that it is secure when doing so. Failure to correctly manage data transfer will leave your business in breach of GDPR and liable for a hefty fine.

Furthermore, when looking at the gates in which data enters and streams out of an organization, we must also consider the third ‘V’ – velocity, the amount of data an organization’s systems can process at any given time.

Of course, the velocity of data an organization can handle is most often tied to how efficiently a business operates. Effectively dealing with high velocities of data requires faster analysis and times to market.

However, it’s arguably a matter of security too. Although not a breach, DDOS attacks are one such vulnerability associated with data velocity.

DDOS attacks are designed to put the aforementioned data gates under pressure, ramping up the amount of data that passes through them at any one time. Organizations with the infrastructure to deal with such an attack, especially one capable of scaling to demand, will suffer less preventable down time.

Enterprise Architecture and Harvesting the Farm

Making sure you can access, understand and use your data for strategic benefit – including fostering digital trust – comes down to effective data management and governance. And enterprise architecture is a great starting point because it provides a holistic view of an organization’s capabilities, applications and systems including how they all connect.

Enterprise architecture at the core of any data-driven business will serve to identify what parts of the farm need extra protections – those fences and gates mentioned earlier.

It also makes GDPR compliance and overall data governance easier, as the first step for both is knowing where all your data is.

For more data management best practices, click here. And you can subscribe to our blog posts here.

Tags enterprise architecture, data-driven, GDPR, data management, data governance, three vs, data security, volume velocity variety, digital trust, equifax breach, data breach, data farm, DDOS

erwin Expert Blog

Enterprise Architecture for GDPR Compliance

Post author By Bunny Tharpe
Post date November 2, 2017
No Comments on Enterprise Architecture for GDPR Compliance

With the May 2018 deadline for the General Data Protection Regulation (GDPR) fast approaching, enterprise architecture (EA), should be high on the priority list for organizations that handle the personal data of citizens in any European Union state.

GDPR compliance requires an overview of why and how personal data is collected, stored, processed and accessed. It also extends to third-party access and determining – within reason – what internal or external threats exist.

Because of EA’s holistic view of an organization and its systems, enterprise architects are primed to take the lead.

Enterprise architecture for GDPR: Data privacy by design

The fragmented nature of data regulation and the discrepancies in standards from country to country made GDPR inevitable. Those same discrepancies in standards make it very likely that come May 2018, your organization will be uncompliant if changes aren’t made now.

So, organizations have two issues to tackle: 1) the finding problem and 2) the filing problem.

First, organizations must understand where all the private, personal and sensitive data is within all their systems . This also includes all the systems within their respective value chains. Hence, the finding problem.

Second, organizations must address the filing problem, which pertains to how they process data. As well as being a prerequisite for GDPR compliance, tackling the filing problem is essentially a fix to ensure the original finding problem is never as much of a headache again.

Starting with business requirements (A) and working through to product application (B), organizations have to create an environment whereby data goes from A to B via integral checkpoints to maintain data privacy.

This ensures that through every instance of the application development lifecycle – analysis, design, development, implementation and evaluation – the organization has taken all the necessary steps to ensure GDPR standards are met.

Enterprise architecture provides the framework of data privacy by design. By understanding how your organization’s systems fit together, you’ll see where data is as it moves along the application development lifecycle.

Enterprise architecture for GDPR: The benefits of collaboration

Of course, one of the requirements of GDPR is that compliance and all the steps to it can be demonstrated. Dedicated EA tools have the capacity to model the relevant information.

A dedicated and collaborative enterprise architecture tool takes things to the next level by simplifying the export and sharing of completed models.

But there’s more. Truly collaborative EA tools allow relevant stakeholders (department heads, line managers) directly involved in handling the data of interest to be involved in the modeling process itself. This leads to more accurate reporting, more reliable data, and faster turnaround, all of which have a positive effect on business efficiency and the bottom line.

Approaching GDPR compliance with enterprise architecture does more than complete a chore or tick a box. It becomes an opportunity for constant business improvement.

In other words, organizations can use enterprise architecture for GDPR as a catalyst for deeper, proactive digital transformation.

erwin partner Sandhill Consultants has produced a three-part webinar series on Navigating the GDPR Waters.

The first webinar covers the identification and classification of personally identifiable information and sensitive information and technologies, such as enterprise architecture, that can assist in identifying and classifying this sort of data.

Click here to access this webinar.

Tags enterprise architecture, GDPR, General Data Protection Regulation, data management, enterprise architecture collaboration, GDPR date, data privacy, gdpr privacy, navigating the GDPR waters, Sandhill, Sandhill GDPR

erwin Expert Blog

Using Enterprise Architecture to Improve Security

Post author By Bunny Tharpe
Post date September 21, 2017
No Comments on Using Enterprise Architecture to Improve Security

The personal data of more than 143 million people – half the United States’ entire population – may have been compromised in the recent Equifax data breach. With every major data breach comes post-mortems and lessons learned, but one area we haven’t seen discussed is how enterprise architecture might aid in the prevention of data breaches.

For Equifax, the reputational hit, loss of profits/market value, and potential lawsuits is really bad news. For other organizations that have yet to suffer a breach, be warned. The clock is ticking for the General Data Protection Regulation (GDPR) to take effect in May 2018. GDPR changes everything, and it’s just around the corner.

Organizations of all sizes must take greater steps to protect consumer data or pay significant penalties. Negligent data governance and data management could cost up to 4 percent of an organization’s global annual worldwide turnover or up to 20 million Euros, whichever is greater.

With this in mind, the Equifax data breach – and subsequent lessons – is a discussion potentially worth millions.

Proactive Data Protection and Cybersecurity

Given that data security has long been considered paramount, it’s surprising that enterprise architecture is one approach to improving data protection that has been overlooked.

It’s a surprise because when you consider enterprise architecture use cases and just how much of an organization it permeates (which is really all of it), EA should be commonplace in data security planning.

So, the Equifax breach provides a great opportunity to explore how enterprise architecture could be used for improving cybersecurity.

Security should be proactive, not reactive, which is why EA should be a huge part of security planning. And while we hope the Equifax incident isn’t the catalyst for an initial security assessment and improvements, it certainly should prompt a re-evaluation of data security policies, procedures and technologies.

By using well-built enterprise architecture for the foundation of data security, organizations can help mitigate risk. EA’s comprehensive view of the organization means security can be involved in the planning stages, reducing risks involved in new implementations. When it comes to security, EA should get a seat at the table.

Enterprise architecture also goes a long way in nullifying threats born of shadow IT, out-dated applications, and other IT faux pas. Well-documented, well-maintained EA gives an organization the best possible view of current tech assets.

This is especially relevant in Equifax’s case as the breach has been attributed to the company’s failure to update a web application although it had sufficient warning to do so.

By leveraging EA, organizations can shore up data security by ensuring updates and patches are implemented proactively.

Enterprise Architecture, Security and Risk Management

But what about existing security flaws? Implementing enterprise architecture in security planning now won’t solve them.

An organization can never eliminate security risks completely. The constantly evolving IT landscape would require businesses to spend an infinite amount of time, resources and money to achieve zero risk. Instead, businesses must opt to mitigate and manage risk to the best of their abilities.

Therefore, EA has a role in risk management too.

In fact, EA’s risk management applications are more widely appreciated than its role in security. But effective EA for risk management is a fundamental part of how EA for implementing security works.

Enterprise architecture’s comprehensive accounting of business assets (both technological and human) means it’s best placed to align security and risk management with business goals and objectives. This can give an organization insight into where time and money can best be spent in improving security, as well as the resources available to do so.

This is because of the objective view enterprise architecture analysis provides for an organization.

To use somewhat of a crude but applicable analogy, consider the risks of travel. A fear of flying is more common than fear of driving in a car. In a business sense, this could unwarrantedly encourage more spending on mitigating the risks of flying. However, an objective enterprise architecture analysis would reveal, that despite fear, the risk of travelling by car is much greater.

Applying the same logic to security spending, enterprise architecture analysis would give an organization an indication of how to prioritize security improvements.

Tags enterprise architecture, GDPR, Data protection, data management, data governance, Equifax Data Breach, Cybersecurity, risk, compliance, risk management

erwin Expert Blog

Enterprise Architecture for the Cloud – Get Ahead in the Cloud

Post author By Bunny Tharpe
Post date September 14, 2017
No Comments on Enterprise Architecture for the Cloud – Get Ahead in the Cloud

It’s almost 2018, so there’s a good chance a portion of your business relies on the cloud. You’ve either moved some legacy systems there already, or you’re adopting entirely new, cloud-based systems – or both. But what about enterprise architecture for the cloud? After all, it’s the glue that helps tie all your disparate IT threads together.

The transition to the cloud is owed heavily to improvements in its technology foundation, especially increased security and safeguards. For the likes of governments, banks and defense organizations, these enhancements have been paramount.

Additionally, organizations across industry increasingly turn to the cloud to keep costs low and maximize profits. These options are usually easier and cheaper to install than their on-premise counterparts.

A 2016 RightScale study found that 31 percent of the 1,060 IT professionals surveyed said their companies run more than 1,000 virtual machines. This demonstrates a sizeable uptick compared to 2015, when only 22 percent of participants answered the same.

The rate of adoption is even more impressive when you consider how forecasts have been outpaced. In 2014, Forrester predicted the cloud market would be worth $191 billion by 2020. In 2016, this estimate was revised to $236 billion.

The cloud is big business.

Why Enterprise Architecture for the Cloud?

We’ve established the case for the growing cloud market. So why is enterprise architecture for the cloud so important? To answer that question, you have to consider why the cloud market is so expansive.

In short, the answer is competition. Although there are some colossal, main players in the cloud market – Amazon Web Services (AWS), Azure, Google Cloud Platform and IBM make up more than an estimated 60% – they act as hosts to smaller cloud businesses spanning copious industries.

Unlike the hosts, this layer of the cloud market is incredibly and increasingly saturated. Such saturation is due to an even more complex web of disparate systems for which a business must account.

And said business must account for these systems to establish what it has right now, what it needs to reach the organization’s future state objectives, and what systems can be integrated for the sake of efficiency.

If enterprise architecture (EA) isn’t actioned in bridging the gap between an organization’s current state and its desired future state, then it’s fundamentally underperforming.

Enterprise Architecture for Introducing Cloud Systems

The above details how EA benefits a business in managing its current cloud-based systems. The primary benefit being the ability to see how and where the newer, disparate cloud systems fit with legacy systems.

But EA’s usefulness doesn’t start once cloud systems are already in place. As established, a key objective of EA is to bridge an organization’s current and future state objectives.

Another key objective is to better align an organization with IT for better preparation in the face of change. In this way, EA enables organizations to make or face enterprise changes with minimal disruptions and costs.

Cloud systems have disrupted how businesses operate already.

And with competition in the cloud space as populated as it is today, more disruption is coming. Therefore, having well-executed EA will make it easier for businesses to manage such inevitable disruption. It will also enable organizations undergoing digital transformation to implement new cloud systems with less friction.

erwin Expert Blog

Data Modeling in a Jargon-filled World – In-memory Databases

Post author By Kevin Schofield
Post date August 24, 2017
No Comments on Data Modeling in a Jargon-filled World – In-memory Databases

With the volume and velocity of data increasing, in-memory databases provide a way to keep processing speeds low.

Traditionally, databases have stored their data on mechanical storage media such as hard disks. While this has contributed to durability, it’s also constrained attainable query speeds. Database and software designers have long realized this limitation and sought ways to harness the faster speeds of in-memory processing.

The traditional approach to database design – and analytics solutions to access them – includes in-memory caching, which retains a subset of recently accessed data in memory for fast access. While caching often worked well for online transaction processing (OLTP), it was not optimal for analytics and business intelligence. In these cases, the most frequently accessed information – rather than the most recently accessed information – is typically of most interest.

That said, loading an entire data warehouse or even a large data mart into memory has been challenging until recent years.

There are a few key factors in making in-memory databases and analytics offerings relevant for more and more use cases. One such factor has been the shift to 64-bit operating systems. Another is that it makes available much more addressable memory. And as one might assume, the availability of increasingly large and affordable memory solutions has also played a part.

Database and software developers have begun to take advantage of in-memory databases in a myriad of ways. These include the many key-value stores such as Amazon DynamoDB, which provide very low latency for IoT and a host of other use cases.

Another way businesses are taking advantage of in-memory is through distributed in-memory NoSQL databases such as Aerospike, to in-memory NewSQL databases such as VoltDB. However, for the remainder of this post, we’ll touch in more detail on several solutions with which you might be more familiar.

Some database vendors have chosen to build hybrid solutions that incorporate in-memory technologies. They aim to bridge in-memory with solutions based on tried-and-true, disk-based RDBMS technologies. Such vendors include Microsoft with its incorporation of xVelocity into SQL Server, Analysis Services and PowerPivot, and Teradata with its Intelligent Memory.

Other vendors, like IBM with its dashDB database, have chosen to deploy in-memory technology in the cloud, while capitalizing on previously developed or acquired technologies (in-database analytics from Netezza in the case of dashDB).

However, probably the most high-profile application of in-memory technology has been SAP’s significant bet on its HANA in-memory database, which first shipped in late 2010. SAP has since made it available in the cloud through its SAP HANA Cloud Platform, and on Microsoft Azure and it has released a comprehensive application suite called S/4HANA.

Like most of the analytics-focused in-memory databases and analytics tools, HANA stores data in a column-oriented, in-memory database. The primary rationale for taking a column-oriented approach to storing data in memory is that in analytic use cases, where data is queried but not updated, it allows for often very impressive compression of data values in each column. This means much less memory is used, resulting in even higher throughput and less need for expensive memory.

So what approach should a data architect adopt? Are Microsoft, Teradata and other “traditional” RDBMS vendors correct with their hybrid approach?

As memory gets cheaper by the day, and the value of rapid insights increases by the minute, should we host the whole data warehouse or data mart in-memory as with vendors SAP and IBM?

It depends on the specific use case, data volumes, business requirements, budget, etc. One thing that is not in dispute is that all the major vendors recognize that in-memory technology adds value to their solutions. And that extends beyond the database vendors to analytics tool stalwarts like Tableau and newer arrivals like Yellowfin.

It is incumbent upon enterprise architects to learn about the relative merits of the different approaches championed by the various vendors and to select the best fit for their specific situation. This is something that’s admittedly, not easy given the pace of adoption of in-memory databases and the variety of approaches being taken.

But there’s a silver lining to the creative disruption caused by the increasing adoption of in-memory technologies. Because of the sheer speed the various solutions offered, many organizations are finding that the need to pre-aggregate data to achieve certain performance targets for specific analytics workloads is disappearing. The same goes for the need to de-normalize database designs to achieve specific analytics performance targets.

Instead, organizations are finding that it’s more important to create comprehensive atomic data models that are flexible and independent of any assumed analytics workload.

Perhaps surprisingly to some, third normal form (3NF) is once again not an unreasonable standard of data modeling for modelers who plan to deploy to a pure in-memory or in-memory-augmented platform.

Organizations can forgo the time-consuming effort to model and transform data to support specific analytics workloads, which are likely to change over time anyway. They also can stop worrying about de-normalizing and tuning an RDBMS for those same fickle and variable analytics workloads, focusing on creating a logical data model of the business that reflects the business information requirements and relationships in a flexible and detailed format, that doesn’t assume specific aggregations and transformations.

The blinding speed of in-memory technologies provides the aggregations, joins and other transformations on the fly, without the onerous performance penalties we have historically experienced with very large data volumes on disk-only-based solutions. As a long-time data modeler, I like the sound of that. And so far in my experience with many of the solutions mentioned in this post, the business people like the blinding speed and flexibility of these new in-memory technologies!

Please join us next time for the final installment of our series, Data Modeling in a Jargon-filled World – The Logical Data Warehouse. We’ll discuss an approach to data warehousing that uses some of the technologies and approaches we’ve discussed in the previous six installments while embracing “any data, anywhere.”