Tag: data management

Categories
erwin Expert Blog

Enterprise Architecture for the Cloud – Get Ahead in the Cloud

It’s almost 2018, so there’s a good chance a portion of your business relies on the cloud. You’ve either moved some legacy systems there already, or you’re adopting entirely new, cloud-based systems – or both. But what about enterprise architecture for the cloud? After all, it’s the glue that helps tie all your disparate IT threads together.

The transition to the cloud is owed heavily to improvements in its technology foundation, especially increased security and safeguards. For the likes of governments, banks and defense organizations, these enhancements have been paramount.

Additionally, organizations across industry increasingly turn to the cloud to keep costs low and maximize profits. These options are usually easier and cheaper to install than their on-premise counterparts.

A 2016 RightScale study found that 31 percent of the 1,060 IT professionals surveyed said their companies run more than 1,000 virtual machines. This demonstrates a sizeable uptick compared to 2015, when only 22 percent of participants answered the same.

The rate of adoption is even more impressive when you consider how forecasts have been outpaced. In 2014, Forrester predicted the cloud market would be worth $191 billion by 2020. In 2016, this estimate was revised to $236 billion.

The cloud is big business.

Enterprise architecture for the cloud

Why Enterprise Architecture for the Cloud?

We’ve established the case for the growing cloud market. So why is enterprise architecture for the cloud so important? To answer that question, you have to consider why the cloud market is so expansive.

In short, the answer is competition. Although there are some colossal, main players in the cloud market – Amazon Web Services (AWS), Azure, Google Cloud Platform and IBM make up more than an estimated 60% – they act as hosts to smaller cloud businesses spanning copious industries.

Unlike the hosts, this layer of the cloud market is incredibly and increasingly saturated. Such saturation is due to an even more complex web of disparate systems for which a business must account.

And said business ­must account for these systems to establish what it has right now, what it needs to reach the organization’s future state objectives, and what systems can be integrated for the sake of efficiency.

If enterprise architecture (EA) isn’t actioned in bridging the gap between an organization’s current state and its desired future state, then it’s fundamentally underperforming.

Enterprise Architecture for Introducing Cloud Systems

The above details how EA benefits a business in managing its current cloud-based systems. The primary benefit being the ability to see how and where the newer, disparate cloud systems fit with legacy systems.

But EA’s usefulness doesn’t start once cloud systems are already in place. As established, a key objective of EA is to bridge an organization’s current and future state objectives.

Another key objective is to better align an organization with IT for better preparation in the face of change. In this way, EA enables organizations to make or face enterprise changes with minimal disruptions and costs.

Cloud systems have disrupted how businesses operate already.

And with competition in the cloud space as populated as it is today, more disruption is coming. Therefore, having well-executed EA will make it easier for businesses to manage such inevitable disruption. It will also enable organizations undergoing digital transformation to implement new cloud systems with less friction.

Data-Driven Business Transformation

Categories
erwin Expert Blog

Data Modeling in a Jargon-filled World – The Logical Data Warehouse

There’s debate surrounding the term “logical data warehouse.” Some argue that it is a new concept, while others argue that all well-designed data warehouses are logical and so the term is meaningless. This is a key point I’ll address in this post.

I’ll also discuss data warehousing that incorporates some of the technologies and approaches we’ve covered in previous installments of this series (1, 2, 3, 4, 5, 6 ) but with a different architecture that embraces “any data, anywhere.”

So what is a “logical data warehouse?”

Bill Inmon and Barry Devlin provide two oft-quoted definitions of a “data warehouse.” Inmon says “a data warehouse is a subject-oriented, integrated, time-variant and non-volatile collection of data in support of management’s decision-making process.”

Devlin stripped down the definition, saying “a data warehouse is simply a single, complete and consistent store of data obtained from a variety of sources and made available to end users in a way they can understand and use in a business context.”

Although these definitions are widely adopted, there is some disparity in their interpretation. Some insist that such definitions imply a single repository, and thus a limitation.

On the other hand, some argue that a “collection of data” or a “single, complete and consistent store” could just as easily be virtual and therefore not inherently singular. They argue that the language is down to most early implementations only being single, physical data stores due to technology limitations.

Mark Beyer of Gartner is a prominent name in the former, singular repository camp. In 2011, he said “the logical data warehouse (LDW) is a new data management architecture for analytics which combines the strengths of traditional repository warehouses with alternative data management and access strategy,” and the work has since been widely circulated.

So proponents of the “logical data warehouse,” as defined by Mark Beyer, don’t disagree with the value of an integrated collection of data. They just feel that if said collection is managed and accessed as something other than a monolithic, single physical database, then it is something different and should be called a “logical data warehouse” instead of just a “data warehouse.”

As the author of a series of posts about a jargon-filled [data] world, who am I to argue with the introduction of more new jargon?

In fact, I’d be remiss if I didn’t point out that the notion of a logical data warehouse has numerous jargon-rich enabling technologies and synonyms, including Service Oriented Architecture (SOA), Enterprise Services Bus (ESB), Virtualization Layer, and Data Fabric, though the latter term also has other unrelated uses.

So the essence of a logical data warehouse approach is to integrate diverse data assets into a single, integrated virtual data warehouse, without the traditional batch ETL or ELT processes required to copy data into a single, integrated physical data warehouse.

One of the key attractions to proponents of the approach is the avoidance of recurring batch extraction, transformation and loading activities that, typically argued, cause delays and lead to decisions being made based on data that is not as current as it could be.

The idea is to use caching and other technologies to create a virtualization layer that enables information consumers to ask a question as though they were interrogating a single, integrated physical data warehouse and to have the virtualization layer (which together with the data resident in some combination of underlying application systems, IoT data streams, external data sources, blockchains, data lakes, data warehouses and data marts, constitutes the logical data warehouse) respond correctly with more current data and without having to extract, transform and load data into a centralized physical store.

Logical Data Warehouse

While the moniker may be new, the idea of bringing the query to the data set(s) and then assembling an integrated result is not a new idea. There have been numerous successful implementations in the past, though they often required custom coding and rigorous governance to ensure response times and logical correctness.

Some would argue that such previous implementations were also not at the leading edge of data warehousing in terms of data volume or scope.

What is generating renewed interest in this approach is the continued frustration on the part of numerous stakeholders with delays attributed to ETL/ELT in traditional data warehouse implementations.

When you compound this with the often high costs of large (physical) data warehouse implementations, it’s not hard to see why. Especially if it’s based on MPP hardware, juxtaposed against the promise of some new solutions from vendors like Denodo and Cisco that capitalize on the increasing prevalence of new technologies, such as the cloud and in-memory.

One topic that quickly becomes clear as one learns more about the various logical data warehouse vendor solutions is that metadata is a very important component. However, this shouldn’t be a surprise, as the objective is still to present a single, integrated view to the information consumer.

So a well-architected, comprehensive and easily understood data model is as important as ever, both to ensure that information consumers can easily access properly integrated data and because the virtualization technology itself must depend on a properly architected data model to accurately transform an information request into queries to multiple data sources and then correctly synthesize the result sets into an appropriate response to the original information request.

We hope you’ve enjoyed our series, Data Modeling in a Jargon-filled World, learning something from this post or one of the previous posts in the series (1, 2, 3, 4, 5, 6 ).

The underlying theme, as you’ve probably deduced, is that data modeling remains critical in a world in which the volume, variety and velocity of data continue to grow while information consumers find it difficult to synthesize the right data in the right context to help them draw the right conclusions.

We encourage you to read other blog posts on this site by erwin staff members and other guest bloggers and to participate in ongoing events and webinars.

If you’d like to know more about accelerating your data modeling efforts for specific industries, while reducing risk and benefiting from best practices and lessons learned by other similar organizations in your industry, please visit erwin partner ADRM Software.

Data-Driven Business Transformation

Categories
erwin Expert Blog

Data Modeling in a Jargon-filled World – The Cloud

There’s no escaping data’s role in the cloud, and so it’s crucial that we analyze the cloud’s impact on data modeling. 

Continue reading “Data Modeling in a Jargon-filled World – The Cloud”
Categories
erwin Expert Blog

Data Modeling in a Jargon-filled World – Managed Data Lakes

Managed Data Lakes

More and more businesses are adopting managed data lakes.

Earlier in this blog series, we established that leading organizations are adopting a variety of approaches to manage data, including data that may be sourced from a wide range of NoSQL, NewSQL, RDBMS and unstructured sources.

In this post, we’ll discuss managed data lakes and their applications as a hybrid of less structured data and more traditionally structured relational data. We’ll also talk about whether there’s still a need for data modeling and metadata management.

The term Data Lake was first coined by James Dixon of Pentaho in a blog entry in which he said:

“If you think of a data mart as a store of bottled water – cleansed and packaged and structured for easy consumption – the data lake is a large body of water in a more natural state. The contents of the lake stream in from a source to fill the lake, and various users of the lake can come to examine, dive in, or take samples.”

Use of the term quickly took on a life of its own with often divergent meanings. So much so that four years later Mr. Dixon felt compelled to refute some criticisms by the analyst community by pointing out that they were objecting to things he actually never said about data lakes.

However, in my experience and despite Mr. Dixon’s objections, the notion that a data lake can contain data from more than one source is now widely accepted..

Similarly, while most early data lake implementations used Hadoop with many vendors pitching the idea that a data lake had to be implemented as a Hadoop data store, the notion that data lakes can be implemented on non-Hadoop platforms, such as Azure Blob storage or Amazon S3, has become increasingly widespread.

So a data lake – as the term is widely used in 2017 – is a detailed (non-aggregated) data store that can contain structured and/or non-structured data from more than one source implemented on some kind of inexpensive, massively scalable storage platform.

But what are “managed data lakes?”

To answer that question, let’s first touch on why many early data lake projects failed or significantly missed expectations. Criticisms were quick to arise, many of which were critiques of data lakes when they strayed from the original vision, as established earlier.

Vendors seized on data lakes as a marketing tool, and as often happens in our industry, they promised it could do almost anything. As long as you poured your data into the lake, people in the organization would somehow magically find exactly the data they needed just when they needed it. As is usually the case, it turned out that for most organizations, their reality was quite different. And for three important reasons:

  1. Most large organizations’ analysts didn’t have the skillsets to wade through the rapidly accumulating pool of information in Hadoop or whichever new platforms had been chosen to implement their data lakes to locate the data they needed.
  2. Not enough attention was paid to the need of providing metadata to help people find the data they needed.
  3. Most interesting analytics are a result of integrating disparate data points to draw conclusions, and integration had not been an area of focus in most data lake implementations.

In the face of growing disenchantment with data lake implementations, some organizations and vendors pivoted to address these drawbacks. They did so by embracing what is most commonly called a managed data lake, though some prefer the label “curated data lake” or “modern data warehouse.”

The idea is to address the three criticisms mentioned above by developing an architectural approach that allows for the use of SQL, making data more accessible and providing more metadata about the data available in the data lake. It also takes on some of the challenging work of integration and transformation that earlier data lake implementations had hoped to kick down the road or avoid entirely.

The result in most implementations of a managed data lake is a hybrid that tries to blend the strengths of the original data lake concept with the strengths of traditional large-scale data warehousing (as opposed to the narrow data mart approach Mr. Dixon used as a foil when originally describing data lakes).

Incoming data, either structured or unstructured, can be easily and quickly loaded from many different sources (e.g., applications, IoT, third parties, etc.). The data can be accumulated with minimal processing at reasonable cost using a bulk storage platform such as Hadoop, Azure Blob storage or Amazon S3.

Then the data, which is widely used within the organization, can be integrated and made available through a SQL or SQL-like interface, such as those from Hive to Postgres to a tried-and-true commercial relational database such as SQL Server (or its cloud-based cousin Azure SQL Data Warehouse).

In this scenario, a handful of self-sufficient data scientists may wade (or swim or dive) in the surrounding data lake. However, most analysts in most organizations still spend most of their time using familiar SQL-capable tools to analyze data stored in the core of the managed data lake – an island in the lake if we really want to torture the analogy – which is typically implemented either using an RDBMS or a relational layer like Hive on top of the bulk-storage layer.

It’s important to note that these are not two discrete silos. Most major vendors have added capabilities to their database and BI offerings to enable analysis of both RDBMS-based and bulk-storage layer data through a familiar SQL interface.

This enables a much larger percentage of an organization’s analysts to access data both in the core and the less structured surrounding lake, using tools with which they’re already familiar.

As this hybrid managed data lake approach incorporates a relational core, robust data modeling capabilities are as important as ever. The same goes for data governance and a thorough focus on metadata to provide clear naming and definitions to assist in finding and linking with the most appropriate data.

This is true whether inside the structured relational core of the managed data lake or in the surrounding, more fluid data lake.

As you probably guessed from some of the links in this post, more and more managed data lakes are being implemented in the cloud. Please join us next time for the fifth installment in our series: Data Modeling in a Jargon-filled World – The Cloud.

Categories
erwin Expert Blog

erwin Brings NoSQL into the Enterprise Data Modeling and Governance Fold

“NoSQL is not an option — it has become a necessity to support next-generation applications.”

Continue reading “erwin Brings NoSQL into the Enterprise Data Modeling and Governance Fold”
Categories
erwin Expert Blog

Five Steps to Digital Transformation

Digital Transformation

Digital transformation is ramping up in all industries. Facing regular market disruptions, and landscape-changing technological breakthroughs, modern businesses must be both malleable and willing to change.

To stay competitive, you must be agile.

Digital Transformation is Inevitable

Increasing numbers of organizations are undergoing a digital transformation. The tried-and-tested yet rigid methods of doing business are being replaced by newer, data-orientated approaches that require thorough but fast analysis.

Some businesses – like Amazon, Netflix and Uber – are leading this evolution. They all provide very different services, but at their core, they are technology focused.

And they’re reaping rewards for it too. Amazon is one of the most valuable businesses in the world, perhaps one of the first companies to reach a $1-trillion valuation.

It’s not too late to adopt digital transformation, but it is  too late to keep fighting against it. The tide of change has quickened, and stubborn businesses could be washed away.

But what’s the best way to get started?

Step One: Determine Your End Goal

Any form of change must start with the end in mind, as it’s impossible to make a transformation without understanding why and how.

Before you make a change, big or small, you need to ask yourself why are we doing this? What are the positives and negatives? And if there are negatives, what can we do to mitigate them?

To ensure a successful digital transformation, it’s important to plot your journey from the beginning through your end goal, understanding how one change or a whole series of changes will alter your business.

Business process modeling tools can help map your digital transformation journey.

Step Two: Get Some Strategic Support

For businesses of any size, transformational change can disrupt day-to-day operations. In most organizations, the expertise to manage a sizeable transformation program doesn’t exist, and from the outset, it can appear quite daunting.

If your goal is to increase profits, it can seem contradictory to pay for support to drive your business forward. However, a slow or incorrect transformational process can be costly in many ways. Therefore, investing in support can be one of the best decisions you make.

Effective strategic planning, rooted in enterprise architecture, can help identify gaps and potential oversights in your strategy. It can indicate where investment is needed and ensure transformative endeavors aren’t undermined by false-starts and U-turns.

Many businesses would benefit further by employing strategic consultants. As experts in their fields, strategic consultants know the right questions to ask to uncover the information you need to influence change.

Their experience can support your efforts by identifying and cataloging underlying components, providing input to the project plan and building the right systems to capture important data needed to meet the business’s transformation goals.

Step Three: Understand What You Have

Once you know where you want to go, it’s important to understand what you currently do. That might seem clear, but even the smallest organizations are underpinned by thousands of business processes.

Before you decide to change something, you need to understand everything about what you currently do, or else a change could have an unanticipated and negative impact.

Enterprise architecture will also benefit a business here, uncovering strategic improvement opportunities – valuable changes you might not have seen.

As third-parties, consultants can provide an impartial view, rather than letting historic or legacy decisions cloud future judgment.

Businesses will also benefit from data modeling. This is due to the exponential increase in the volume of data businesses have to manage – as well as the variety of disparate sources.

Data modeling will ensure data is accessible, understood and better prepared for analysis and the decision-making process.

Step Four: Collect Knowledge from Within

Your employees are a wealth of knowledge and ideas, so it’s important to involve them in the enterprise architecture process.

Consultants can facilitate a series of staff workshops to enable employee insights to be shared and then developed into real, actionable changes.

Step Five: Get Buy-in Across the Business

Once you’ve engaged with your staff to collect the knowledge they hold, make sure you don’t cut them off there. Business change is only successful if everyone understands what is happening and why, with continuous updates.

Ensure that you take your employees through the change process, making them  part of the digital transformation journey.

Evidence suggests that 70 percent of all organizational change efforts fail, with a primary reason being that executives don’t get enough buy-in for new initiatives and ideas.

By involving relevant stakeholders in the strategic planning process, you can mitigate this risk. Strategic planning tools that enable collaboration can achieve this. Thanks to technological advancements in the cloud, collaboration can even be effectively facilitated online.

Take your employees through your digital transformation journey, and you’ll find them celebrating with you when you arrive at your goal.

If you think now is the right time for your business to change, get in touch with us today.

Data-Driven Business Transformation

Categories
erwin Expert Blog

What Are Customer Journey Architects, and Do You Need One?

Customer journey architects are becoming more relevant than ever before.

For businesses that want to make improvements, enterprise architecture has long been a tried and tested technique for mapping out how change should take place.

Continue reading “What Are Customer Journey Architects, and Do You Need One?”
Categories
erwin Expert Blog

Enterprise Architecture vs. Data Architecture vs. Business Process Architecture

Enterprise architects, Data architects, Process architects

Despite the nomenclature, enterprise architecture, data architecture and business process architecture are very different disciplines. Despite this, organizations that combine the disciplines enjoy much greater success in data management.

Both an understanding of the differences between the three and an understanding of how the three work together, has to start with understanding the disciplines individually:

What is Enterprise Architecture?

Enterprise architecture defines the structure and operation of an organization. Its desired outcome is to determine current and future objectives and translate those goals into a blueprint of IT capabilities.

A useful analogy for understanding enterprise architecture is city planning. A city planner devises the blueprint for how a city will come together, and how it will be interacted with. They need to be cognizant of regulations (zoning laws) and understand the current state of city and its infrastructure.

A good city planner means less false starts, less waste and a faster, more efficient carrying out of the project.

In this respect, a good enterprise architect is a lot like a good city planner.

What is Data Architecture?

The Data Management Body of Knowledge (DMBOK), define data architecture as  “specifications used to describe existing state, define data requirements, guide data integration, and control data assets as put forth in a data strategy.”

So data architecture involves models, policy rules or standards that govern what data is collected and how it is stored, arranged, integrated and used within an organization and its various systems. The desired outcome is enabling stakeholders to see business-critical information regardless of its source and relate to it from their unique perspectives.

There is some crossover between enterprise and data architecture. This is because data architecture is inherently an offshoot of enterprise architecture. Where enterprise architects take a holistic, enterprise-wide view in their duties, data architects tasks are much more refined, and focussed. If an enterprise architect is the city planner, then a data architect is an infrastructure specialist – think plumbers, electricians etc.

For a more in depth look into enterprise architecture vs data architecture, see: The Difference Between Data Architecture and Enterprise Architecture

What is Business Process Architecture?

Business process architecture describes an organization’s business model, strategy, goals and performance metrics.

It provides organizations with a method of representing the elements of their business and how they interact with the aim of aligning people, processes, data, technologies and applications to meet organizational objectives. With it, organizations can paint a real-world picture of how they function, including opportunities to create, improve, harmonize or eliminate processes to improve overall performance and profitability.

Enterprise, Data and Business Process Architecture in Action

A successful data-driven business combines enterprise architecture, data architecture and business process architecture. Integrating these disciplines from the ground up ensures a solid digital foundation on which to build. A strong foundation is necessary because of the amount of data businesses already have to manage. In the last two years, more data has been created than in all of humanity’s history.

And it’s still soaring. Analysts predict that by 2020, we’ll create about 1.7 megabytes of new information every second for every human being on the planet.

While it’s a lot to manage, the potential gains of becoming a data-driven enterprise are too high to ignore. Fortune 1000 companies could potentially net an additional $65 million in income with access to just 10 percent more of their data.

To effectively employ enterprise architecture, data architecture and business process architecture, it’s important to know the differences in how they operate and their desired business outcomes.Enterprise Architecture, Data Architecture and Business Process Architecture

Combining Enterprise, Data and Business Process Architecture for Better Data Management

Historically, these three disciplines have been siloed, without an inherent means of sharing information. Therefore, collaboration between the tools and relevant stakeholders has been difficult.

To truly power a data-driven business, removing these silos is paramount, so as not to limit the potential analysis your organization can carry out. Businesses that understand and adopt this approach will benefit from much better data management when it comes to the ‘3 Vs.’

They’ll be better able to cope with the massive volumes of data a data-driven business will introduce; be better equipped to handle increased velocity of data, processing data accurately and quickly in order to keep time to markets low; and be able to effectively manage data from a growing variety of different sources.

In essence, enabling collaboration between enterprise architecture, data architecture and business process architecture helps an organization manage “any data, anywhere” – or Any2. This all-encompassing view provides the potential for deeper data analysis.

However, attempting to manage all your data without all the necessary tools is like trying to read a book without all the chapters. And trying to manage data with a host of uncollaborative, disparate tools is like trying to read a story with chapters from different books. Clearly neither approach is ideal.

Unifying the disciplines as the foundation for data management provides organizations with the whole ‘data story.’

The importance of getting the whole data story should be very clear considering the aforementioned statistic – Fortune 1000 companies could potentially net an additional $65 million in income with access to just 10 percent more of their data.

Download our eBook, Solving the Enterprise Data Dilemma to learn more about data management tools, particularly enterprise architecture, data architecture and business process architecture, working in tandem.

Categories
erwin Expert Blog

Data-Driven Business – Changing Perspective

Data-driven business is booming. The dominant, driving force in business has arguably become a driving force in our daily lives for consumers and corporations alike.

We now live in an age in which data is a more valuable resource than oil, and five of the world’s most valuable companies – Alphabet/Google, Amazon, Apple, Facebook and Microsoft – all deal in data.

However, just acknowledging data’s value won’t do. For a business to truly benefit from its information, a change in perspective is also required. With an additional $65 million in net income available to Fortune 1000 companies that make use of just 10 percent more of their data, the stakes are too high to ignore.

Changing Perspective

Traditionally, data management only concerned data professionals. However, mass digital transformation, with data as the foundation, puts this traditional approach at odds with current market needs. Siloing data with data professionals undermines the opportunity to apply data to improve overall business performance.

The precedent is there. Some of the most disruptive businesses of the last decade have doubled down on the data-driven approach, reaping huge rewards for it.

Airbnb, Netflix and Uber have used data to transform everything, including how they make decisions, invent new products or services, and improve processes to add to both their top and bottom lines. And they have shaken their respective markets to their cores.

Even with very different offerings, all three of these businesses identify under the technology banner – that’s telling.

Common Goals

One key reason for the success of data-driven business, is the alignment of common C-suite goals with the outcomes of a data initiative.

Those goals being:

  • Identifying opportunities and risk
  • Strengthening marketing and sales
  • Improving operational and financial performance
  • Managing risk and compliance
  • Producing new products and services, or improve existing ones
  • Monetizing data
  • Satisfying customers

This list of C-suite goals is, in essence, identical to the business outcomes of a data-driven business strategy.

What Your Data Strategy Needs

In the early stages of data transformation, businesses tend to take an ad-hoc approach to data management. Although that might be viable in the beginning, a holistic data-driven strategy requires more than makeshift efforts, and repurposed Office tools .

Organizations that truly embrace data, becoming fundamentally data-driven businesses, will have to manage data from numerous and disparate sources (variety) in increasingly large quantities (volume) and at demandingly high speeds (velocity).

To manage these three Vs of data effectively, your business needs to take an “any-squared” (Any2) approach. That’s “any data” from “anywhere.”

Any2

By leveraging a data management platform with data modeling, enterprise architecture and business process modelling, you can ensure your organization is prepared to undergo a successful digital transformation.

Data modeling identifies what data you have (internal and external), enterprise architecture determines how best to use that data to drive value, and business process modeling provides understanding in how the data should be used to drive business strategy and objectives.

Therefore, the application of the above disciplines and associated tools goes a long way in achieving the common goals of C-suite executives.

For more data advice and best practices, follow us on Twitter, and LinkedIn to stay up to date with the blog.

For a deeper dive into best practices for data, its benefits, and its applications, get the FREE whitepaper below.

Data-Driven Business Transformation

Categories
erwin Expert Blog

Where to begin business process modeling?

Knowing where to begin business process modeling can seem impossible – you have a wealth of information spread out in front of you and no clue where to start. 

Continue reading “Where to begin business process modeling?”