Back in March at our erworld virtual conference, I participated in a DM Radio session that discussed the reality of living in a hybrid data world. It was interesting and eye-opening in many ways, yet confirmed what I always believed – Big Data is a reality (no surprise) but companies are still living in a “Small Data” world from an operational perspective (still no surprise, but it’s good to have it confirmed).
One of the analogies I used at the time leverages my own experiences over the years in IT. In the late 80’s, the rise of distributed computing threatened to take over from the mainframe. Over 25 years later, it still is “threatening,” but the mainframe is alive and well. Why? Because both have their purposes, and every large IT shop these days lives in a hybrid world.
Skip forward a number of years to the start of virtual computing. I recall speaking with a large customer who loved using the leading virtualization tool for testing and QA duties but “never for production.” A chance follow-up meeting a couple of years later revealed that, indeed, they were now putting VMs into production “where and when it made sense.” They’re living in a hybrid world.
Finally, I thought about cloud computing and how things were a few years back. Everyone was setting up private clouds and using SaaS applications (i.e. public cloud); I was leading an initiative at erwin around assessment and management of hybrid cloud environments – whether workloads should be placed in private or public clouds. Today, customers that I speak with on this topic confirm that they look at both options. They live in a hybrid world.
This leads us back to the topic at hand – the hybrid nature of data.
I’ve spoken with data architects, data modelers, and other database professionals who have built their careers with SQL & structured databases. It’s clear by looking at market trends that there will still be an increasing demand for these skills and products to help them in the future. Companies, however, are getting interested in Big Data and what value it has to bring business insights into seemingly unrelated (and previously unavailable) data.
One of the major analyst firms publishes an annual graphic showing emerging technologies and where they fit with respect to perception vs. reality. For the first time in 2015, Big Data is no longer present on this list, which proves that it’s no longer a promise. It’s reality. By my estimate, somewhere between 10 and 15% of large organizations have some sort of NoSQL/unstructured initiative in full production. Many more are in various stages of exploration and exploitation of Big Data.
The challenge that we at erwin see is that the practices, procedures and levels of governance with “small” data don’t yet fully exist with “big” data. I saw this same thing with distributed vs. mainframe computing, virtual vs. physical computing, and cloud vs. on-premise computing. Similar challenge, similar problem.
Big Data initiatives to date have been largely focused on the “why” and “what” and not so much the “how” – how they will manage them. Govern them. That’s why we released our Data Governance edition of the erwin Web Portal; you can look at all sources regardless of the modeling tool used. Ultimately it’s not about “big” data vs. “small” data, it’s just data.
There is a French saying: “plus ça change, plus c’est la même chose.” It means “the more things change, the more things stay the same.” I’ve found this to be true of IT as well. Rather than Big Data becoming a disruptive force and a replacement for small data, it’s a complement to small data. There’s ample room for both.
So, it’s time to stop questioning where we’ll end up. We’re living in a hybrid world with a lot of opportunity for data management professionals. Good times are ahead!
