What Is So Big About Smart Data?

A couple of years ago almost no one had heard of “big data.” Now the term has become overused, and we already have to learn a new name for a similar concept. Is “smart data” just going to be another overused term that has been stripped of tangible value?

Big vs. Smart Data

Big data refers to the idea that large collections of data open opportunities that were not possible with smaller amounts. The term originated from massive offline astrophysics and genomics data sets and was later used for applications that employ machine-learning, such as Internet search, voice recognition and language translation. These days many products with data analysis and statistics features tend to use the term “big data” simply for the sake of marketing.

Smart data, on the other hand, is not about data per se, but rather refers to the ways to analyze and make sense of it. From business perspective, big data is what we know about consumer behavior, while smart data is how we discover the underlying rationale and predict repetition of such behavior.

Technically, it is about separating and ignoring the noise, finding relevant data points, and extracting signals of higher value. Practically, it is about providing meaningful information, change recommendations and interactive visualization – all for a particular business context.

In short, smart data is adding advanced business intelligence (BI) on top of big data, in order to provide actionable insights.

Business Challenges

There is a spectacular gap between what smart data means by definition and how it can really benefit your company.

As with any new business trend, the first problem is talent. There are not enough people who know how to ask the right questions – or query the data – let alone how to extract the answers. This requires a unique blend of mathematics, psychology and business acumen. McKinsey & Company estimates that, in the US alone, there is a shortfall of 190,000 analytics experts and 1.5 million data-savvy managers.

According to research by Tata Consultancy Services, the greatest challenges of getting business value from big data are as much cultural as they are technological. The highest-rated challenge is getting business units within a company to share information across the organizational silos, such as divisions, business functions, etc. This issue has plagued companies for decades. Business departments become protective of their data and often don’t have any incentive to share it internally.

Because of this, in large organizations inconsistent versions of the same data may be used in different parts of its operations. This problem can be solved with master data management that consistently provides a single point of reference. Data must be accessible by analysts across multiple domains in order for them to be most effective. By providing this access, analysts are empowered to ask broader and ultimately better questions.

Other possible organizational issues that concern decision-making workflow include selecting correct key performance indicators (KPIs) for various roles; trust between data scientists and functional managers; visualization usability and interactivity; and recording and implementing decisions in real life.

Technology Limitations

Perhaps one of the biggest challenges for big data analytics is the sheer volume, diversity and speed with which data is now being collected and processed.

Data can be analyzed in many different forms, depending on the way it is measured and its origin. With more sources stemming from web services, networks and cloud computing, the diversity of the data is growing more complex by the day. And so is the amount of work involved in its intelligent integration.

Big data is generally unstructured and requires investment in new tools, technologies, skill sets and team members to manage it. In recent research from TDWI (The Data Warehousing Institute), 88% of organizations cited structured transactional and analytical data as their primary type.

If your data sets are too large and complex to manage within traditional relational databases, then scaling of both resources and performance might be an issue. A study by Enterprise Management Associates shows that nearly half of big data projects are still based on Oracle and/or SQL servers that are incapable of managing them, and only 28% are concerned that their current systems cannot scale to meet the growing demands.

Data-Driven Marketing

The use of smart data in marketing is often called data-driven marketing. For example, if you run an ecommerce business, traffic logs represent only raw data. What you really need to know is not how many unique users visited your storefront, but rather what percentage converted to actual customers and why.

You need to ask smart questions, such as how behavior is different depending on the referral source; are there any bottleneck in the sales funnel on mobile devices; can you build up a habit of using your website; and do customers admire your product and service offering, or just cope with it? All of these questions can be answered with advanced analytics based on customers’ behavior data.

Mark Jeffery in his book, Data-Driven Marketing: The 15 Metrics Everyone in Marketing Should Know, suggests 15 main metrics of data-driven marketing: brand awareness, test-drive, churn, customer satisfaction, take rate, profit, net present value, internal rate of return, payback, customer lifetime value, cost per click, transaction conversion rate, return on ad dollar spent, bounce rate, and social media reach. HubSpot mentions six in its white paper: customer acquisition cost (CAC), marketing % of CAC, ratio of customer lifetime value (LTV) to CAC, time to payback CAC, marketing originated customer % and marketing influenced customer %.

Your business case may require other insights to make informed decisions. It is the Chief Marketing Officer’s job to take ownership and responsibility for technology issues to a new degree. No longer can one rely on cloud solutions and work around IT operation.

If we want big data to become smart, new technology should meet traditional business needs and provide answers that actually improve customer lives and experiences.

This article and more information can be found at: http://blog.nxcgroup.com/2014/what-is-so-big-about-smart-data/#.VU02bemUCP8