Data Mining Is Becoming The New Market Research

General Jul 14, 2016

Krux's head of data science explores how data mining uses algorithms that tease out valuable and actionable insights in the same way market research has for marketers for decades,

Data Mining Post
By Yacov Salomon, Head of Data Science at Krux

You’d be hard pressed to find a marketer today who isn’t obsessed with campaign data. Thanks to digital advertising and programmatic technologies, marketing and market research now occur simultaneously. But this new age of personal 1:1 marketing also comes at a price, and if you’re a marketer for a national brand, there’s a pretty good chance you’re drowning in data.

Luckily, Big-data technologies are ready to ride to the rescue, making it possible to process the mountain of data and produce actionable insights that inform business decisions. Unfortunately, there is a class of marketing problems that even typical Big-Data technologies cannot handle on their own. Consider the problem of finding important connections between all your data points, such as studying all potential paths to conversion. This size of this calculation exceeds most systems internal computing power since there can be (literally) septillion ways (or more) to combine data points. (What’s a septillion? It’s a figure with at least 24 numbers in it such as 4,000,000,000,000,000,000,000).

But don’t be daunted by the size of the task at hand; marketing-focused data mining, the application of Machine-Learning algorithms to Big-Data, is emerging as an effective tool in the marketer’s arsenal, and national brands are learning just how powerful it can be. Data mining goes way beyond basic off-the-shelf Big-Data technologies, it requires data scientists to develop algorithms that tease out valuable and actionable insights at scale, in the same way market research has done for marketers for decades.

Let’s dig a little deeper.

Data mining is used to uncover high-dimensional correlations in massive datasets. It ploughs through that septillion of combinations looking for groups of consumers who share attributes with one another. If it finds enough people whom meet a particular set of attributes (in data science speak, if it finds “sufficient density”), and if that set of attributes is relevant to the question at hand, we deem it a statistically relevant pattern. By studying those patterns, especially those in which a conversion occurs, data scientists can pinpoint key drivers of conversion.

Because data mining operates in high-dimensional spaces, it’s pretty much impossible for brand marketers to carry it out independently. It requires computing power that rivals Google to sift through data sets with septillions of possible combinations. This is why Krux has launched our Data Science as Service (DSaaS) offering.

A Real Life Example

But that doesn’t mean there’s no role for the marketer – quite the opposite is true. When combined with the marketer’s deep domain knowledge, data mining becomes an incredibly powerful tool for uncovering deep insights and helping them better understand their customers.

Here’s a real-life example. We recently looked at a huge dataset for an automotive brand that wanted to better understand the scenarios and events that prompt consumers to download a brochure or requests a test drive via its website. These two actions are considered critical events on the path to purchase. To do this we needed to look at three buckets of data: consumers, consumer attribute and marketing touch points.

The dataset involved in this analysis was too large to analyze without the assistance of intelligent machines. Indeed, for this analysis, we had to explore 4.7e21 combinations of factors (that’s 47,000,000,000,000,000,000,000) to determine a pattern!

And yet the scope of the marketing effort was hardly extraordinary. We needed to look at combinations of 35 touch points (site visits, page visits, exposure to campaigns and channels), and 37 analytic points, which were mostly third-party segments, such as auto buyers and Android users. In terms of size and scope, the campaign was your run-of-the-mill cross-channel campaign.

Data mining allowed the auto brand to spot highly relevant patterns, which it then use to better focus marketing spend, to deliver the right message to the right consumer. For instance, the brand discovered that patterns containing a specific car-affinity targeting (example: auto buyers-of brand X) contributed to a higher download brochure rate, but not to a higher rate of requests for a test drive. From there we helped the brand deduce that there were two buckets of converters: 1) those who are inclined to begin their purchase consideration by downloading a brochure, and 2) those who begin it with a test drive.

Tapping into the brand’s domain expertise, we determined that the first group is made up of folks who care about details, and prefer to read about a model’s features prior to going through the trouble of scheduling a test drive. To drive conversions among this group, the brand targeted them with ads featuring specific models and with a link to a specs page.

The second groups wants to know how a car feels, which they can only accomplish through a test drive. If they like what they see, they’ll investigate the details. To convert members of this group, the brand targeted ads that played to their senses, and a call-to-action of scheduling a test drive.

With these insights in hand, the brand had what it needed to drive media efficiency and campaign performance.

Data is opening up a world of possibilities for marketers. The challenge is to make sense of a world defined by data abundance, and data mining will play an increasingly critical role in helping marketers navigate this data-rich environment.