Model Factory from Modern Analytics Offers High Scale Predictive Modeling for Marketers

Remember when I asked two weeks ago whether predictive models are becoming a commodity? Here’s another log for that fire: Model Factory from Modern Analytics, which promises as many models as you want for a flat fee starting at $5,000 per month. You heard that right: an all-you-can eat, fixed-price buffet for predictive models. Can free toasters* and a loyalty card be far behind?

Of course, some buffets sell better food than others. So far as I can tell, the models produced by Model Factory are quite good. But buffets also imply eating more than you should. As Model Factory’s developers correctly point out, many organizations could healthily consume a nearly unlimited number of models. Model Factory is targeted at firms whose large needs can’t be met at an acceptable cost by traditional modeling technologies. So the better analogy might be Green Revolution scientists increasing food production to feed the starving masses.

In any case, the real questions are what Model Factory does and how. The "what" is pretty simple: it builds a large number of models in a fully automated fashion. The "how" is more complicated.  Model Factory starts by importing data in known structures, so users still need to set up the initial inputs and do things like associate customer identities from different systems. Modern Analytics has staff to help with that, but it can still be a substantial task. The good news is that set-up is done only when you’re defining the modeling process or adding new sources, so the manual work isn't repeated each time a model is built or records are scored. Better still, Modern Analytics has experience connecting to APIs of common data sources such as Salesforce.com, so a new feed from a familiar source usually takes just a few hours to set up.  Model Factory stores the loaded data in its own database. This means models can use historical data without reloading all data from scratch before each update.

Once the data flow is established, users specify the file segments to model against and the types of predictions.  The predictions usually describe likelihood of actions such as purchasing a specific product but they could be something else. Again there’s some initial skilled work to define the model parameters but the process then runs automatically. During a typical run, Model Factory evaluates the input data, does data prep such as treating outliers and transforming variables, builds new models, checks each model for usable results, and scores customer records for models that pass.

The quality check is arguably the most important part of the process, because that’s what prevents Model Factory from blindly producing bad scores due to inadequate data, quality problems, or other unanticipated issues. Model Factory flags bad models – measured by traditional statistical methods like the c-score – and gives users some information their results. It’s then up to the human experts to dig further and either accept the model as is or make whatever fixes are required. Scores from passing models are pushed to client systems in files, API calls, or whatever else has been set up during implementation.

If you’ve been around the predictive modeling industry for a while, you know that automated model development has been available in different forms for long time. Indeed, Model Factory's own core engine was introduced five years ago. What made Model Factory special, then and now, is automating the end-to-end process at high scale.  How high?  There's no simple answer because the company can adjust the hardware to provide whatever performance a client requires.  In addition to hardware, performance is driven by types of models, number of records, and size of each record.  A six-processor machine working with 100,000 large records might take 2 to 40 minutes to build each model and score all records in 30 seconds per model.**

Model Factor now runs as a cloud based service, which lets users easily upgrade hardware to meet larger loads. A new interface, now in beta, lets end-users manage the modeling process and view the results.  Even with the interface, tasks such as exploring poorly performing models take serious data science skills.So it would still be wrong to think of Model Factory as a tool for the unsophisticated. Instead, consider Model Factory as a force multiplier for companies that know what they’re doing and how to do it, but can’t execute the volumes required.

Pricing for Model Factory starts at $5,000 per month for modest hardware (4 vCPU/8Gb RAM machine with 500 Gb fast storage).  Set-up tasks are covered by an implementation fee, typically around $10,000 to $20,000. Not every company will have the appetite for this sort of system, but those that do may fine Model Factory a welcome addition to their marketing technology smorgasbord.

_____________________________________________________________________________

* For the youngsters: banks used to give away free toasters to attract new customers. This was back, oh, during the 1960’s. I wasn’t there but have heard the stories.

** The exact example provided by the company was: On a 6 vCPU, 64Gb RAM machine, building 500 models on between 20K and 178K records with up to 20,000 variables per record takes an average between 2 and 40 minutes to build each model and 30 seconds per model to score all records.  This hardware configuration would cost $12,750 per month.

EverString Takes Another $65 Million and (More Important) Launches Predictive Ad Targeting Solution

EverString announced a $65 million funding round and new ad targeting product on Tuesday. (It also released a new survey on predictive marketing which is probably interesting, but I just can't face after last weekend’s data binge.)

The new funding is certainly impressive, although the record for a B2B predictive marketing vendor is apparently InsideSales’ $100 million Series C in April 2014.  It confirms that EverString has become a leader in the field despite its relatively late entry.

But the new product is what’s really intriguing. Integration between marketing and advertising technologies has now gone from astute prediction to overused cliché, so nobody gets credit for creating another example. But the new EverString product isn’t the usual sharing of a prospect list with an ad platform, as in display retargeting, Facebook Custom Audiences, or LinkedIn Lead Accelerator. Rather, it finds prospects who are not yet on the marketer’s own list by scanning ad exchanges for promising individuals. More precisely, it puts a tag on the client's Web site to capture visitor behavior, combines this with the client's CRM data and EverString's own data, and then builds a predictive model to find prospects who are similar to the most engaged current customers.  This is a form of lookalike modeling -- something that was separately mentioned to me twice this week (both times by big marketing cloud vendors), earning it the coveted Use Case of the Week Award.

Once the prospects are ranked, EverString lets users define the number of new prospects they want and set up real time bidding campaigns with the usual bells and whistles including total and daily budgets and frequency caps per individual.  EverString doesn’t identify the prospects by name, but it does figure out their employer and track their behaviors over time. If this all rings a bell, you’re on the right track: yes, EverString has created its very own combined Data Management Platform / Demand Side Platform and is using it build and target audience profiles.

In some ways, this isn’t such a huge leap: EverString and several other predictive marketing vendors have long assembled large databases of company and/or individual profiles. These were typically sourced from public information such as Web sites, job postings, and social media. Some vendors also added intent data based on visits to a network of publisher Web sites, but those networks capture a small share of total Web activity. Building a true DMP/DSP with access to the full range of ad exchange traffic is a major step beyond previous efforts. It puts EverString in competition with new sets of players, including the big marketing clouds, several of which have their own DMPs; the big data compilers; and ad targeting giants such LinkedIn, Google, and Facebook. Of course, the most direct competitors would be account based marketing vendors including Demandbase, Terminus, Azalead, Engagio, and Vendemore. While we’re at it, we could throw in the mix other DMP/DSPs such as RocketFuel, Turn, and IgnitionOne.

At this point, your inner business strategist may be wondering if EverString has bitten off more than it can chew or committed the cardinal sin of losing focus. That may turn out to be the case, but the company does have an internal logic guiding its decisions. Specifically, it sees itself as leveraging its core competency in B2B prospect modeling, by using the same models for multiple tasks including lead scoring, new prospect identification, and, now, ad targeting. Moreover, it sees these applications reinforcing each other by sharing the data they create: for example, the ad targeting becomes more effective when it can use information that lead scoring has gathered about who ultimately becomes a customer.

From a more mundane perspective, limiting its focus to B2B prospect management lets EverString concentrate its own marketing and sales efforts on a specific set of buyers, even as it slowly expands the range of problems it can help those buyers to solve. So there is considerably more going on here than a hammer looking for something new to nail.

Speaking of unrelated topics*, the EverString funding follows quickly on the heels of another large investment  – $58 million – in automated testing and personalization vendor Optimizely, which itself followed Oracle’s acquisition of Optimizely competitor Maxymiser. I’ve never thought of predictive modeling and testing as having much to do with each other, although both do use advanced analytics. But now that they’re both in the news at the same time, I’m wondering if there might be some deeper connection. After all, both are concerned with predicting behavior and, ultimately, with choosing the right treatment for each individual. This suggests that cross-pollination could result in a useful hybrid – perhaps testing techniques could help evolve campaign structures that use predictive modeling to select messages at each step. It’s a half-baked notion but does address automated campaign design, which I see as the next grand challenge for the combined martech/adtech (=madtech) industry. On a less exalted level, I suspect that automated testing and predictive modeling can be combined to give better results in their current applications than either by itself. So I’ll be keeping an eye out for that type of integration. Let me know if you spot any.

_____________________________________________________________
*lamest transition ever

Marketing Technology of the Future: Beyond the Customer Data Platform

The last three minutes of my MarTech Conference presentation are driving me crazy.

The preceding portions cover the current state of Customer Data Platforms. I have no trouble talking about that. But it somehow got into my head that the last section should look at how CDPs will fit into the long-term future of marketing technology. I have some fuzzy notions that this future martech will be radically different from today.  But to cover it succinctly I must first think it through in detail. That has been considerably harder than I expected. Here’s what I have so far.

Current Trends

These are developments happening now that will provide the context for industry changes. Some will be the topic of other MarTech presentations.

• Convergence of Adtech with Martech. These have until recently been largely separate: Adtech deals with messages to audiences whose members may share common characteristics but are not individually identified. Martech deals with known individuals. As anonymity becomes increasingly unavailable, marketers will know exactly who is receiving their advertising messages. Martech targeting techniques will therefore be used in adtech as well. Conversely, some adtech features will become standard martech practice.  More about that later.

• Contextual data. Social networks, mobile devices including phones and wearables, and the Internet of Things will provide ever-more details about the precise situation of each customer during each interaction.  Location is an obvious data point, but marketers will also know your local weather, your mood and physical condition, what you’re wearing, when and what you last ate, and whether your car needs gas. This may sound seriously creepy but the good news is you’ll get better-targeted messages.  I’d love to call this “contextual marketing” but that term is taken.

• Marketing to machines. I’m not talking about marketing with machine-generated data (e.g., your running shoes telling Nike how many miles you’ve logged) or marketing through messages on your machine (your washer suggesting you buy Tide-brand detergent).  I’m talking marketing to machines that are making purchase decisions on their own.  I discussed this last year in Do Self-Driving Cars Pick Their Own Gas Stations? and More on Marketing To Things.  Frankly, I've been surprised to see little else written on the topic. Trust me, this will be big: imagine convincing Siri to recommend your restaurant every time someone asks where they should go to lunch.

Implications for Martech

Given the trends I’ve just listed, I see martech changing significantly.

• Data synergy. I just made that term up but the idea is old: related bits of data are worth more when they’re combined. So knowing you just booked a trip to Alaska and knowing you just walked into a department store are each marginally useful by themselves, but together they let you offer me a great deal on a warm coat. It helps even more if I know you don’t already have one. The implication of this is that there’s a lot of value gained from combining data from different sources into shared repositories. It also implies there’s a lot of value in “identity association” technologies that link related data to the same person.  If you've been wondering why companies like Oracle, Acxiom, and Nielsen have been buying big data aggregators, you can stop.

• Everything is biddable. Another implication of data synergy is that each opportunity to communicate with an individual will be much more valuable to some people than others. Let’s stick with that Alaska trip: selling you a coat might be worth less than selling you a hotel room. So, when you walk into the store, the hotel chain might be willing to pay more for the chance to send you a message than the store itself. Having all the messaging devices connected to a central database and bidding system makes this possible – in fact, it already happens with real time bidding on Web ads. Now Martech platforms get to do the same. And oh, to make this work well, Martech has to hugely improve its ability to measure the actual impact of each message – so advanced, predictive attribution also plays a leading role in the martech world of the tomorrow.

• Campaigns are dead but the customer journey lives on. If each interaction is bid separately, then the notion of campaigns that lead the customer through a sequence of contacts on a flow chart is irrelevant. Honestly, I’m glad to see it go: the very first campaign management system I saw, more than twenty years ago, had exactly that sort of interface and it’s well past time for a change. But this doesn’t mean we can stop thinking about customer journeys. Any model that supports bidding on individual messages must understand where each customer is in her journey and how the message will influence the result.  Of course, different marketers will be tracking different journeys.

• Automation takes over. It’s obvious that mass data consolidation, real time bidding, optimized messaging, and omnichannel execution require near-total automation of the entire process. This has to be really intelligent automation that finds patterns, notices when they change, and optimizes treatments with minimal human guidance. And where will all that highly tailored content come from? You guessed it: automated content creation systems, which are more common and further advanced than you may realize. See Paul Roetzer’s recent post on artificial intelligence in marketing automation for an introduction to the topic  and watch Humans Need Not Apply if you want to get really scared.

• Machines buy martech. If the over-all trend is machines selling to other machines, why should  martech be left out? In fact, testing the huge number of new martech options (or even generating Scott Brinker's martech landscape supergraphic) is something machines could do really well. Once suitably open architectures are in place, it should be easy to plug in new components like a better predictive modeling system, new type of video promotion, or the latest social media app. Even if those components are less than fully automated, they could be identified, screened, integrated, and assessed with minimal human intervention.  Machines would almost surely assess results better than humans, since they’d be more objective and better able to look for subtle effects on customer behavior than human analysts.

• Humans keep things running. It’s possible that machines will eventually control every aspect of our marketing.  But I think humans will have a role to play at least for a while. This won’t necessarily be the traditional “creative” work such as copywriting and design, which machines already do better than we care to admit. But they’ll still need people to come up with non-incremental products, non-obvious insights, and deals with other organizations. Even things that machines could do better than people won’t be wholly machine-run for a while, just due to the normal lags in technology and organizational development. This may sound like a dark view of humanity’s future, but I’m more optimistic than it seems. Technology never works quite as well as promised, so I figure humans will still be needed to keep things running.

Implications for Marketers

If Martech moves in the directions I’m proposing, marketers need to do certain things to prepare.

• Expect imperfection. Sure we’ll have vastly more data than ever, but don’t assume it will be perfectly complete, accurate, or integrated. In fact, you can guarantee it won't.  Predictive models make mistakes as well. Look for systems that are designed to accommodate incomplete information, check for differences between actual and expected performance, and adapt gracefully to failures. Above all, demand transparency so you can see what the automated systems are doing and have some idea of why. This will probably require help from other systems, but make sure the monitors are as independent as possible so they’re not fooled by shared mistakes. If you’re really feeling clever, examine imperfections for opportunities – you may find bargains in bidding on messages to customers that other systems have rejected because their data is unavailable or contradictory.

• Plan for change. As anyone who has tried to modify a complex campaign workflow already knows, sophisticated systems can be brittle. High performance automated systems are likely to optimize themselves for specific conditions, which is great until those conditions change. Be sure you can easily introduce new data sources, components, objectives, and execution channels. And be sure you can always revert to a simpler, more manual mode of operation if things really go bad.

• Focus on the analytics layer.  One implication of data synergy is that the richest databases will live outside of your company’s own data center: they’ll be too big, too complicated, and updated too frequently to maintain a copy in-house. Similarly, if companies are bidding to deliver messages everywhere a customer appears, they won’t own the touchpoints. So the only piece the company can expect to own is the analytical layer – the bidding and content engines. Those engines should be freely swappable as well. What’s left to hold things together is a core of profile data shared by the analytical and content engines. This is connected to the external data store on one end and the touchpoint systems on the other. All told, it’s a rather wispy little framework, but it should be enough to provide the glue needed to link all the other components.

So, where does that leave us?  Gigantic external data pools linked to personal identities, real time bidding, messages delivered through paid channels: it's "adtech without the privacy" if you want to put it in a nutshell.  That isn't where I expected to end up, but that's exactly why I needed to write this. I can’t guarantee I won’t change my mind after further reflection, but for now I think this gives a reasonable picture of what martech might look like five or ten years from now.  In the shorter term, I still expect the central role will be played by Customer Data Platforms or (more likely) by Marketing Platforms that combine the data parts of a CDP with a multi-purpose analytical and decision layer.

Now all I have to do is figure out how to cram this into three minutes. 

...hmm...

On further reflection, it comes down to a hybrid of martech plus adtech, which is inevitably named madtech:

Wise.io Provides Another Choice for Automated Predictive Modeling

I’m beginning to feel like Lucille Ball in the chocolate factory: predictive modeling systems are coming at me faster than I can review them. I had already planned this week to write about Wise.io and then yesterday omnichannel personalization vendor Sailthru announced their own predictive solution . Now, Sailthru is interesting in its own right – it’s a Customer Data Platform with strong decisioning capabilities – but they’ll have to wait their turn. This week, I’ll stick with Wise.io.

By now, you can probably recite along with me as I list the key differentiators for predictive systems. Let’s run through them with Wise.io as the subject.

• inputs. Wise.io connects to any system with an open API, which includes most major software-as-a-service products. Vendor staff does some basic mapping for each client, which usually takes a couple of hours at most. Most of that time is spent working with the client to decide what data to include in the feed. One important feature of Wise.io is that it can handle very large numbers of inputs – hundreds or thousands of elements – so there’s not much pressure to restrict the inputs too carefully. The system can also take non-API feeds such as batch data loads, although this takes more custom work. It can handle pretty much any type of data and includes advanced natural language processing to extract information from text.

• external data. Many predictive modeling systems, especially for B2B lead scoring, supplement the client’s data with company and individual information they gather themselves from sources like social networks, Web sites, job boards, and government files. Wise.io doesn’t do this.

• data management. Wise.io maintains a database of information it has loaded from source systems. It can accept inputs from multiple sources in different formats. Data is stored on Amazon S3 and Postgres, allowing Wise.io to handle very large volumes. But the system doesn’t link records belonging to the same individual or company unless they have already been coded with a common key.

• automation. Wise.io has almost fully automated the data loading, variable selection, model building, and scoring processes. The system has sophisticated features to automatically adjust for missing values, outliers, inconsistencies, and similar real-world problems that usually require human intervention. To build a new model, users simply select the items to predict and the locations to place the results. The system’s machine learning engine automatically uses existing records in the client’s database to create the model and then places the predictions in the specified fields.

• set-up time. New clients usually have their first model within one day, assuming credentials are available to connect with source systems and the vendor and client can quickly agree on what to import. This is about as quick as it gets. While other vendors work even faster, they do this by limiting themselves to prebuilt connectors to standard systems. There’s nothing wrong with that but bear in mind that even those vendors will take longer once you start to add other inputs.


• outputs. Wise.io generates predictions, confidence scores for the predictions, and lists of drivers that show the reasons for the predictions. These are loaded into client systems where they can generate reports (see below) or be integrated with CRM or customer support agent interfaces.

• self-service.  After the initial setup, clients can build new models for themselves through a simple interface that basically involves specifying the source data, item to predict, and destination for the results. Adding a new data source would take some help from the vendor but should be pretty quick unless the source lacks a standard API or export tools.

• update frequency. Wise.io will load data in real time as it is updated in client systems, assuming the client system supports this. Scores will reflect the latest data. The system continuously and automatically updates its models to reflect new results.

• applications. Wise.io can be used for pretty much any predictive application, but the company has focused its initial efforts on customer support and retention. This involves tasks such as identifying churn risks and assigning support cases to the proper agent.

• cost. Pricing is based on the number of predictions the system generates, whether those are support tickets, email messages, or customer lists. Enterprise edition installations start in the mid-five figures (i.e., around $50,000) and can go considerably higher. A new self-service edition is limited to specific marketing automation, customer support, and CRM systems and costs somewhat less.

• vendor. The company was launched in 2013 and has some modest venture funding (published figures range from $2.5 million to $3.5 million). It has about a dozen production clients and another two dozen or so in pilot. Client include both consumer and business marketers.

SalesPredict Offers Highly Automated, Highly Flexible Predictive Modeling

A couple of weeks ago, I wrote that “predictive everywhere” is one of major trends in data-driven marketing.  I meant both that predictive models guide decisions at every stage in many marketing programs, and that models are used throughout the organization by marketing, sales, and service.

I might have added a third meaning: that systems to do predictive modeling are everywhere as well. SalesPredict is a perfect example: a small vendor with a powerful system that just launched earlier this year. Back in, say, 2008, a product like this would be big news. Today, I simply add them to my list and try to understand what makes them different.

In this case, the main technical differentiator is extreme automation: SalesPredict imports customer data, builds models, scores current records, and deploys the results with virtually no human intervention.  This is possible primarily because the painstaking work of preparing data for analysis – which is where model builders spend most of their time – is avoided by connecting to a few standard sources, currently Salesforce.com and Marketo with HubSpot soon to follow. Because it knows what to expect, the system can easily load customer data and sales results from those systems.  It then enhances the data with business and demographic information from public Web pages, social profiles, and third party sources including Zoominfo, InsideView, and Orb Intelligence.  Finally, it produces models that rank customers based on how closely they resemble members of any user-specified list, such as customers with deals that closed or who failed to renew.  Results appear as lists in a CRM interface or as scores on a marketing databaset. The whole process takes just a few hours from making the Salesforce.com connection to seeing scored records, with most of the time spent downloading CRM data and scanning the Web for other information. Once SalesPredict is installed, models are continuously updated based on new CRM information and on feedback provided by users as they review the scored records. This enables the system to automatically adjust as buyer behaviors and conditions change.

User interface is a second differentiator. CRM users see a ranked list of customer records with a system-assigned persona derived using advanced natural language processing, suggested actions such as which products to offer, and the key data values that influenced the ranking.  Users can drill further into each record to see more customer and company information including previous interactions, products owned, and won or lost deals. The company information is assembled from internal and external sources using SalesPredict’s own matching methods, so results are not at the mercy of data quality within the CRM. As previously noted, users can adjust a ranking if they feel the model is wrong; this is fed back to the system to adjust future predictions. Another screen shows which data values are most powerful in predicting success.  This helps users understand the model and suggests criteria for targeting increased marketing investment. Although there’s no great technical wizardry required to provide these interfaces (except perhaps the name and account matching), they do make results more easily understood than many other predictive modeling products.

The final differentiator is flexibility.  The system can model against any user-defined list, meaning that SalesPredict can score new leads, identify churn risk, or find the most likely buyers for new products. Recommendations also draw on a common technology, whether the system is suggesting which products a customer is most likely to buy, which content they are most likely to download, or which offers they are most likely to accept. That said, SalesPredict’s primarily integration with Salesforce.com, user interface, and company name itself suggest the vendor’s main focus is on helping sales users spend their time on the most productive lead.  This is somewhat different from predictive modeling vendors who have focused primarily on helping marketers with lead scoring.

Is SalesPredict right for you? Well, the automation and flexibility are highly attractive, but the dependence on CRM data may limit its value if you want to incorporate other sources. Pricing was originally based on the number of leads but is currently being revised, with no new details available.  However, it’s likely that the company will remain small-business-friendly in its approach. SalesPredict currently has about 15 clients, mostly in the technology industry but also with some in financial services and healthcare.

6Sense Finds B2B Prospects Using Web Site Activities

I mentioned 6Sense briefly in a recent post about vendors who help companies find prospects on the Web. Since then, I’ve had a more detailed briefing, which clarified that their scope extends well beyond prospect lists to predictive models applied across all stages of the purchase cycle. We also clarified that users can extract company-level profiles including attributes (industry, revenue, etc.) and key activities (Web site visits, topics researched) and scores at both company and individual levels.

The extraction features are important – at least to me – because they determine whether 6Sense qualifies as a “customer data platform” (CDP), a type of system I see as fundamental for future marketing. As a quick refresher, CDP is defined as “a marketer-controlled system that supports external marketing execution based on persistent, cross-channel customer data.” The part about “supports external marketing execution” is where data extraction comes in: it means that external systems can access data within the CDP for their own use. 6Sense wouldn't be a CDP if it merely displayed its data on a CRM screen without letting the CRM system import it.  If 6Sense exposed model scores but no other data, it would qualify as a CDP by the thinnest margin possible.

Of course, there are more important things about 6Sense than whether I consider it a CDP. Starting at the beginning, the system imports a list of each client’s customers and sales opportunities from CRM and marketing automation systems. Standard integrations are available for Salesforce.com, Oracle Eloqua and Marketo.  APIs can load data from other sources, potentially including other CRM marketing automation products, Web logs and tags, order processing, bookings, call centers, media impressions, and pretty much anything else.

The system standardizes and deduplicates this data at the individual and company levels. It then matches against company profiles that 6Sense itself has gathered from the usual Web sources – public social media, Web sites, job boards, directories, etc. – and from a network of third-party Web sites. The Web site network is unusual if not unique among B2B data providers; the most similar offerings I can think of are audience profiles from B2C site networks, from owners of large B2B sites, and based on other B2B activity such as email response. The advantage of Web site activity is it finds companies early in the buying cycle, when they are most open to considering new vendors. The system can map known individuals to individuals on partner Web sites, using hashing techniques to avoid passing personally identifiable information.  .

The result of all this is a database with deep company and individual profiles including both attributes and activities. 6Sense uses this to build company and individual-level predictive models.  Company models score each company’s likelihood to buy from the client.  Individual models predict the individual’s likelihood to be the best sales contact. Models are built by 6Sense staff using automated techniques and take about three weeks to complete.

The system can also estimate what product each company is most likely to purchase, when it will buy, and what stage it has reached in the buying process. Stages are defined in consultation with the client. Assignment rules might use purchase likelihood or a predictive model trained against a sample of companies in each buying stage.

Outputs from 6Sense can include lists of likely new prospect companies (not in the client’s existing database), contacts at those companies, current prospects organized by purchase stage and ranked by purchase likelihood, current contacts within each company, and key indicators that drive each company’s score. The key indicators can be very specific, such as searches for competitors’ names, visits to product detail pages, or activity by known leads.

Users can define segments based on these or other attributes and export their related data to CRM, marketing automation, ad targeting, or Web personalization systems via file transfers or API calls. 6Sense can also display the information on screen to help guide sales conversations and is now testing an extension to recommend specific talking points.  

Pricing for 6Sense starts at more than $100,000 and is based on factors including the number of models created and volume of new net contacts provided.  The company was founded in 2013 and released early versions of its product that same year. Formal release was in May 2014. It has ten current customers and more in the pipeline.

Marketing Automation News from Dreamforce: B2B More Integrated, B2C Stays Separate

I spent the early part of this week at Salesforce.com’s annual Dreamforce conference. Here are my observations.

The big news was for geeks. The main theme of the conference was Salesforce1, a new set of technologies that make it vastly easier to deliver and integrate mobile versions of Salesforce-based applications. It is apparently a major technical accomplishment and at least one of my technical friends was hugely impressed. But I can’t say I personally found it all that exciting. Perhaps we’ve reached the point where we expect technology to do pretty much everything, so the line between what's already available and what's new is only visible to experts.  Any way you slice it, focusing on platform technology is much less exciting than last year's vision of "social enterprise".

The bad news was for B2B marketing automation. Conference presentations confirmed that Pardot, the B2B marketing automation system that Salesforce acquired as part of its ExactTarget acquisition, has been separated from the rest of ExactTarget and made part of the Sales cloud. There, Pardot is described only as providing lead scoring and nurture programs, which ignores landing pages, behavior tracking, and other features that B2B marketing automation usually provides (and Pardot includes). In terms of infrastructure, Pardot will eventually work directly from the CRM data objects, rather than maintaining its own synchronized database. (Data outside the CRM structure, such as detailed Web behaviors, will remain separate.)

What this means is that Salesforce sees B2B marketing automation as just an appendage of sales automation.  This is pretty much the same constricted view of marketing automation that Salesforce management has held all along.  The logical consequence is to make lead scoring and nurture campaigns standard features within the Sales offering and discard Pardot as a separate product.  I should stress that no one at Salesforce said this was their plan, but it seems inevitable. If and when that does happen, only the most demanding companies will purchase a separate B2B marketing automation product.

To put a more optimistic spin on the same news: Salesforce will continue to let independent B2B marketing automation apps synch with Sales.  If Salesforce does merge Pardot features into its core Sales product, then marketers who have a more expansive view of B2B marketing automation functions (or who simply want a system of their own) will be forced to buy from someone else.

The interesting news was that B2C marketing automation remains separate. Salesforce’s list of business groups includes the Sales Cloud, Service Cloud, and ExactTarget Marketing Cloud. Did you notice that just one of these has its own brand? As this suggests, and conference presentations confirm, Salesforce has kept B2C marketing distinct from its Sales and Service businesses, most importantly at the data and platform levels. The ExactTarget Marketing Cloud does now include Salesforce’s previously-purchased social marketing components, Radian6 social monitoring and Social.com social advertising. It also includes the iGoDigital predictive personalization technology that came along with the ExactTarget acquisition.

Salesforce did announce some plans to integrate the Marketing cloud with Sales and Service, but they are pretty much arm’s length: Marketing can receive alerts about changes in Sales (and I assume Service) data, even though that data remains separate; Sales and Service can send emails through the ExactTarget engine; Sales and Service can receive content recommendations from the Marketing predictive modeling tool. As near as I can tell, this is the same type of API-level integration available with any third-party system. For what it’s worth, the ExactTarget Marketing Cloud APIs are also part of Salesforce1, but don’t confuse that with sharing the same underlying platform.They don't.

The good news is the B2C marketing vision. It’s not really surprising that Salesforce kept its B2C platform separate, since Salesforce's core technology isn’t engineered for the massive data volumes and analytical processing needed for B2C in general and consumer Web marketing in particular. Happily, this technical necessity is accompanied by what strikes me as a sound vision for customer management.  ExactTarget framed this around three goals: single view of the customer; managing the customer journey; and personalized content across all channels and devices. It described major features for each of these: a unified metadata layer to access (and optionally import) data from all sources; a “customer journey” engine to manage multi-step, branching flows; and predictive modeling to select the best offers and contents across email and Web messages.

This felt like a more coherent approach than Salesforce described for the Sales cloud, where external data and predictive modeling in particular were barely mentioned (or, more precisely, are still being left to App Exchange partners). The ExactTarget cloud still lacks tools to associate customer identities across email, phone, postal, social, and other systems, although there are plenty of partners to provide them. I didn’t get a close look at the details of the ExactTarget functions, which will really determine how well it competes with other customer management platforms. But the general approach makes sense.

News of the revolution may be exaggerated. Salesforce argued during the AppExchange Partner keynote that the AppExchange and Salesforce platform have created a “golden age of enterprise apps” by enabling small software developers to sell to big enterprises. One part of the argument is that the platform itself lets small vendors break through the credibility and scalability barriers that have historically protected large enterprise software vendors. The other is that end-users can purchase and deploy apps without involving the traditional gatekeepers in enterprise IT departments. A corollary to this is that end-users have different priorities than IT buyers – in particular, end users care more about ease of use – so successful software will be different.

Of course, this is exactly what the AppExchange partners wanted to hear and exactly the strategy behind Salesforce’s platform approach in the first place. But that doesn’t necessarily make it untrue: and, if correct, it would indeed be a revolution in the enterprise software industry.

But some revolutions are bigger than others.  Even in an app-based world, individual users won't be making personal decisions about how to run core business processes.  Rather, systems will be chosen at the department level because companies can more or less safely assume that whatever the department chooses will integrate smoothly with the corporate backbone. That's certainly a change but bear in mind that departmental buyers will have the same preference as corporate IT groups for working with the smallest possible number of vendors. This means there will still be the familiar tendency for individual vendors to add more functions over time. So industry dynamics may change less than you’d expect.

idio Does Sophisticated Content Recommendation

Systems in our new Guide to Customer Data Platforms range from B2B data enhancement to campaign managers to audience platforms. This may lead you to wonder whether there’s anything we actually left out.  In fact, there was: although the final choices were admittedly a bit subjective, I tried to ensure the report only included systems that met specific critieria including a persistent database, customer-level data, marketer control, and marketing-related outputs to external systems. In most cases, I could judge whether a system fit before doing a lot of detailed research. But a few systems were so close to the border that I only made the final call after I had evaluated them in depth.

idio was one of those. The company positions itself as a tool to deliver “personalized and relevant multi-channel communications”, which sure sounds like a CDP.  Indeed, it meets almost all the critieria listed above, including the most important one of building and maintaining a persistent customer database. But I ultimately excluded idio because it is tightly focused on identifying the content that customers are most likely to select, a function I felt was too narrow for a proper CDP. The folks at idio didn’t necessarily agree with this judgment, and pointed to planned developments that could indeed change the verdict (more about that later).  But, for now, let’s not worry about CDPs and take idio on its own terms.

The full description on idio's home page reads “idio understands your customer’s interests and intent through the content they consume and uses this to deliver personalized and relevant multi-channel communications” and that pretty much says it all. What idio does is ingest content – typically from a publisher such as ESPN, Virgin Media, Guardian Media, or eConsultancy (all clients) – but also from brands with large content stores such as Diageo, Unilever, and C Spire (also all clients). It uses advanced natural language processing to extract entities and concepts from this content, classifying it with the vendor’s own 23 million item taxonomy.

The system then monitors the content selected by its clients’ customers in emails, Web pages, mobile platforms, and some social platforms and builds an interest profile for each customer.  This in turn lets the system recommend which existing content the customer is most likely to select next. The recommendations are typically fed back to execution systems, such as email generators or Web content managers, which insert links to the recommended content into Web pages, emails, or newsletters.  Reports show selection rates by content, segment, or campaign, and can also show the most common topics published and the most commonly selected. Pricing is based on recommendation volume and starts around $60,000 per year for ten million recommendations.

Describing idio’s basic functions makes it sound similar to other recommendation systems, which doesn’t really do it justice. What sets idio apart are the details and technology.

• Content can include ads, offers and products as well as conventional articles.
• The natural language system classifies content without users tagging each item, a huge labor savings where massive volumes are involved, and can handle most European languages.
• idio's largest client ingests more than 1,000 items per day and stores more than one million items, a scale far beyond the reach of systems designed to choose among a couple hundred offers or products.
• Interest profiles take into account the recency of each selection and give different weights to different types of selections – e.g., more weight to sharing something than just reading it.
• Users can apply rules that limit the set of contents available in a particular situation.
• The system returns recommendations in under 50 milliseconds, which is fast enough to support online advertising selection.
• It stores customer data in a schema-less system that can make any type of input available for segmentation and reporting, although not to help with recommendations.
• It can build a master list of identifiers for each individual, allowing systems to submit any identifier and access a unified customer profile.
• It can return a content abstract, full text, images, or HTML, or simply a pointer to content stored elsewhere.
• It captures responses directly as the content is presented.

Most of these capabilities are exceptional and the combination is almost surely unique. The ultimate goal is to increase engagement by offering content people want, and idio reports it has doubled or even quadrupled selection rates vs. previous choices. All this explains why a small company whose product launched in 2011 has already landed so many large enterprises among its dozen or so clients.

Impressive as it is, I don’t see idio as a CDP because it is primarily limited to interest profiles and  content recommendations. What might yet change my mind is idio’s plan to go beyond recommending content based on likelihood of response, to recommending content based on its impact on reaching future goals such as making a purchase. The vendor promises such goal-driven recommendations in about six months.

Idio is also working on predicting future interests, based on behavior patterns of previous customers.  For example, someone buying a home might start by researching schools, then switch to real estate listings, then to mortgages, then moving companies, and so on. Those predictions could be useful in their own right and also feed predictions of future value, which could support conventional lead scoring applications. Once those features become available, idio may well be of interest to buyers well beyond its current customer base and would probably be flexible enough to serve as as Customer Data Platform.

Infer Keeps It Simple: B2B Lead Scores and Nothing Else

I’ve nearly finished gathering information from vendors for my new study on Customer Data Platform systems and have started to look for patterns in the results. One thing that has become clear is that the CDP vendors fall into several groups of systems that are similar to each other but quite different from the rest. This makes sense: most of the existing CDP systems were built to solve specific problems , not as general-purpose data platforms. Features will probably converge as vendors extend their products to attract more clients. But right now the groups are quite distinct.

One of these categories is systems for B2B lead scoring. I found three CDPs in this group: Lattice Engines (which I reviewed in April), Mintigo (reviewed in June), and Infer, which I'm reviewing right now.

Like the others, Infer builds a proprietary database of pretty much every company on the Internet by scanning Web sites, blogs, social media, government records, and other sources for company information and relevant events.  It then imports CRM and marketing automation data from its clients' systems, enhances the imported records with information from its big proprietary database, and builds predictive models that score companies and individuals on their likely win rate, conversion rate, deal size, and lifetime revenue.

The models are applied to new records as they enter a client’s system, creating scores that are returned to marketing automation and CRM to use as those systems see fit. The most typical application is deciding which leads should go to sales, be further nurtured by marketing automation,  or discarded entirely. But Infer customers also use the scores to prioritize leads for salespeople within CRM, to measure the quality of leads produced by a marketing program, assess salesperson performance based on the quality of leads they received, and even adjust paid search campaigns based on the quality of leads generated by each source and keyword.

Infer differs from its competitors in many subtle ways: the scope of its data sources, its matching processes to assemble company and individual data, the exact types of scores it produces, its modeling techniques, and reporting.  It also differs in one very obvious way: it returns only scores, while competitors return both scores and enhanced profiles on individual prospects.  Infer gathers the individual detail needed for such profiles, but has decided so far not to make them available. Its reasoning is that scores provide the major value from its system and profiles would detract from them – perhaps because sales people might ignore them scores in favor of profile data. Focusing on scores alone also makes Infer simpler to set up, operate, and understand.

Infer might be right, but it’s hard to imagine they'll will stick with this position once they start selling directly against competitors that offer scores plus profiles.  They will surely lose many deals for that reason alone.  On the other hand, Infer’s initial clients have been companies where free trials versions generate huge lead volumes, including Box, Tableau, NitroPDF, Zendesk, Jive and Yammer. Scores that accurately filter non-productive leads are more important to those companies than individual lead profiles.  Perhaps there are enough such firms for Infer to succeed by selling only to them.

Whether or not Infer expands its outputs, it faces another challenge: convincing buyers that its scores and data are better than its competitors. This might well be true: based on the information I’ve gathered, Infer seems to have a richer set of data sources and more sophisticated identity matching than at least some competitors. But my impressions may be wrong, and most buyers will won’t dig deeply enough to form an opinion.  Instead, their eyes will glaze over when the vendors start to get into the details, and they’ll simply assume that everybody’s data, matching, and modeling are roughly equivalent.

The only real way to measure relative quality is through competitive testing of which scores work better.  Each buyer needs to run her own tests since results may vary from business to business. How many buyers will take the time to do this, and which vendors will agree to cooperate, is a very open question.

That said, I did speak with some current Infer users, who were quite delighted with how easy it had been to deploy the system and with results to date. This is hardly a random sample – these were pioneer users (the system was only launched about a year ago) and hand-picked by the vendor. But their experience does confirm that performance is solid.

Infer pricing is based on the number of records processed and connected systems.  The vendor doesn’t reveal the actual rates but did say it is looking at options to make the system more affordable for smaller clients.

AgilOne Combines Marketing Database, Analytics and Execution: Yep, That's a Customer Data Platform

Well, this is embarrassing.

Here I am, all excited about discovering a new category of Customer Data Platform systems, which combine marketing database management, predictive modeling, and decision engines. Then I bump into Omer Artun, CEO of AgilOne , which he founded seven years ago to combine marketing database management, predictive modeling, and decision engines. It makes me feel much less clever.

But I guess I can’t hold that against AgilOne. As Artun tells the story, the company was created to provide marketers with a packaged, cloud-based version of the advanced data management, analytics, and execution capabilities that are usually available only to the largest and richest firms. The key is a set of 400 standard metrics, which AgilOne derives by mapping each client’s unique data into a standard structure. This, combined with advanced machine learning techniques, lets AgilOne build ten standard predictive models (engagement, next product, lifetime value, etc.) and three standard cluster models (products, behaviors, and brands) with minimal effort. The system builds on these to deliver packages of standard alerts, reports, guided analytics, individual customer profiles, and campaign lists. It also makes its data and predictions accessible to external systems such as call centers and Web sites via real time API calls, so those systems can use them to guide their own customer treatments.

This quick summary doesn’t do justice to the cleverness or sophistication of AgilOne’s approach. Clever, because the standardization allows it to quickly and cheaply deliver a full stack of capabilities, starting with database building and ending with advanced analytics, recommendations, and execution. Sophisticated, because it tailors the standard structures to each client’s business, so what it delivers isn’t some simple, cookie-cutter output.

Some of the tailoring is unavoidably manual, such as mapping client data sources to the standard data model. But much is highly automated, such as predictive models, clusters, and recommendations. I was particularly intrigued by the standard alerts, which look for significant changes in key performance indicators such as churn, margin, or average order value.  That sort of alerting is exactly what I've long felt marketers really wanted from their analytics tools.  AgilOne takes this a step further by automatically listing the data attributes with the greatest statistical impact on each item. The company refers to these items as goals to prioritize, which is a bit of a stretch – the most powerful variable isn’t necessarily the one marketers should focus on the most. But, as Damon Runyon said*, that’s the way to bet.

The system also recommends actions related to each alert, such as certain types of marketing campaigns. Again, there’s a bit less here than meets the eye, since the recommendations are drawn from a knowledgebase that’s the same for all clients. But that’s still better than nothing, and clients can customize their copy of the knowledgebase if they want.

The other especially noteworthy strength of AgilOne is data preparation. My original concept of the Customer Data Platform included customer data integration, which involves standardizing and matching customer records from different systems. I’ve pulled back from that because almost none of the vendors actually do such processing. Most assume it will be done elsewhere, or not at all, and only associate records with an exact match on a key such as a customer ID.  AgilOne does the hard stuff: quality checks, outlier detection, name parsing, address standardization, geocoding, phonetic matching, persistent ID management, and more. This is also highly automated and uses the company’s own technology. The lack of these capabilities prevents many companies from building a truly integrated customer database at many companies, so it’s extremely valuable for AgilOne to provide it.

If AgilOne has a weakness, it's at the execution end of the process.  Users can set up campaigns that generate lists on demand or on a regular schedule.  But I didn't see multi-step campaign flows or sophisticated decision management, such as arbitration across multiple eligible offers.  Some of that can probably be managed through advanced filters and custom models, which the system does provide.  However, making it truly accessible to non-technical users requires a specialized interface that the system apparently lacks.

While AgilOne just recently appeared on my personal radar, plenty of other people had already noticed: the company says nearly 100 brands are using the system. Sales efforts have been concentrated among mid-size B2C organizations, typically with at least 200,000 customers and $15 to $20 million in revenue. Pricing is published on the company Web site and is based on the features used and number of active customers. Entry price for the complete set of features starts around $9,000 per month.



_________________________________________________________________________________
*“The race is not always to the swift nor the battle to the strong, but that's the way to bet.” Runyon himself credited Chicago journalist Hugh Keough.

Mintigo InterestBase Harvests Web and Social Data for Marketing and Sales

Every marketer recognizes that the Web and social media could be rich sources of information about customers and prospects. But harvesting that data has been frustratingly difficult.  Doing it yourself  takes multiple tools to gather different kinds of information, and then patching the result together into personal profiles. Most tools do little more than keyword searches, which only capture a fraction of the potential information and only cover keywords that marketers know in advance are important.

More advanced technology does exist. Semantic engines can extract information such as executive changes and product announcements from press releases and social media profiles. Sentiment analysis can (with limited reliability) detect the attitudes that individuals express. Identity aggregators can link email, social media, and other addresses for the same individual. Predictive models can show how different attributes correlate with targeted behaviors such as purchasing a product.

Few marketers have the skill or resources to pull all these tools together for themselves. Vendors are another matter: there’s inherent scale economy to scanning the Web and social media once and applying the results to many different clients. I recently wrote about Lattice Engines,  which has assembled these pieces to create prospect lists. Infer starts with your own customer data, enhances it with information mined from the Web, and generates predictive scores.

Mintigo has also been mining Web and social data to build prospect lists, starting in 2011. This week it announced a new platform, InterestBase, that gives clients an interface to define target groups, analyze group members’ interests, push prospect lists to marketing automation and CRM systems, and enhance individual lead records.

The foundation of InterestBase is a central repository of 30 million names and 3 million companies (and growing), built by scanning Web sites and social media for job postings, product and technology names, group memberships, accounts followed, hashtags, Javascript calls, and other information. The system uses this data to assign individuals and companies such attributes as job title, company size, technologies used, hiring plans, and interest scores for products and topics. Marketers can use titles and other attributes to define their own target groups, called personas.

Lists containing members of a persona can be assigned to marketing campaigns and sent to external marketing automation or CRM systems for execution. Connectors are currently available for Marketo and Salesforce.com, with an Eloqua connector due soon. A campaign list can include the entire persona universe or a quantity specified by the user. Once the campaign is run, responders are loaded back into Mintigo and the system will identify attributes that distinguish them from non-responders.

Clients can also upload their own lists of customers or campaign respondents.  Mintigo will determine which attributes correlate with group membership, display the most important ones in reports, and use the findings in predictive models that score the entire database on likelihood of purchase or response. Clients can also upload other lists for Mintigo to enhance with its own information. This enhanced data can be used in lead scoring or to help guide salespeople.  External systems like Web sites can also accessed the data in real time via API calls.

The features are interesting, but what really matters about Mintigo is the data: fresh, powerful, and unique information about a large share of the business universe. Richer information lets Mintigo clients identify new prospects they’d otherwise miss, distinguish strong prospects from weak ones, and target messages to each prospect’s interests. The result is substantially more effective marketing and sales operations, finally letting marketers use data the Web has so tantalizingly exposed.

In case you're wondering, I do consider Mintigo a Customer Data Platform: it assembles a persistent customer database, uses predictive models to classify the members, and makes the data available to external systems for marketing execution.  

Pricing for InterestBase is based on the number of names in the client’s prospect pool, based on automated analysis of their actual customers. An average client starts around $3,000 per month.

Provenir Adds Social Listening to Customer Decisions: Another Customer Data Platform

I’m still collecting examples to illustrate my new category of Customer Data Platform (CDP) systems. The latest is Provenir, a company founded in 1992 that has long sold a system to make credit risk and fraud decisions in real time. Over the past year, the company has added “social listening” capabilities and begun offering itself to marketing agencies as a customer interaction manager. It has met with good success and is now offering its “social listening platform” more broadly. *


It’s a slight stretch to call Provenir a CDP, because it doesn’t manage a permanent customer database.  Rather, like most interaction managers, it calls data from external sources during each decision.  But Provenir does have some customer matching capabilities and stores at least some information internally. Moreover, it completely meets the other three CDP criteria: predictive modeling, real-time decisions/recommendations executed through external systems, and a non-technical user interface. It’s also sold as the “glue” connecting data sources, modeling, and execution systems, which is exactly the role played by a CDP.  So, what the heck…welcome to the club!

Provenir is organized around process flows, which cover a particular task such as reacting to a Web site visit. Users define each process by building a flow chart, or, as the cool kids call them today, a graph.** These, um, graphs***, can contain branches, loops, and other advanced structures.  The nodes can also contain other graphs that define a subprocess in more detail. Nodes can perform a wide range of operations including data gathering, calculations, updates, decisions, and messages to external systems. Although setting these up is inevitably rigorous, Provenir makes it as painless as possible by providing help such as letting users draw lines to map fields from one system to another; building rules through score cards, tables and decision trees; and warning if a flow is incomplete.

Provenir relies on external systems to assemble, integrate, and store customer data.  Users can build matching processes with system graphs, although the vendor recommends connecting to other products to load reference data or do advanced "fuzzy" matching.  Provenir can monitor source systems for selected events and issue queries to assemble data as needed. The social listening features can monitor Twitter for keywords and Tweets by specified individuals.  These can trigger process flows that can retweet a message, send a direct Twitter message to the poster, or respond through another channel. The system can also monitor and post messages on Facebook. Other channels will be added over time.

Predictive modeling in Provenir is also done in external systems. The system can import PMML code or call models in SAS, R, or even Excel. Data mapping functions can automatically extract the list of required variables from PMML, do basic transformations and calculations when loading model inputs, and manage parameters, constants, and local variables.

Decisioning is Provenir’s greatest strength. The process flow…I mean graph…is inherently very flexible, and the ability to define rules as tables, trees, score cards, and other formats adds even more power. Users can set up champion/challenger tests as splits within a process flow; results are stored in a database for analysis and reporting. Users can also build simulated data sets, containing specified distributions of particular variables, and use these to forecast results of their flow designs. Such simulation is one mark of a mature decision system.

Provenir has some built-in messaging capabilities, but most decisions are executed externally.  The system has been connected with email, Web content management, call centers, campaign management, text messaging, and other execution platforms.

Pricing for Provenir’s social listening product is based on the size of the customer database. Starting price can be as a low as several thousand dollars per month. The system is usually sold on a Software-as-a-Service (SaaS) basis, but on-premise licenses are also available.


_______________________________________________________________
* For extra credit, compare and contrast Provenir’s primary Web site  with the site for their listening division.

** Defined in Wikipedia as “mathematical structures used to model pairwise relations between objects”.

*** Would it be even cooler to call them grafs or, better still, grafz?

I've Discovered a New Class of System: the Customer Data Platform. Causata Is An Example.

It has taken me a while to connect the dots, but I’m now pretty sure I see a new type of software emerging. These systems that gather customer data from multiple sources, combine information related to the same individuals, perform predictive analytics on the resulting database, and use the results to guide marketing treatments across multiple channels. This differs quite radically from standard marketing automation systems, which use databases built elsewhere, rarely include integrated predictive modeling, and are focused primarily on moving customers through multi-step campaigns. In fact, the new systems complement rather than compete with marketing automation, which they treat as just one of several execution platforms. The new systems can also feed sales, customer service, online advertising, point of sale, and any other customer-facing systems.

Given how much vendors and analysts love to create new categories, I’m genuinely perplexed that no one has yet named this one. I’ll step in myself, and hereby christen the concept as “Customer Data Platform”.  Aside from having a relatively available three letter abbreviation (see Acronym Finder for other uses of CDP), the merits of this name include:

- “Customer” shows the scope extends to all customer-related functions, not just marketing;
- “Data” shows the primary focus is on data, not execution; and
- “Platform” shows it does more than data management while supporting other systems

But, you may ask, is this really new? Certainly systems for Customer Data Integration (CDI) have been around for decades: these include specialized products like Harte-Hanks Trillium and SAS DataFlux, CDI features within general data management suites like Informatica and Pentaho, and integration within cloud-based business intelligence products like GoodData and Birst. Many of those products have limited capabilities for working with newer data sources like Web sites and social networks, but the real distinction between them and CDPs is that the older systems are mainly designed to assemble data.  Some also provide analytics, but they don't extend to real-time decisions based on predictive models.

Similarly, there have long been specialized systems for real-time interaction management (such as Infor Interaction Advisor and Oracle Real Time Decisions) and for predictive modeling (SAS, IBM SPSS, KXEN). Some interaction managers do create predictive models, and the really big vendors (IBM, SAS, Oracle) have all three key components (CDI, real-time decisions, and predictive models) somewhere in their stables. But systems that closely couple just those features with the goal of feeding data as well as recommendations to execution systems? Those are something new.

By now, you’re probably wondering if I’ll ever get around to actually naming the vendors I have in mind. I’ve recently written about some of them, including Reachforce/SetLogik and Lattice Engines.  I also include RedPoint in the mix, because it has all the key capabilities (database development, predictive models, and real time decisions) even though it also offers conventional campaign management. Others I haven’t yet written about include Mintigo and Gainsight. Of course, each has a different mix of features and its own market position.  Indeed, several have specifically told me they do not compete with the others. Fair enough, but I still see enough similarity to group them together.

All this is a very long-winded introduction to Causata, yet another member of this new class. By now, you can probably guess Causata’s main functions: assemble customer data from multiple sources, consolidate it by customer, place it in an analytics-friendly format, run predictive models against it, and respond in real time to recommendation requests from other systems including Web sites, email, banner ads, and call centers. And you’d be right.

But that’s not the end of the story. With any product, it’s the details that matter. Causata is particularly strong in the data management department, accepting both batch and real-time data feeds and storing data as different types of events (email sent, Web site visit, call center interaction, etc.), each having predefined attributes. The system also has a particularly sophisticated “identity association” service, which looks for simultaneous events involving different identifiers as a way to link them, and can chain identifiers that were linked at different times. When I spoke with Causata about two months ago, the association rules were pretty much the same for all clients, but they promised users would get more control in the future. Users could already choose which types of associations to use in specific queries.

Causata stores the assembled data in HBase, a Hadoop-based database management system that is particularly well suited to large data volumes, many different data types, and ad hoc queries. In addition to the raw data, the system can store derived values such as aggregations (e.g., number of Web page view in past 24 hours) and model scores. Users can run SQL queries to extract data for analysis and predictive modeling in third-party software including QlikView, Tableau, SAS, and R. Prebuilt QlikView reports show the predictive power of different variables for user-specified events. The lack of native analysis and modeling tools creates some friction for users, but also lets them stick with familiar products. So the pros and cons probably cancel each other out.

The system’s decision tools are straightforward. For each situation, users define a “decision engine” that can select among multiple options, such as campaigns, products, or marketing content. These options can have qualification rules. To make a decision, the system can test the options in sequence and pick the first one for which a customer is qualified, or pick the option with the highest predictive model score. Users can also specify a percentage of customers to receive a random option, to gather data for future decisions. An engine can return multiple decisions for situations that require more than one option, such as a Web page with several offers. Causata has some machine learning algorithms to help with the decision process. It plans to expand these to automatically select the best option in a given situation.

Decision engines are called by external systems through a Web services API that can respond in under 50 milliseconds. This is fast enough to manage Web banner ads – something not all interaction managers can achieve. Model scores and other data are updated in real time during an interaction.

Causata can be deployed on-premise by a client or as a cloud-based service. The vendor says a typical implementation starts with three or four data sources and is deployed in about 30 days – very fast for this type of system. In February, Causata introduced prebuilt applications for cross-sell, acquisition, and return programs in financial services, communications, and digital media. These will further speed deployment.

Pricing is based on the number of data sources and touchpoints, with additional charges based on data storage. Cost begins around $150,000 per year.

Lattice Engines Automates All Steps in Prospect Discovery

There’s nothing new about using public information to identify business opportunities: it’s why lawyers chase ambulances and bankers phone lottery winners. But the Internet has exponentially grown the amount of data available and made it easily accessible. What’s needed to fully exploit this resource is technology that automates the end-to-end process of assembling the information, identifying opportunities, and delivering the results to sales and marketing systems.

Lattice Engines was founded in 2006 to fill this gap. The system scans public databases, company Web pages, and selected social networks to find significant events such as title changes, product launches, job openings, new locations, and investments. It supplements this with data from the clients' own systems including customer profiles, Web site visits, and purchases. It then looks at past data to find patterns which predict selected outcomes, such as making a first purchase, buying an additional product, or renewing. It uses these patterns to identify the best current prospects for each outcome, and makes the lists available to marketing systems or sales people. The sales people also see explanations of why each person was chosen, what they should be offered, and recommended talking points.

Each of these steps takes significant technology. Lattice Engines currently monitors Web sites of five to 10 million U.S. businesses, checking daily for changes.  The system’s semantic engine reads structured texts such as management biographies and press releases, extracting entities and relationships but not trying to understand more subtle meanings such as sentiment. Clients specify blogs to follow, which receive similar treatment. The company also monitors Twitter, Facebook company pages, Quora, and LinkedIn profiles of people within each sales person’s network. Additional data comes from standard sources such as business directories and from special databases requested by clients. Information from all these sources is loaded into a single database available to all Lattice Engine clients.

Lattice Engines also imports data from the clients own systems, although of course this isn’t shared with anyone else. Again, there’s some clever technology needed to recognize individuals and companies across multiple sources. Lattice Engines doesn’t try to link personal and business identities for individuals.

All this information is placed in a timeline so that modeling systems can look at events before and after the target activities. The models themselves are built automatically, once users specify the target activity, product, and time horizon. Users can then build a list of customers or prospects, have the model score it, and send high-ranking names to marketing or sales for further contact. Results can be exported to a marketing automation system or appear within the sales person’s CRM interface. Lattice Engines is directly integrated with cloud-based CRM from Salesforce.com, Microsoft Dynamics, and Oracle, and via file transfer with SAP CRM. Users can export lists to Excel and Marketo, with connectors for Eloqua and other marketing automation systems on the way.

The net result of this is a single system that performs all the tasks needed to exploit the wide range of information available about customers and prospects.  Marketers could theoretically use separate systems for each step in the process, and integrate the results for themselves.  But few really have the skills to do this.  And, in most cases, it would be more expensive than purchasing a single system like Lattice Engines.  It's particularly helpful that Lattice Engines supports both prospecting and customer management -- further reducing the need for multiple products, and further encouraging cooperation between marketing and sales departments. 

Pricing for Lattice Engines starts at $75,000 per year and grows based on the number of data sources and sales users. Client data volume doesn't affect the cost, since Lattice Engines’ own databases are vastly larger than any client data. The company has close to 50 deployments, nearly all at large B2B marketers including Dell, HP, Microsoft, ADP, and Staples.

Marketing Automation Beer Goggles: What I Think I Learned at Dreamforce

I’m writing this on my way home from Dreamforce, the Salesforce.com user conference that has become the primary industry gathering for marketing automation vendors. With a reported 90,000 attendees (I didn't count them personally), the show is fragmented into many different experiences. My own experience was mostly talking to marketing technology vendors in the exhibit hall, private meetings, and maybe a party or two. I did attend the main keynote and the “marketing cloud” announcement, but neither contained  major product news and the basic story – that social networks change everything – was true but far from novel.

So what did I learn? On reflection, there were two themes that hadn’t expected when I arrived.

The first was data. I generally think of marketing systems as relying primarily on data from the company’s own marketing, sales, and operational systems. But the exhibit hall was filled with vendors offering information – mostly from Web crawling or social media – to supplement the company’s internal resources. Of course, this isn’t new but it seems that external sources are becoming increasingly important. The main reason is so much valuable public information is now available. A lesser factor may be that there’s less internal information, at least for sales and marketing, because so many prospects engage indirectly and anonymously until deep in the buying process.

But there’s more to data than the data itself. The theme includes easier connectivity to external data, via standard connectors in general and the Salesforce.com AppExchange in particular. A closely related trend is real-time, on-demand access to the external data: say, when a salesperson views a lead record or a lead is first added to marketing automation. This requires immediate matching to find the right person in the supplier’s database, and, sure enough, matching was another popular technology on the show floor. I also saw broader use of Hadoop to handle all this new data: as you probably know, Hadoop effectively handles large volumes of unstructured and semi-structured data, so it’s a key enabling technology for data expansion. A final component is continued growth in the reporting, analytics, and predictive modeling systems that make productive use of the newly-available data.

Some products combine all these attributes, others offer a few, and some just one. Obviously a single integrated solution is easiest for the buyer, but as Scott Brinker recently pointed out in an insightful blog post, platforms like Salesforce.com may actually make it practical for marketers to mix and match individual products without the technical pain traditionally associated with integration. It therefore makes sense to view the data-related systems as a cluster of capabilities that will develop as parts of single ecosystem, collectively raising the utility and importance of external data to marketers.

The second theme, considerably less grand, was lead scoring. I suppose this is really just a subset of the analytics component of the data theme, but I saw enough new lead scoring features from enough different vendors to treat it separately. In particular, predictive modeling vendor KXEN announced a free, cloud-based service to automatically score a new Salesforce.com lead’s likelihood of converting into a contact. (If you’re not familiar with Salesforce.com terminology: contacts are linked to an account, while leads are not. The conversion usually indicates the salesperson has deemed the person a valid prospect and is thus a critical stage in most sales processes.)

The KXEN service requires absolutely no set-up; users just install it from the AppExchange. KXEN then reads the data, builds a predictive model based on past results, and returns the scores on current leads. From a technical standpoint, the modeling is nothing new, and indeed the people I met at the KXEN booth seemed to feel the product was barely worth discussing. But I’ve long felt that an automated, predictive-model-based scoring service was a major business opportunity because it would replace the time-consuming, complicated, and surely suboptimal lead scoring models that most companies now build by hand, usually with little basis in real data. Of course, there are plenty of other predictive modeling systems available for marketers, but I’m excited because I don’t think anyone else has made model-based lead scoring as simple as the KXEN offering. Maybe I need to get out more.

Speaking of which, I met SetLogik at a loud party after several glasses of wine, so I may have been wearing the marketing technology equivalent of beer goggles. But if I understood correctly, it tackles the really hard part of revenue attribution by using advanced matching technologies to connect the right leads and contacts to sales (reflected in closed opportunities in Salesforce.com). Once you’ve done that, determining which marketing touches influenced those people is relatively easy.  It’s a unique solution to a huge industry problem. Come to think of it, correct linkages are also critical for building effective lead scoring models, which it turns out that SetLogik also does. (I'll admit it: I Googled them the next day.) So they're part of that theme as well.

As I mentioned earlier, data and lead scoring were themes that emerged for me during the conference. I did have some other themes in mind when I started, which are also worth sharing. I’ll do that another day.

Finally, it’s worth noting that the conference itself was tremendously well run. It sometimes felt that one-third of those 90,000 people were Salesforce.com employees hired to stand around and answer questions. Where they found so many cheerful people outside of the Midwest I’ll never know. Congratulations and thanks to the Salesforce.com team that made it happen.