How we use data and machine learning to solve the lead quality problem

This post originally appeared on Clearbit’s Blog.

When Simon Whittick joined Geckoboard as its first VP of Marketing, he took all the standard steps to attract more visitors to their site, convert them, and grow the SaaS company’s revenue. He and his team wrote content for their popular blog, ran paid advertising campaigns, and set up email nurture campaigns. At the end of his first year, he was as successful as almost any other marketing executive in the industry. The site was attracting hundreds of thousands of visitors every month, and the business was booking millions in annual recurring revenue. But unknowingly, his success was driving one of his coworkers crazy.

While 10,000 leads a month earned Whittick applause at the company’s weekly all-hands meeting, it was keeping Geckoboard’s only sales development rep (SDR), Alex Bates, at the office on nights and weekends. Many of the inbound leads were self-serve customers who required no conversation with sales, or tire kickers who were not ready to buy. This left Alex manually qualifying leads and wasting tons of his time.

As a result, Geckoboard’s sales efficiency—one of the most critical metrics for any company—was slumping. In other words, Whittick wasn’t only driving a junior sales rep crazy; he was leaving money on the table.

Over the course of the next year, Whittick built a data-backed machine learning process to solve his company’s lead-qualification problems. In the process, he turned Bates into not only an adoring fan of his, but a one-man sales team as efficient as a typical ten-person SDR team. Without any technical background, Whittick figured out a way to change the shape of his company using data and a bit of machine learning.

One day toward the end of last year, Bates and Whittick sat down to discuss how they could solve their lead-quality problem. They had close to 10,000 leads coming in each month, but they needed to figure out which of those leads to send to sales. Their first instinct was to refine their ideal customer profile. They’d both read all the sales and marketing blogs preaching its importance. They started with a Ideal Customer Profile based on some simple audience rules.

On paper, Geckoboard’s ideal customer was a software company with more than 100 employees; they typically sold to a director or VP. But the truth was that a lot of companies outside that explicit profile would be great customers. For example, their initial model excluded a company with 95 employees even if it looked almost identical to one of their best customers. When they looked at their past data, they learned that leads in what they believed to be their ideal customer profile converted at twice the rate. But they only accounted for 0.7% of the conversions. They needed a more nuanced and flexible inbound model.


Prior to joining the Geckoboard team, Whittick had worked for Marin Software. While he was there, he began to notice a shift in the way companies approached marketing. The most forward-thinking companies had begun to hire technical employees with degrees in mathematics instead of business. He heard stories of companies that were replacing entire teams and doubling revenue by using publicly (or privately) available information and crunching it to their advantage. As time went on, many of those employees left their jobs to provide the same level of automation to smaller companies without the budget to hire data scientists.

Between his time at Marin Software and Geckoboard, dozens of startups popped up to help larger companies embrace the data revolution. Companies like Clearbit mined the web for demographic and firmographic data that could be used to better filter leads. My own company, MadKudu, makes it possible to pull insights from that data without having a PhD in computer science. By 2016, the entire marketing technology landscape had shifted. With an executive team that embraced innovation and big bets, Whittick decided to make it Geckoboard’s competitive advantage.

The first step Whittick took was to develop his own flexible point-based scoring system. Previously a lead was either given a 1 or a 0. A lead was either a software company with 100 or more employees or it wasn’t. It was binary. The obvious problem of this model was that a company with 95 employees would be excluded. In addition, a company with 100 employees was given the same score as a company with 1,000 employees, even though the latter was twice as valuable.

In his new model, Whittick gave leads a score based on multiple criteria. For example, he’d give a lead 50 points for having 500 employees or more, and negative 2 points if it had less than 10 employees. A director-level job title would receive 10 points, whereas a manager would only receive 3. This created an exponentially larger score range, which meant that Bates could prioritize leads. If he called the top score leads, he’d have the option to call B-tier leads. The model was weighted toward the large accounts Geckoboard knew could significantly impact revenue. For example, a US-based real estate company with 500 employees and a director buyer would be routed to the top of Alex’s lead list, even though it didn’t fit the software industry criteria.


This new model was similar to the way SDRs have scored leads for over a decade, only more efficient. Prior to automated lead scoring, sales reps were told by their managers to prioritize leads based on four criteria: budget, authority, need, and timing (or as it’s commonly referred to, BANT). This method is more flexible than a rigid ideal customer profile, but it is only as strong as the rep behind it. Human error, irrational judgment, and varying levels of experience lead to a process with little rhyme or reason. That’s why Whittick chose to automate the task and take humans out of the process entirely.


Immediately the company began to see results from their lead-scoring investment. Within the first month, leads were converting at twice the rate. As a result, Bates was spending less time to close more deals. Sales efficiency—revenue collected divided by the time and resources to earn it—rose significantly. Still, Whittick knew he could improve the results and save Bates even more time.

One of the biggest shifts that Whittick saw in the technology industry was the speed at which data could be put to use as a result of new tools. In the old world that he inhabited, a lead couldn’t be scored until it hit a company’s CRM. Enrichment software took hours to append valuable data to a lead.

That information could be sent to the CRM and the lead scored accordingly before the visitor began typing in the next text box.

After his first lead scoring success, Whittick decided to make another bet. Bates frequently complained about leads that were obviously bad fits—the type of conversation that takes 30 seconds to know there isn’t a mutual fit. Many of the companies were too small to need sophisticated dashboards yet. Whittick enlisted one of the company’s front-end developers to help him solve the problem. They built logic into the product demo request page that would ask for a visitor’s email address and then, before sending them to the next page, score the lead. On the back end, additional information would be appended to the lead using Clearbit, and it would be run through MadKudu’s scoring algorithm. If it received a high-enough score, the next page would ask for the lead’s phone number and tell them to expect a call shortly; if the score was low, they’d be routed through a low-touch email cadence. It was radically successful.

madkudu form demo geckoboard

Before implementing their real-time lead scoring solution, only about 15% of Bates’ conversations were meaningful. The new website logic meant that he could cut 85% of the calls he took every day and focus on higher quality accounts. Once again, sales efficiency increased significantly.

In addition to the speed at which information could be appended, processed, and acted on, Whittick saw another change in the marketing technology world: there was suddenly more data than most companies knew what to do with. Marketers could know what CRM, email server, and chat service a company used. They could know when a company was hiring a new employee, when they were written about by a major news outlet, and how much money they’d raised. It was overwhelming. But thanks to tools like Segment, marketers could pipe all that data into a CRM or marketing automation system and act on it. Then they could combine it with information like how frequently someone visited their own site, how often they emailed sales or support, and when they went through a specific part of the onboarding process. For a data-driven marketer like Whittick, this new world was utopia.

In conversations with Bates, Whittick learned that the best leads were ones that went through their onboarding process pre-sales conversation. During the Geckoboard free trial, users were prompted to build their first dashboard, connect a data source, and upload their company’s logo and color palette. As is the case with many SaaS solutions, most users dropped off before completing all the steps. Those users weren’t ready for a conversation with sales. But when Bates was looking at his lead list, he had no way of knowing whether or not a free trial user had completed onboarding. As a result, he was spending at least half of his time with people that weren’t ready to talk or buy.

Combining usage data from the website and their app, Whittick set out to refine the lead scoring model even further. Each time a free-trial user completed a step in the onboarding process, it was recorded and sent back to the CRM using Segment. The model would then give that lead a couple of additional points. If the user completed all of the steps, bonus points would be added and the lead would be raised to the top of Bates’ lead queue in Salesforce. Again, Bates began spending less time talking to customers prematurely and more time having high-quality conversations that led to revenue. Whittick had figured out how to save the sales team time and increase sales efficiency further.


But while Whittick and Bates were celebrating their improved conversion rate success, a new problem was emerging. By the summer of 2016, they had enlisted my team at MadKudu to automate their lead scoring. Rather than manually analyzing conversion data and adjusting their lead scoring model accordingly, our machine learning tool was built to do all the work for them. There was a small problem. Today, machine learning algorithms are only as strong as the humans instructing them. In other words, they are incredibly efficient at analyzing huge sets of data and optimizing toward an end result, but a human is responsible for setting that end result. Early on, Whittick set up the model so that it would optimize for the shortest possible sales cycle and the highest account value. He didn’t, however, instruct it to account for churn, an essential metric for any SaaS company. As a result, the model was sending Bates leads that closed quickly, but dropped the service fast too. Fortunately, the solution was simple.

After learning about the problems with his model, Whittick instructed MadKudu’s algorithm to analyze customers by lifetime value (LTV) and adjust the model to optimize for that. He also instructed it to analyze the accounts that churned quickly and score leads that looked like this negatively.

Example: For Geckoboard, Digital Agencies were very likely to convert and the old scoring algorithm scored them highly. However, agencies had a 5X chance of churning after 3 months when the project they were working on ended.

At this point, the leads being sent to Bates were significantly better in aggregate than the leads he had previously been receiving. However, there were still false positives that would throw him off. While the overall stats on scored leads were looking great, the mistakes the model made hurt sales and marketing trust and were hard to accept. To combat this and make the qualification model close to perfect, Whittick had Bates start flagging any highly scored leads that made it through.

Through this process, they found that many of the bad leads that made it through were students (, fake signups (, or more traditional companies that did not have the technology profile of a company who would likely use Geckoboard ( Whittick was then able to add specific, derived features to their scoring system to effectively filter these leads out and yet again improve the leads making it to Bates.

At this point, Geckoboard can predict 80% of their conversions from just 12% of their signups. By increasing sales efficiency with machine learning, Whittick found a way to enable Bates to do the work an average sales team of five could typically handle.

From self-driving trucks to food delivery robots, this is the story of twenty-first-century business. Companies like Geckoboard are employing fewer people and creating more economic value than enterprises ten times their size. Leaders like Whittick are center stage in this revolutionary tale, figuring out how to optimize sales efficiency or conversion rates or any other metric given to them, just like the artificial intelligence they now employ. But of course, this has been happening over many years, even decades. The difference—and this cannot be overstated—is that Whittick doesn’t have a PhD in applied math or computer science. The technology available to marketers today enables companies to generate twice the revenue with half the people.

Activity Scoring & Enterprise Free Trials: how to do it right.

Redpoint Ventures Partner Tomasz Tunguz recently published the results of their Free Trial Survey, which included responses from 590 professionals working at freemium SaaS businesses of all sizes and shapes. The survey had many interesting takeaways and I recommend taking the time to dive into the slides that Tunguz shared at SaaStr Annual this year.

One of the more interesting takeaways (that Tunguz discussed on his blog) was that activity scoring seems to negatively impact free trial conversions for high ACV leads. Tunguz found that Enterprise SaaS businesses using activity scoring see a 4% conversion rate for free trials vs. 15% for those not using activity scoring.

MadKudu has written a lot about free trials in the past, including the article that inspired Tunguz referenced in launching his survey, so it was natural for us to weigh in on this conclusion.

I asked for a few clarifications from Tunguz in preparing this article:

  • The survey defined activity scoring as lead scoring that leverages in-app activity (not email marketing, webinar engagement or other activities).
  • The conversion rate (4%/15%) is calculated against all leads, not leads that scored well, so we’re measuring the effectiveness of the funnel not the precision of the score itself.
  • We’re only looking at leads who participate in the free trial, not leads that schedule a demo or otherwise enter the funnel.

With that in mind, I believe there are two main takeaways and some data to support those conclusions.

Enterprise leads don’t want a Free Trial.

summary: our data shows Enterprise prefer to schedule a demo and self-serve prefer a free trial (if available). Putting either in their counterpart’s funnel negatively impacts their likelihood to convert.

Free trial products design enterprise buyer journey calls-to-action – “contact sales” “schedule a demo” &  “request a quote” – in order to entice enterprise prospects. As Tunguz pointed out in the analysis of his survey, enterprise leads don’t typically try before they buy. They may sign up for a free trial to get a feel for the interface and do some preliminary feature validation, but the buying process is more sophisticated than that and lasts longer than your free trial.

One hypothesis for why activity scoring decreases conversion for enterprise leads in free trials is that enterprise leads shouldn’t be running free trials – or at least, they shouldn’t be having the same free trial experience. It is worth reading Tunguz’s piece about assisted vs. unassisted free trials to dive deeper into this subject.

Supporting this hypothesis is an experiment run by Segment & MadKudu looking at the impact of a free trials & demo requests on the likelihood that a self service & enterprise lead would convert. Segment Forced the Funnel by dynamically qualifying & segmenting website traffic, personalizing the website based on the predicted spend. This allowed us to predict whether traffic was self-serve or enterprise.

“Self-serve traffic” would not see the option to schedule a demo while “enterprise traffic” would not see the option to sign up for a trial. They also ran a control to measure the impact on the funnel.

They found a negative correlation between self-serve conversion & requesting a demo. They also found a negative correlation between enterprise conversion & signing up for a free trial.  Each buyer segment has an ideal customer journey and deviating from it (even into another buyer segment’s ideal journey) negatively impacts conversion.

The converse is equally true: pushing leads into their ideal customer journey increases their conversion rate by 30%.

Startups using activity scoring on high ACV leads should work to get high ACV leads out of their free trial by identifying them early on. Algolia, for example, prompts self-serve trial signups who have a high ACV to get in touch with someone for an assisted free trial.

Scoring activity at the account level

For SaaS businesses that go up-market or sell exclusive to enterprise, activity scoring at the lead level may not be sufficient. We worked with InVision to identify sales opportunities at the account level, importing all activity data from HubSpot & Segment and merging at the account level. We analyzed the impact that various user personas had on the buyer journey and product experience.

Profiles that were more likely to be active in the product  – Marketing, analysts & designers – had a less than average impact on the likelihood to convert. Personas associated with higher likelihood to convert – Directors, founders, CEOs – had a smaller impact on activation.

Multiple personas are needed to create optimal conditions for an account to activate & convert on InVision. Their marketing team uses this knowledge to focus on post-signup engagements that will increase the Likelihood to Buy, the behavioral score built by MadKudu.

We see similar findings in the buyer journey as we examine how various personas’ involvement in an account impacts opportunity creation vs. opportunity closed-won. Opportunities are more likely to be created when marketers & designers are involved, but they are more likely to close when CEOs & Directors get involved.

For InVision, interestingly enough, founders have a smaller impact on opportunity closed-won than they do on product conversion.

While a single lead may never surpass the activity threshold that correlated with sales readiness at InVision, scoring activity at the account level surfaced accounts that exceeded the account activity threshold. Both thresholds were defined by MadKudu & InVision using the same data sources.

The above slides are all from our HubSpont Inbound 2018 talk and are available here.

Measuring Scoring Model effectiveness

Looking at the results of experiments run with our customers and the data from Tunguz’s survey, it’s clear that activity scoring doesn’t work in a vacuum. Both our MQA model for InVision & our models for Segment require firmographic, technographic and intent data in combination with behavioral data in order to build a predictive model.

The impact that a model will have on sales efficiency & velociate depends on its ability to identify X% of leads that represent Y% of Outcomes. The power of this function increases as X tends to 0 and Y tends towards 100. “Outcomes” can represent opportunities created, opportunities won, pipeline created, or revenue, depending on the metric your sales strategy is optimizing for.

We similarly expect that the X% of leads will convert at a significantly higher percentage than lower-quality leads. As seen in the above graphic, a very good lead may be 17x as likely to convert than a low quality lead, which makes a strong case for sales teams prioritizing very good leads as defined by their predictive model – at least if they want to hit quota this quarter.

If you’re selling exclusively to enterprise leads, an assisted free trial operates a lot like a schedule a demo flow – you will score leads firmographically early on, evaluate the opportunity, and then assist them in onboarding to your product to trial it, scoring activity throughout the trial to evaluate likelihood to convert.

Most SaaS don’t sell exclusively offer free trials to high ACV leads, which is why activity scoring becomes crucial. A lead that is a great fit for self-service is also a bad fit for enterprise. Self-service leads convert quickly and consistently with a small to medium sized ACV, whereas enterprise leads have a small percentage chance of converting, but convert with a much higher ACV. Velocity sales requires a steady stream of quick conversions – a long sales cycle for low ACV is a loss – while enterprise sales can take months or years to close a single account while still being a success.

For customers with velocity & enterprise leads, MadKudu scores every lead against both models, which enables us to identify whether it’s a good fit for self-serve, enterprise, both or none (it’s almost never both).

How MadKudu makes Salesforce Einstein smarter

…Or why Salesforce Einstein won’t be the next IBM Watson.

Is the AI hype starting to wither? I believe so, yes.
Reality of the operational world is slowly but steadily catching up with the idyllic marketing fantasy. The report Jeffries put together challenging IBM Watson proves alarm bells are ringing. The debacle of the Anderson implementation goes to show how marketing promises can be unrealistic and the downfall will be dreadful.

With that said, not all is lost as we keep learning from our past mistakes. Being part of the Salesforce Einstein focused incubator, we are witnessing first-hand how the CRM giant is looking to succeed where Watson and others are struggling. Hopefully these insights can help others rethink their go-to-market strategy, in an era of unkept commitments.

Salesforce, a quick refresher

A few weeks ago, I was being interviewed for an internal Salesforce video. The question was “how has the Salesforce eco-system helped your startup?”. To contextualize my thoughts it’s important to know that while Salesforce is one of our main integrations, we consider it as an execution platform among others (Segment, Marketo, Intercom, Eloqua…). I’ve always admired Salesforce for its “platform” business model. Being part of the Salesforce ecosystem facilitated our GTM. It gave MadKudu access to a large pool of educated prospects.

However I believe the major value add for startups is the focus induced by working with Salesforce customers. Since Salesforce is a great execution platform there are a plethora of applications available addressing specific needs. This means, as a startup, you can focus on a clearly defined and well delimited value proposition. You can rely on other solutions to solve for peripheral needs. As David Cohen reminded us during our first week at Techstars, “startups don’t starve, they drown”. Salesforce has helped us stay afloat and navigate its large customer base.

What is Salesforce Einstein?

I’m personally very excited about Salesforce Einstein. For the past 5 years, I’ve seen Machine Learning be further commoditized by products such as Microsoft Azure,… We’ve had many investors ask us what our moat was given this rapid democratization of ML capabilities and our answer has been the same all along. In B2B Sales/Marketing software, pure Machine Learning should not be considered a competitive advantage mainly because there are too few data sets available that require non-generic algorithms. The true moat doesn’t reside in the algorithms but rather in all the aspects surrounding them: feature generation, technical operationalization, prediction serving, latency optimization, business operationalization… The last one being the hardest yet the most valuable (hence the one we are tackling at MadKudu…).
Salesforce Einstein is the incarnation that innovation will be in those areas since anyone can now run ML models with their CRM.

We’ve been here before

Just a reminder, this is not a new thing. We’ve been through this not so long ago.
Remember the days when “Big Data” was still making most of the headlines on Techcrunch? Oh how those were simpler times…

Big Data vs Artificial Intellligence search trends over the past 5 years

There were some major misconceptions as to what truly defined Big Data especially within the context of the Enterprise. The media primarily focused on our favorite behemoths: google, facebook, twitter and their scalling troubles. Big data became synonymous for Petabytes and unfathomly large volumes of data more generally. However scholars defined a classification that qualified data as “big” for 3 reasons:
– volume: massive amounts of data that required distributed systems from storage to processing
– velocity: quickly changing data sets such as product browsing. This meant offline/batch processing needed an alternative
– variety: data originating from disparate sources meant complex ERDs had to be maintained

In the Enterprise, volume was rarely the primary struggle. Velocity posed a few issues to large retailers and companies like RichRelevance nailed the execution of their solution. But the main and most challenging data issue faced was with the variety of data.

What will make Salesforce Einstein succeed

Einstein will enable startups to provide value to the Enterprise by focusing on the challenges of:
– feeding the right data to the platform
– defining a business playbook of ways to generate $$ out of model predictions

We’ll keep the second point for a later blog post but to illustrate the first point with DATA, I put together an experiment. I took one of our customers’ dataset of leads and opportunities.
The goal was to evaluate different ways of building a lead scoring model. The objective was to identify patterns within the leads that indicated a high likelihood of it converting to an opportunity. This is a B2B SaaS company selling to other B2B companies with a $30k ACV.
I ran an out-of-the-box logistic regression on top of the usual suspects: company size, industry, geography and alexa rank. For good measure we had a fancy tech count feature which looked at the amount of technologies that could be found on the lead’s website. With about 500 opportunities to work on, there was a clear worry about overfitting with more features. This is especially true since we had to dummy the categorical variables.
Here’s how the regression performed on the training data (70% of the dataset) vs the test dataset (30% that were not used for training and ensuring if a company is part of the training it is not part of testing – see we did not fool around with this test)

Regression model using firmo and technographic featuresmodel performance on test dataset using available data points

Not bad right?! There is a clear overfitting issue but the performance is not dreadful apart for a blob in the center

Now we ran the same logistic regression against 2 feature: predicted number of tracked users (which we know to be highly tied to the value of the product) and predicted revenue. These features are the result of predictive models that we run against a much larger data set and take into account firmographics (Alexa rank, business model, company size, market segment, industry…) along with technographics (types of technologies used, number of enterprise technologies…) and custom data points. Here’s how the regression performed:

Regression with MadKudu featuresmodel performance on test dataset using 2 MadKudu features

Quite impressive to see how much better the model performs with fewer features. At the same time, we are less running the risk of overfitting as you can see.

The TL;DR is that no amount of algorithmic brute force applied to these B2B data sets will ever make up for appropriate data preparation.

In essence, Salesforce is outsourcing the data science part of building AI driven sales models to startups who will specialize in verticals and/or use-cases. MadKudu is a perfect illustration of this trend. The expert knowledge we’ve accumulated by working with hundreds of B2B SaaS companies is what has enabled us to define these smart features that make lead scoring implementations successful.

So there you have it, MadKudu needs Salesforce to focus on its core value and Salesforce needs MadKudu to make its customers and therefore Einstein successful. That’s the beauty of a platform business model.
I also strongly believe that in the near future there will be a strong need for a “Training dataset” marketplace. As more of the platforms make ML/AI functionalities available, being able to train them out-of-the-box will become an important problem to solve. These “training datasets” will contain a lot of expert knowledge and be the results of heavy data lifting.

Feel free to reach out to learn more

Google trends
MadKudu demo Jam 3

PS: To be perfectly clear, we are not dissing on IBM’s technology which is state of the art. We are arguing that out-of-the-box AI have been overhyped in the Enterprise and that project implementation costs have been underestimated due to a lack of transparence on the complexity of configuring such platforms.

Are Automation and AI BS?

A couple weeks ago, I ended up taking Steli’s click bait and read his thoughts on sales automation and AI. There isn’t much novelty in the comments nor objections presented. However I felt compelled to write a answer. Part of the reason why, is that MadKudu is currently being incubated by Salesforce as part of the Einstein batch. Needless to say the word AI is uttered every day to a point of exhaustion.

The mythical AI (aka what AI is not today)

The main concern I have around AI is that people are being confused by all the PR and marketing thrown around major projects like Salesforce’s Einstein, IBM’s Watson and others – think Infosys Nia, Tata Ignio, the list goes on.

Two months ago, at the start of the incubator, we were given a truly inspiring demo of Salesforce’s new platform. The use-case presented was to help a solar panel vendor identify the right B2C leads to reach out to. A fairly vanilla lead scoring exercise. We watched in awe how the CRM was fed google street view images of houses based on the leads’ addresses before being processed through a “sophisticated” neural network to determine if the roof was slanted or not. Knowing if the roof was slanted was a key predictor of the amount of energy the panels could deliver. #DeepLearning

This reminded me of a use-case we discussed with Segment’s Guillaume Cabane. The growth-hack was to send addresses of VIP customers through Amazon’s mechanical turk to determine which houses had a pool in order to send a targeted catalogue about pool furniture. Brilliant! And now this can all be orchestrated within the comfort of our CRM. Holy Moly! as my cofounder Sam would say.

To infinity and beyond, right?

Well not really, the cold truth is this could have also been implemented in excel. Jonathan Serfaty, a former colleague of mine, for example wrote a play-by-play NFL prediction algorithm entirely in VBA. The hard part is not running a supervised model, it’s the numerous iterations to explore the unknowns of the problem to determine which data set to present the model.

The pragmatic AI (aka how to get value from AI)

Aside from the complexity of knowing how to configure your supervised model, there is a more fundamental question to always answer when considering AI. This foundational question is the purpose of the endeavor. What are you trying to accomplish with AI and/or automation? Amongst all of the imperfections in your business processes which one is the best candidate to address?

Looking through history to find patterns, it appears that the obvious candidates for automation/AI are high cost, low leverage tasks. This is a point Steli and I are in agreement on: “AI should not be used to increase efficiency”. Much ink has been spilled over the search for efficiency. Henry Ward’s eShares 101 is an overall amazing read and highly relevant. One of the topics that strongly resonated with me was the illustrated difference between optimizing for efficiency vs leverage.

With that in mind, here are some examples of tasks that are perfect fits for AI in Sales:

  • Researching and qualifying
  • Email response classification (interested, not interested, not now…)
  • Email sentiment classification
  • Email follow up (to an email that had some valuable content in the first place)
  • Intent prediction
  • Forecasting
  • Demo customization to the prospect
  • Sales call reviews

So Steli is right: No, a bot will not close a deal for you but it can tell you who to reach out to, how, why and when. This way you can use your time on tasks where you have the highest leverage: interacting with valuable prospects and helping them throughout the purchase cycle. While the recent advent of sales automation has led to an outcry against the weak/gimmicky personalization I strongly believe we are witnessing the early signs of AI being used to bring back the human aspect of selling.

Closing thoughts

AI, Big Data, Data Science, Machine Learning… have become ubiquitous in B2B. It is therefore our duty as professionals to educate ourself as to what is really going on. These domains are nascent and highly technical but we need to maintain an uncompromising focus on the business value any implementation could yield.

Want to learn more or discuss how AI can actually help your business? Feel free to contact us

3 steps to determine the key activation event

Most people by now have heard of the “Product key activation event”. More generally, Facebook’s 7 friends in the first 10 days, Twitter’s 30 followers… get lots of mentions in the Product and Growth communities. Theses examples have helped cement the idea of statistically determining goals for the onboarding of new users. A few weeks ago, somebody from the Reforge network asked how to actually define this goal and I felt compelled to dive deeper into the matter.

I love this topic and while there’s already been some solid answers on Quora by the likes of Uber’s Andrew Chen, AppCues’ Ty Magnin and while I have already written about how this overarching concept a couple weeks ago (here) I wanted to address a few additional/tactical details.

Below are the three steps to identify your product’s “key activation event”.

Step 1: Map your events against the Activation/Engagement/Delight framework

This is done by plotting the impact on conversion of performing and not performing an event in the first 30 days. This is the core of the content we addressed in our previous post.

To simplify, I will call “conversion” the ultimate event you are trying to optimize for. Agreeing on this metric in the first place can be a challenge of itself…

Step 2: Find the “optimal” number of occurrences for each event

For each event, you’ll want to understand what is the required occurrence threshold (aka how many occurrences maximize my chances of success without hitting diminishing returns). This is NOT done with a typical logistic regression even though many people try and believe so. I’ll share a concrete example to show why.

Let’s look at the typical impact on conversion of performing an event Y times (or not) within the first X days:

There are 2 learnings we can extract from this analysis:
– the more the event is performed, the more likely to convert the users are (Eureka right?!)
– the higher the threshold of number of occurrences to perform, the closer the conversion rate of people who didn’t reach it is to the average conversion rate (this is the important part)

We therefore need a better way to correlate occurrences and conversion. This is where the Phi coefficient comes into play to shine!

Below is a quick set of Venn diagrams to illustrate what the Phi coefficient represents:

Using the Phi coefficient, we can find the number of occurrences that maximizes the difference in outcome thus maximizing the correlation strength:

Step 3: Find the event for which “optimal” number of occurrences has the highest correlation strength

Now that we have our ideal number of occurrences within a time frame for each event, we can rank events by their highest correlation strength. This will give us for each time frame considered, the “key activation event”.

Closing Notes:

Because Data Science and Machine Learning are so sexy today, everyone wants to run regression modeling. Regression analyses are simple, interesting and fun. However they lead to suboptimal results as they maximize for likelihood of the outcome rather than correlation strength.

Unfortunately, this is not necessarily a native capability with most analytics solutions but you can easily dump all of your data in redshift and run an analysis to mimic this approach. Alternatively, you can create funnels in Amplitude and feed the data into a spreadsheet to run the required cross-funnel calculations. Finally you can always reach out to us.

Don’t be dogmatic! The results of these analyses are guidelines and it is more important to pick one metric to move otherwise you might spiral down into an analysis-paralysis state.

Analysis << Action
Remember, an analysis only exists to drive action. Ensure that the events you push through the analysis are actionable (don’t run this with “email opened”-type of events). You should always spend at least 10x more time on setting up the execution part of this “key activation event” than on the analysis itself. As a reminder, here are a couple “campaigns” you can derive from your analysis:

  • Create a behavioral onboarding drip (case study)
  • Close more delighted users by promoting your premium features
  • Close more delighted users by sending them winback campaigns after their trial (50% of SaaS conversions happen after the end of the trial)
  • Adapt your sales messaging to properly align with the user’s stage in the lifecycle and truly be helpful

– MadKudu Grader (2015)
– MadKudu “Happy Path” Analysis Demo Sample

Improve your behavioral lead scoring model with nuclear physics

According to various sources (SiriusDecision, SpearMarketing) about 66% of B2B marketers leverage behavioral lead scoring. Nowadays we rarely encounter a marketing platform that doesn’t offer at least point based scoring capabilities out of the box.

However, this report by Spear Marketing reveals that only 50% of those scores include an expiration scheme. A dire consequence is that once a lead has reached a certain engagement threshold, the score will not degrade. As put it in the report, “without some kind of score degradation method in place, lead scores can rise indefinitely, eventually rendering their value meaningless.” We’ve seen this at countless companies we’ve worked with. It is often a source of contention between Sales and Marketing.

So how do you go about improving your lead scores to ensure your MQLs get accepted and converted by Sales at a higher rate?

Phase 1: Standard Lead scoring

In the words of James Baldwin, “If you know whence you came, there are absolutely no limitations to where you can go”. So let’s take a quick look at how lead scoring has evolved over the past couple of years.

Almost a decade ago, Marketo revolutionized the marketing stack by giving marketers the option to build heuristical engagement models without writing a single line of code. Amazing! A marketer, no coding skills required, could configure and iterate over a function that scored an entire database of millions of leads based on specific events they performed.

Since the introduction of these scoring models, many execution platforms have risen. The scoring capability has long become a standard functionality according to Forester when shopping for marketing platforms.

This was certainly a good start. The scoring mechanism had however 2 major drawbacks over which much ink has been spilt:

  • The scores don’t automatically decrease over time
  • The scores are based on coefficients that were not determined statistically and thus cannot be considered predictive

Phase 2: Regression Modeling

The recent advent of the Enterprise Data Scientist, formerly known as the less hype Business Analyst, started a proliferation of lead scoring solutions. These products leverage machine learning techniques and AI to accommodate for the previous models inaccuracies. The general idea is to solve for:  

Y = ∑𝞫.X + 𝞮


Y is the representation of conversion
X are the occurrences of events
𝞫 are the predictive coefficients


So really the goal of lead scoring becomes finding the optimal 𝞫. There are many more or less sophisticated implementations of regression algorithms to solve for this, from linear regression to trees, to random forests to the infamous neural networks.

Mainstream marketing platforms like Hubspot are adding to their manual lead scoring some predictive capabilities.

The goal here has become helping marketers configure their scoring models programmatically. Don’t we all prefer to blame a predictive model rather than a human who hand-picked coefficients?!

While this approach is greatly superior, there are still a major challenge that need to be addressed:

  • Defining the impact of time on the scores

After how long does having “filled a form” become irrelevant for a lead? What is the “thermal inertia” of a lead, aka how quickly does a hot lead become cold?

Phase 3: Nuclear physics inspired time decay functions

I was on my way home some time ago, when it struck me that there was a valid analogy between Leads and Nuclear Physics. A subject in which my co-founder Paul holds a masters degree from Berkeley (true story). The analogy goes as follows:
Before the leads starts engaging (or being engaged by) the company, it is a stable atom. Each action performed by the lead (clicking on a CTA, filling a form, visiting a specific page) results in the lead gaining energy, thus furthering it from its stable point. The nucleus of an unstable atom will start emitting radiation to lose the gained energy. This process is called the nuclear decay and is quite well understood. The time taken to free the energy is defined through the half-life (λ) of the atom. We can now for each individual action compute the impact over time on leads and how long the effects last.

Putting all the pieces together we are now solving for:

Y = ∑𝞫.f(X).e(-t(X)/λ) + 𝞮


Y is still the representation of conversion
X are the events
f are the features functions extracted from X
t(X) is the number of days since the last occurrence of X
𝞫 are the predictive coefficients
λ are the “half-lives” of the events in days


This approach yields better results (~15% increase in recall) and accounts very well for leads being reactivated or going cold over time.

top graph: linear features, bottom graph: feature with exponential decay


Next time we’ll discuss how unlike Schrödinger’s cat, leads can’t be simultaneously good and bad…


xkcd Relativistic Baseball:
Marketo behavioral lead score:
Amplitude correlation analysis:
HubSpot behavioral lead score:
MadKudu: lead score training sample results