This chapter is the combination of two critical capabilities of a marketing automation platform:
The degree to which your marketing automation platform allows you to do hypothesis testing.
The most important thing to consider is how the tests are reported. Nearly all platforms allow you to send split your audience into many to send them individual variations. (A/B test) But how reliable the testing results are? Were the results driven by randomness?
Check how reliably the platform addresses these questions in its reporting.
You ran a push notification campaign and saw a jump in sales. Well, how much of the conversion can be directly attributed to push notification? What was the conversion rate for users who weren’t treated with any push campaign at all? What is the conversion rate of each variant? Conversion tracking answers these pertinent questions.
In this chapter, we will evaluate a marketing automation software on the basis of above two parameters:
This chapters covers following topics
1. Control Group:
2. A/B Testing:
5.1 Conversion tracking and Control Group
Why it’s important?
Conversion tracking like the names suggests is the measurement of the impact of the campaign in terms of how you wanted it.
To track conversion of any campaign say email, you zero in the right conversion event. This event is the desired action that you wish to achieve from the campaign. Say, from a reactivation campaign you desire reactivation of the user.
When the campaign is sent, you gauge the number of conversions that can be directly attributed to the campaign in question.
Now, it may happen that the customers would have reacted in the same way or converted by the same magnitude or even better regardless of your campaign. Perhaps the campaign was not required. We verify this hypothesis by creating Control group.
What is Control group?
‘Control group’ is the special set of users who, despite being part of the segment, are excluded from receiving the campaign but, and it’s very important, their conversion is tracked. It is basically a neutral group which is created only to gauge the metrics of the users who were not part of the campaign versus the users who were.
By setting up a control group(CG) you would be able to measure the conversion of the users who were treated with your campaign against the users who weren’t. This would essentially let you know the true impact of your campaign.
- What is the conversion uplift provided by your campaign
- Identify the best variation
The difference in the conversion of the control group and campaign is the uplift provided by it.
Doing conversion tracking for multiple variants tells you the winning variation and also the one/s which is performing below average.
5.2 A/B Testing
What is it?
A/B testing can be split into two:
A/B test of screens- wherein you test the screens of the app (mobile and web) for optimization. Previously, this capability was limited to only dedicated A/B testing tools like VWO, Optimizely. Now, even marketing automation solutions have begun to offer testing of the app screens. Obviously, you cannot expect this capability to be as comprehensive as dedicated testing solutions like Optimizely and VWO but a basic capability would help.
A/B test of message- wherein you test the different variation of the same message.
When the list size runs in millions than it is important to have advanced capability to run tests. Again A/B testing is in vogue. It’s everywhere. Let’s look at some of the caveats that one should be wary of while analyzing the variant testing capability of the platform.
Key questions/ considerations
- What is the maximum number of variants that we can create?
- Can we create variants across all the channels, particularly web messaging?
- If the email campaign has more than one link do you provide the stats for each of them?
- How does the reporting UX look like?
Aggregate CTR only gives the incomplete information. A solid email solution should be able to give you the heatmap view of the CTR.
Check for the reporting UX, as in how the overall campaign stats are displayed. It should be intuitive so that there is the least friction in comparing the metrics of all the variants.