All of our clients derive significantly higher conversions compared to standalone campaigns using Journeys. But how do you benchmark these conversions? In our quest to highlight the contribution of your Journeys towards your business objectives, we now show percentage uplift in conversion from Journeys. This uplift will be benchmarked on the conversions from the control group.
The conversion uplift from a Journey is calculated by comparing the following:
- Journey conversions: Percentage of users who received any of the Journey campaigns and then converted.
- Journey control group conversions: Percentage of users who were part of the control group (eligible to receive any of the Journey campaigns but did not receive) and converted.
What does this mean?
To understand this in more detail, we will first have to understand how Journey control groups work.
Control groups for Journeys are defined for the whole Journey, not for individual campaigns of the Journey.
Users in the control group enter the Journey and traverse the blocks for which they are eligible. This behavior is the same as that for non-control group users. Differentiation happens when control group users reach any of the Action blocks responsible for sending a message (Send Email / Send SMS / Send Push / Show In-App Message / Show Web Message / Send Web Push). They move ahead in the Journey without the message being sent to them. They might still end up performing the conversion event, in which case this conversion is added to the control group conversion count.
This is a fair comparison since the control group users are also following the exact same Journey as non-control group users. This is important because there might be blocks in the Journey which change the user’s attributes, call APIs, or wait for a particular event before sending messages. Which makes it important to compare control group users and non-control group users using both Journey and user contexts.
How is this approach better?
To put all of this into context, let’s look at how we had implemented the conversion uplift calculation prior to this update. Previously, control group users did not enter the Journey at all. The comparison for conversions was unfair in this case because:
- Control group users not entering the Journey meant that we were taking into account conversions for all users in the control group, while we should be taking into account only for the ones who are eligible to receive the messages for a fair comparison. Eligible users are the ones who reach at least one of the message Action blocks in the Journey.
- All of the control group users were getting a lot of time to complete conversion since the conversion deadline was based on the time when Journey ends.
- If the Journey included Flow Control blocks (Wait for some time / Wait for an Event / Wait for a Date), then non-control group users could be in the Journey for days whereas users in the control group who did conversion event would get out of the Journey and later enter it again, which would further mess up the uplift numbers.
These issues have been corrected now by
- Making control group users enter the Journey but not sending them messages when they reach the message Action blocks.
- Then comparing their conversions with non-control group users.
Prakhya Nair
Harshita Lal
