Hazelnuts? Horseshoes? No we're talking about hypotheses! We are Retention SCIENCE after all. The foundation of science is all based around testing out hypotheses. Scientists (and Marketing Scientists - yeah that's you!) utilize questions they have about the world (or customer base) and develop tests to answer the questions. In this article we'll discuss what a hypothesis is, how to frame and test a specific hypothesis and how to generate these questions on your own.
She Blinded Me With Science
The Scientific Method has been used for years to advance human kind in technology. Without it you wouldn't be reading this article right now, heck, you probably wouldn't be working at an e-commerce company. The point is that small incremental changes and optimization can lead to long term gains. This is how you should approach all of your marketing questions and strategies.
In order to structure this into a marketing setting we'll first need to understand the process. The basic Scientific Method is composed of 5 basic steps:
- Ask a Question
- Construct a Hypothesis
- Analyze and Draw Conclusions
- Iterate and test again
We'll go into each one of the steps in more detail below. Each one of these steps is essential from gaining true insight into your customer base, their behavior and how to reap incremental gains through each test you construct.
1. Ask a Question
This question should be fundamental to either your business, email strategy or user group that you want to gather more data from. It's best to think of each stage as a group of people exhibiting a specific behavior. This gives you easy identifiable groups to target with specific strategies. The users in the individual stages may change but the overall behavior across the stage is the same. This gives us a testing population that will exhibit similar behavior (the controlled or constant variable) in response to your testing criteria and not based on other external factors.
Some examples of questions to ask about your user base are below. These can be utilized in any of the stages, but careful consideration should be made when thinking about these questions within the context of a specific stage (i.e. Ready To Buy users will have a higher likelihood of purchase than needs nurturing, testing an incentive in these two groups will have much different results).
- Which of my value propositions do new subscribers identify with most?
- What type of incentives can drive more conversions from subscribers with a Low Intent to purchase?
- How often should I change/update content in the Ready To Buy Stage?
- Are my customers actually interested in the products they are browsing?
- What kind of content should I produce to reengage my customers?
2. Construct a Hypothesis
A hypothesis is the expected answer to the question that was formulated above. The important part here is that the hypothesis should follow an expected result that we will deem our hypothesis and a negative result which will be our null hypothesis. Normally these statement stake the form of an If _____ then ______ statement, where the if is the question and the then is the expected result. Using the sample question # above, we'll break this down into a clear hypothesis.
The Question: Are my customers actually interested in the products they are browsing?
The Hypothesis: If my customers are interested in the products they are browsing then a template highlighting the specific product that was browsed will perform (based on click) better than a template that does not include the item that was browsed.
As you can see, by reframing this question into a hypothesis we have framed the testing parameters, a template highlighting the browsed item and one without the browsed item, and we defined a metric (clicks) to judge the outcome of the test.
Continuing with our hypothesis from above we now need to determine how to test and for how long the test should be executed before coming to a conclusion. In this situation the testing variable is based around the browsed item. So this would be a template level test. In order to make sure that only the template variable is being tested, be sure to make your subject lines and pre-header lines the same between both templates. You can still have multiple subject lines on each stage, but be sure they are the same.
As you can see in this example from LXR and Co, we completely removed the item that was browsed from the template on the right. This will give us a clean test to confirm or deny our hypothesis. Now some of you reading this may object to this test because the copy is slightly different in the test case on the right. You have every reason to point this out, but it's important to remember what the intent of the test is. In this case the copy change is inline with the intent of testing wether the browsed item is important. If it's not important that the slight change in copy should have a minimal affect. It's up to you and your team to determine what the right level of variables will be to introduce into your test. Our advice is to start with one variable per test, and as you get more comfortable with drawing conclusions you can start introducing multiple variables into your test. It's important to note that multi-variable tests have a higher likelihood of being inconclusive because it will be difficult to determine which variable was successful. You can always break the two variables out in your next iterative test, and test each variable on its own plus the comparable variable.
In general the Data Science team recommends approximately 10000 total sends of the combined variation before a winner should be decided. You can discuss what is appropriate for your user base with your CSM if you are unsure.
4. Analyze and Draw Conclusions
Now for the fun part! The results are all based on the metric that you called out in the hypothesis. Directly compare the two templates and see which one worked in terms of click rate. The template that had the highest click rate is going to be determined as the "winner".
If the template that features the browsed item performed better, then the hypothesis is confirmed, and it is very important to highlight the browsed product in emails.
If the template without the browsed product performs better, then the null hypothesis is confirmed, and it's better to show your customer base other similar items instead of the item that was browsed.
If neither template performed better (or they performed the same) then the test is inconclusive. In this case, the recommendation is to leave both templates running, and begin thinking of a new question to test for the browse abandon stage.
5. Iterate and Test Again!
Oh did you think we were done? The best part about being a marketer is that there are endless tests, improvements and messaging tweaks that you can do. Use what you just learned to inform, draw conclusion and help develop new tests and questions you'd like to discover about your user base. If you learn something that you think is valuable in other stages, put it to the test. Remember that these tests can also go up against your control template (the tried and true template that's been with you from the beginning), as Cortex will make adjustments in case your test completely flops.
Enjoy your Intelligent Email Testing Platform (Cortex) and start gaining incremental lift on every campaign that you run!