educative.io

Backtesting -ads prediction

Courses
,
#1

Hi, I’m a little confused - how can we use backtesting to evaluate the user engagement with ads since it is impossible to know from the past data whether users would chose the control or the variation? Thanks

#2

Hi Roman, Back-testing is here is essentially re-running the same experiment but with control and variation reverse. One question that may arise that why would we do that? Because, we want to more certain that the observations that we had from the first experiment that we launched still hold true.

Essentially back-testing would make sense when we are happy about the results and launch it to user. But at the same time, we want to

  1. Validate the hypothesis by re-running the experiment and be more certain that the improvements/gains are real. Control and Variation are now reversed given we already launched the experiment.

  2. Want to understand the long-term impact of the experiment e.g. if we saw a 5% lift in revenue, was that temporary? or will it continue to hold for a longer period of time?

So, we can’t really lose data as essentially experiment is running on a variation of the system and we are testing that variation through online experimentation.

Team @ Machine Learning Labs

1 Like