aBayesian A/B Testing

Table of Content

1. Bayesian A/B Testing 2. Loss between A & B 3. Stopping Criterion 4. Why Bayesian A/B testing?

1. Bayesian A/B Testing• In the Bayesian approach to A/B testing, the setup is the same as the frequentist approach, except that the way that we're interpreting the results is going to be different.• Just as a reminder, the Bayes' rule is P(H|D)=P(D|H).P(H)

P(D)$$P(H|D)=\frac{P(D|H).P(H)}{P(D)}$$ • The idea is that we want to know the probability of some hypothesis being true given some data.– P(H|D)${\displaystyle P(H|D)}$ → posterior– P(D|H)${\displaystyle P(D|H)}$ → likelihood → probability of some data given the hypothesis– P(H)${\displaystyle P(H)}$ → prior– P(D) = ∑hP(D|Hh)P(Hh)${\displaystyle P(D)=\sum _h^{}P(D|H_h)P(H_h)}$ → probability of data given all the hypotheses (in our case, they are null and alternative hypotheses). • The likelihood term P(D|H)= ∏iPxi(1-P)1-xi${\displaystyle P(D|H)=\prod _i^{}{P}^{x_i}(1-P{)}^{1-x_i}}$• The prior term P(H)= P𝛼-1(1-P)𝛽-1

B(𝛼,𝛽)${\displaystyle P(H)=\frac{P^{𝛼-1}(1-P{)}^{𝛽-1}}{B(𝛼,𝛽)}}$ → Beta distribution (e.g. distribution over CTP${\displaystyle CTP}$ probabilities)– Note: When 𝛼=𝛽=1${\displaystyle 𝛼=𝛽=1}$, it means we have uninformed prior.• • Note: Posterior and prior are going to be the same type of distribution.• Note: We interpret the posterior parameters as distribution.• In the Bayesian approach we compare the two posterior distributions of treatment and control groups (and not just two numbers as in frequentist approach).• Here, we want to know the probability of group B${\displaystyle B}$ being greater than group A${\displaystyle A}$.– We examine the Beta distributions produced by group B${\displaystyle B}$ and group A${\displaystyle A}$.– We sample from A${\displaystyle A}$ and B${\displaystyle B}$'s posterior and we get the percent of times that B > A${\displaystyle B>A}$. 2. Loss between A & B• Let's say we ended up choosing B${\displaystyle B}$ over A${\displaystyle A}$, even though B${\displaystyle B}$ was actually worse than A${\displaystyle A}$. How would this be possible?– The posterior distributions of A${\displaystyle A}$ and B${\displaystyle B}$ can have a small overlaps. – The small overlap means that there's a chance that we could actually pick B${\displaystyle B}$ and get A${\displaystyle A}$'s result.– So what we really want is to sample the differences between A${\displaystyle A}$ and B${\displaystyle B}$ and that will give us the loss from choosing B${\displaystyle B}$ over A${\displaystyle A}$ in the case that B${\displaystyle B}$ is actually worse.• With the loss, we can multiply that by the probability that A > B${\displaystyle A>B}$ → This would give us the expected loss (EL). → – Assuming B>A${\displaystyle B>A}$ when it's not → EL = P(A>B).max(CTPA-CTPB,0)${\displaystyle EL=P(A>B).\max (CT{P}_A-CT{P}_B,0)}$– Note: Expected loss is in the same unit as our metric.• Another way to look at it is to look at the expected gain.– Assuming B>A${\displaystyle B>A}$ when it is → EL = P(B>A).max(CTPB-CTPA,0)${\displaystyle EL=P(B>A).\max (CT{P}_B-CT{P}_A,0)}$ 3. Stopping Criterion• If the expected loss is less than some threshold that we don't care about, then we stop.– That means that as the experiment goes on, if the two groups are actually different, then the expected loss will actually shrink.– This is because the two distributions are getting further and further apart, and the variance surrounding these distributions is growing thinner and thinner.– So, even if we make a mistake and launch the wrong one, the probability of that happening grows smaller and smaller.– The threshold should be something that we're comfortable with losing. 4. Why Bayesian A/B testing?• Generally, it's easier to interpret the results.• If I tell you p-value is 0.05${\displaystyle 0.05}$ and the CI is some amount, although informational, but more often we want to know what's the probability that the experience B${\displaystyle B}$ is better than the experience A${\displaystyle A}$.– With the Bayesian approach we can answer this question directly.• Also, there's often fewer samples that we have to collect to reach a launch decision.– This means faster experiments and faster improvements.• One of the tools, in addition to frequentist tools, is Visual Web Optimizer. Back to Top