What does this calculator do?
This Bayesian Split Test Calculator helps you interpret the results of your split tests using Bayesian statistics. It calculates the probability that each variant in your test is the best performer, as well as providing a 95% credible interval for the true conversion rate of each variant.
Benefits of using a Bayesian calculator
- Intuitive interpretation: Results are presented as probabilities, which are easier to understand and act upon than p-values.
- No fixed sample size: You can analyze your test data at any point without the statistical issues associated with repeated significance testing.
- Handles small sample sizes better: Bayesian methods can provide meaningful results even with smaller sample sizes.
- Incorporates uncertainty: The credible intervals give you a clear picture of the range of likely true conversion rates.
- Allows for multiple comparisons: You can compare more than two variants without increasing the risk of false positives.
Assumptions and limitations
While this calculator is a powerful tool for split test analysis, it's important to be aware of its assumptions and limitations:
- Independent observations: The calculator assumes that each conversion is independent of others. This may not hold true if, for example, the same user can convert multiple times.
- Stable population: It assumes that the underlying population doesn't change significantly during the test period.
- No time effects: The calculator doesn't account for potential changes in conversion rates over time (e.g., due to seasonality).
- Beta prior: The analysis uses a non-informative Beta(1,1) prior. If you have strong prior beliefs about conversion rates, this might not be appropriate.
- Conversion metric: This tool is designed for binary (success/failure) conversion events. It may not be suitable for other types of metrics (e.g., revenue per user).
How to use the results
When interpreting the results:
- Look at the "Probability of Being Best" to understand which variant is most likely to be the top performer.
- Use the 95% credible intervals to gauge the uncertainty around each variant's true conversion rate.
- Consider practical significance: even if a variant has a high probability of being best, the actual difference in conversion rates might be too small to justify implementation costs.
- Remember that statistical analysis is just one input into your decision-making process. Consider other factors like user experience, long-term effects, and business strategy.