How do you distinguish between a placebo and an active treatment?

1 The null and alternative hypotheses

The first step is to formulate two hypotheses: the null hypothesis and the alternative hypothesis. The null hypothesis is the assumption that there is no difference between the placebo and the active treatment, and any observed difference is due to random variation. The alternative hypothesis is the opposite, that there is a difference between the placebo and the active treatment, and the difference is not due to chance. For example, if you are testing a new painkiller, the null hypothesis would be that the painkiller has no effect on reducing pain, and the alternative hypothesis would be that the painkiller does reduce pain.

2 The p-value and significance level

The next step is to calculate the p-value, which is the probability of observing a difference between the placebo and the active treatment as extreme or more extreme than the one observed, assuming that the null hypothesis is true. The smaller the p-value, the less likely it is that the difference is due to chance, and the more likely it is that the alternative hypothesis is true. To decide whether to reject or accept the null hypothesis, you need to compare the p-value with a significance level, which is a threshold that you set before conducting the test. The significance level is usually 0.05, which means that you are willing to accept a 5% chance of making a wrong decision. If the p-value is less than or equal to the significance level, you reject the null hypothesis and conclude that there is a difference between the placebo and the active treatment. If the p-value is greater than the significance level, you fail to reject the null hypothesis and conclude that there is no evidence of a difference.

3 The confidence interval and effect size

Another way to distinguish between a placebo and an active treatment is to use a confidence interval and an effect size. A confidence interval is a range of values that contains the true difference between the placebo and the active treatment with a certain level of confidence, usually 95%. If the confidence interval does not include zero, it means that the difference is statistically significant. An effect size is a measure of how large or meaningful the difference is, regardless of the sample size. There are different ways to calculate the effect size, such as Cohen's d, which is the difference between the means divided by the pooled standard deviation. A larger effect size indicates a stronger effect of the active treatment.

4 The power and sample size

The last step is to consider the power and sample size of your test. The power is the probability of rejecting the null hypothesis when it is false, or finding a difference when there is one. The sample size is the number of participants or observations in your test. The power and sample size are related: the larger the sample size, the higher the power, and vice versa. A low power means that you have a high chance of missing a real difference, or making a type II error. A high power means that you have a high chance of detecting a real difference, or avoiding a type II error. You can use a power analysis to determine the optimal sample size for your test, based on the expected effect size, significance level, and power.

5 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?