Hypothesis testing is the methodology used to test the Null hypothesis, which is the hypothesis essentially of no effect, against some Alternative hypothesis that states that the effect is present and produces test statistic values that tend to be higher (lower) than what would be expected under the Null hypothesis.
The typical hypothesis testing scenario consists of five steps.
- Formulating the Null Hypothesis that states that the observations are the result of pure chance.
- Formulating the Alternative hypothesis that states that the observations show a real effect.
- Defining a test statistic whose distribution under the Null hypothesis is known or can be determined and whose distribution under the Alternative hypothesis is either shifted to larger values or smaller values.
- Computing the probability that the test statistic value would be as large (small) as that observed with the real data. If this probability is small, then it indicates that under the Null hypothesis, the observation made with the real data is highly unlikely. And therefore, it provides evidence against the Null hypothesis. This probability is called the p-value produced by the test.
- Comparing the p-value to pre-determined significance level If the p-level is smaller than the significance level, then the observed effect is statistically significant, and the Null hypothesis is rejected. If the p-level is larger than the significance level, then the observed effect is not statistically significant and the Null hypothesis is not rejected. If the Null hypothesis is not rejected, we consider that there is positive evidence to support the Alternative hypothesis.