A post hoc test is a statistical analysis performed after an initial study has concluded and data has been collected. The term “post hoc” comes from Latin, meaning “after this.” These analyses uncover specific differences between groups when an overall statistical test has indicated that differences exist, allowing researchers to pinpoint the exact sources of variation. They help researchers delve deeper to identify where significant variations truly lie.
The Necessity of Post Hoc Tests
When researchers conduct multiple comparisons within a dataset without proper adjustments, they face an increased risk of making a Type I error. A Type I error occurs when a researcher incorrectly concludes that there is a significant difference between groups, when in reality, none exists. This problem becomes more pronounced with each additional comparison performed, leading to an inflated “family-wise error rate,” which represents the probability of making at least one Type I error across a set of comparisons.
For instance, if a researcher performs many individual comparisons, even with a standard significance level of 0.05, the overall chance of finding a “significant” result purely by chance rises considerably. Post hoc tests are designed to control this inflated family-wise error rate by adjusting the significance level for each individual comparison. This adjustment ensures that the overall probability of a Type I error across all comparisons remains at an acceptable, pre-determined level, typically 5%. This control maintains the integrity of statistical findings.
When to Apply Post Hoc Tests
Post hoc tests are applied following an initial statistical test, often referred to as an “omnibus” test, which has yielded a statistically significant result. An “omnibus” test indicates an overall significant difference among multiple groups without specifying where those differences lie. A common example is the Analysis of Variance (ANOVA), used to compare the means of three or more independent groups.
When an ANOVA or similar omnibus test produces a significant p-value (typically less than 0.05), it signals that at least one group mean is statistically different from the others. However, this initial test does not identify which specific pairs or combinations of groups contribute to this overall difference. Post hoc tests systematically explore all possible group comparisons to pinpoint these differences while controlling for the increased risk of false positives.
Conversely, if the omnibus test does not show a statistically significant result, researchers do not proceed with post hoc analyses. A non-significant omnibus test implies no overall evidence of systematic differences among group means. Performing further comparisons in such cases could lead to discovering spurious differences due to chance alone.
Choosing the Right Post Hoc Test
Selecting an appropriate post hoc test involves considering several factors to ensure the validity and reliability of findings. One consideration is the specific type of comparisons a researcher intends to make. Some tests compare every possible pair of groups (pairwise comparisons), while others compare multiple treatment groups against a single control group.
Another factor involves evaluating whether the underlying assumptions of the statistical tests are met, particularly the assumption of homogeneity of variances. This assumption implies that the variability within each group is roughly equal. If violated, certain post hoc tests may provide inaccurate results, necessitating alternative tests or adjustments.
Researchers also weigh the balance between controlling the Type I error rate and maintaining statistical power. More conservative tests limit false positives but might reduce the power to detect genuine differences. Less conservative tests offer greater power but carry a higher risk of Type I errors. The choice depends on the research question and the potential consequences of making either type of error.
Finally, the sample sizes of the groups can influence the choice of test. Some post hoc tests perform optimally with equal sample sizes, while others suit unequal group sizes, adjusting their calculations accordingly. Considering these factors helps researchers select a post hoc test that aligns with their research objectives, data characteristics, and acceptable levels of statistical error.
Exploring Common Post Hoc Tests
Several post hoc tests are commonly employed by researchers after an overall significant finding from an omnibus test.
Tukey’s Honestly Significant Difference (HSD) test compares all possible pairs of group means, providing a comprehensive overview of differences. This test is suitable when group sizes are equal and the assumption of homogeneity of variance holds, balancing Type I error control and statistical power.
The Bonferroni correction offers a versatile approach, applicable to a wide range of statistical tests. It controls the family-wise error rate by adjusting the significance level for each individual comparison, becoming more stringent as the number of comparisons increases. While straightforward to apply, Bonferroni can be conservative, potentially increasing the risk of Type II errors.
Scheffé’s Test is a conservative post hoc procedure, often employed for complex comparisons beyond simple pairwise differences, such as comparing combinations of groups or linear contrasts. It maintains control over the family-wise error rate, even when researchers explore a large number of unplanned comparisons, making it a reliable choice for exploratory analyses. This test is robust to unequal sample sizes and useful for identifying significant contrasts among the means.
For situations comparing multiple treatment groups to a single control group, Dunnett’s Test is often used. This test is tailored to be more powerful than general pairwise comparison tests like Tukey’s or Bonferroni when the primary interest lies in comparing experimental conditions against a designated baseline. It accounts for the shared control group across comparisons, leading to narrower confidence intervals.