1. Bonferroni-adjusted multiple t-tests(Dunn) 2. Sidak test 3. Dunnett's test 4. Tukey honestly significant difference (HSD) test 5. Games and Howell's modification of Tukey's HSD 6. Tukey's wholly significant difference (WSD) test 7. Newman-Keuls test(Student-Newman-Keuls) 8. Ryan test (REGWQ) 9. The Shaffer-Ryan test 10. The least significant difference test (LSD)/ Fisher's LSD |
11. The Fisher-Hayter test 12. Waller-Duncan test 13. Games-Howell GH 14. Dunnett's T3 and Dunnett's C 15. Tamhane's T2 16. The Tukey-Kramer test 17. The Miller-Winer test 18. Multiple range (homogeneous subset) tests 19. The Scheff test 20. The Duncan's test 21. Hochberg's GT2 test 22. Gabriel test |
Test Name | Property/approach | When to use | Remarks |
Bonferroni-adjusted multiple t-tests(Dunn) | It is a simple type of multiple comparison test | 1) Used only when there are few number of comparisons. 2) It can be suitable for nonpairwise as well as pair wise comparisons | |
Sidak test | The alpha significance level for multiple comparisons is closer than the Bonferroni test | It controls the family wise error rate when the comparisons are orthogal to each other | |
Dunnett's test | Used when one want to compare each treatment group mean with the mean of the control group | ||
Tukey honestly significant difference (HSD) test | It is a very conservative pair wise comparison test as large number of groups would inflate Type I errors | 1) When the number of groups is large. 2) When all pair wise comparisons are tested | Post Hoc Conservative |
Games and Howell's modification of Tukey's HSD | Modified HSD test | Suitable when the homogeneity of variances assumption is violated. | Post HOC relatively liberal |
Tukey's wholly significant difference (WSD) test | Less conservative version of Tukey's HSD | Post HOC less conservative | |
Newman-Keuls test(Student-Newman-Keuls) | Based on the q-statistic, which is used to evaluate partial null hypotheses | 1) Recommended when the researcher wants to compare adjacent means. 2) only when the number of groups to be compared equals three | |
Ryan test (REGWQ): | 1) It is Modified version of Newman-Keuls test and in this test alpha level will decrease when stretch size decrease. 2)It controls the alpha rate at the desired level (ex., .05) even when the number of groups exceeds three but with less power than Newman-Keuls.3) It is a step-down procedure | Best choice as it Maintains good alpha control with 75% of the power | |
The Shaffer-Ryan test | Modified Ryan test and also step-down test, | Best multiple comparison tests in terms of power | |
The least significant difference test (LSD)/ Fisher's LSD | Based on the t-statistic | 1) Suitale for both pair wise and nonpairwise comparisons. 2) It will not require equal sample sizes | LSD is the most liberal of the post-hoc tests with Poor control of alpha |
The Fisher-Hayter test | Modified LSD with control on alpha | It is suitable when all pair wise comparisons are done post-hoc, but power may be low for fewer comparisons | |
The Scheff? test | It checks whether overall null hypothesis is rejected. If rejected F values are computed for all possible comparisons. | When the number of comparisons is large | Mostly used method to control alpha with the cost of less power More conservative than Tukey test |
The Duncan's test | Identical to Student Newman-keuls test.It uses error rate for collection of test. | ||
Hochberg's GT2 test | Similar to Tukey's Honestly Significant test uses studentised maximum modules | ||
Gabriel test | It is Pairwise comparison test uses studentised maximum modules. | Powerful than Hochberg's test when unequal sample size.Liberal when sample size vary largely | |
Waller-Duncan test | Multiple comparison test based on t-statistic and uses bayesian approach | ||
Games-Howell GH | 1) When we have unequal variances and unequal sample sizes this test can be used. 2) It is based on the q-statistic distribution | Liberal test | |
Dunnett's T3 and Dunnett's C | It maintain strict control over the alpha significance level | Unequal variance and unequal sample size | |
Tamhane's T2 | Unequal variance and unequal sample size | Conservative test | |
The Tukey-Kramer test | This is used when we have equal variances and unequal sample size | ||
The Miller-Winer test | When Equal variances assumed | ||
Multiple range (homogeneous subset) tests | Based on q statistics |
Bonferroni-adjusted multiple t-tests (Dunn)
It is a simple type of multiple comparison tests. It keeps the family wise
error to a fixed value. Family wise error is defined as the probability of
having at least any one of the test results in a type 1 error out m series
of test.
Positives
1.It used only when there is few number of comparisons
2.Suitable for non-pairwise as well as pair wise comparisons
Limitations
1.It does not have enough power to detect the difference as significant
Sidak Test
The alpha significance level for multiple comparisons is closer than the Bonferroni test
Positives
Limitations
1.It controls the family wise error rate when the controls are orthogonal to each other
R-code
group<-c(4,1,4,4,2,1,2,1,4,3,2,2,3,1,4,3,3,2,2,1,3,4,3,1,4,2,4,3,1,2,1,2,3,
4,4,2,2,1,4,3,1,2,4,1,3,2,3,1,2,3,1,4,1,2,3,3,2,3,3,2,4,2,4,3,4,4,1,2,1,4,2,3,3,3,2,4,2,1,1,3,3)survival_in_month<-c(145,198,122,108,61,82,121,191,191,109,82,74,85,193,65,173,129,107,157,
175,173,169,104,2,129,181,189,9,190,70,7,190,86,141,64,134,121,200,177,59,26,137,64,
18,55,68,30,147,30,151,86,83,22,169,103,13,2,183,27,60,132,109,172,
13,44,124,87,31,130,90,69,37,57,195,4,195,62,144,35,187,193)
tapply(survival_in_month, group, mean)
pairwise.t.test(survival_in_month, group, p.adj = "bonferroni")
References
1. Kucuk, U., Eyuboglu, M., Kucuk, H. O., & Degirmencioglu, G. (2016). Importance of using proper post hoc test with ANOVA. International journal of cardiology, 209, 346.
2. Rice, W. R. (1989). Analyzing tables of statistical tests. Evolution, 43(1), 223-225.
3. Wallenstein, S. Y. L. V. A. N., Zucker, C. L., & Fleiss, J. L. (1980). Some statistical methods useful in circulation research. Circulation Research,47(1), 1-9.
4. Nakagawa, Shinichi. "A farewell to Bonferroni: the problems of low statistical power and publication bias." Behavioral Ecology 15, no. 6 (2004): 1044-1045.
5. Elliott, A. C., & Hynan, L. S. (2011). A SAS macro implementation of a multiple comparison post hoc test for a Kruskal Wallis analysis. Computer methods and programs in biomedicine, 102(1), 75-80.
6. Brown, A. M. (2005). A new software for carrying out one-way ANOVA post hoc tests. Computer methods and programs in biomedicine, 79(1), 89-95.
7. Blakesley, R. E., Mazumdar, S., Dew, M. A., Houck, P. R., Tang, G., Reynolds III, C. F., & Butters, M. A. (2009). Comparisons of methods for multiple hypothesis testing in neuropsychological research. Neuropsychology, 23(2), 255.
8. Dinno, A., & Dinno, M. A. (2015). Package dunn. test.