vinaitheerthan


Multiple Comparison and Post Hoc Test (Click here for Sample SAS Code)

1. Bonferroni-adjusted multiple t-tests(Dunn)
2. Sidak test
3. Dunnett's test
4. Tukey honestly significant difference (HSD) test
5. Games and Howell's modification of Tukey's HSD
6. Tukey's wholly significant difference (WSD) test
7. Newman-Keuls test(Student-Newman-Keuls)
8. Ryan test (REGWQ)
9. The Shaffer-Ryan test
10. The least significant difference test (LSD)/ Fisher's LSD
11. The Fisher-Hayter test
12. Waller-Duncan test
13. Games-Howell GH
14. Dunnett's T3 and Dunnett's C
15. Tamhane's T2
16. The Tukey-Kramer test
17. The Miller-Winer test
18. Multiple range (homogeneous subset) tests
19. The Scheff  test
20. The Duncan's test
21. Hochberg's GT2 test
22. Gabriel test




Test Name Property/approach When to use Remarks
Bonferroni-adjusted multiple t-tests(Dunn) It is a simple type of multiple comparison test 1) Used only when there are few number of comparisons. 2) It can be suitable for nonpairwise as well as pair wise comparisons  
Sidak test The alpha significance level for multiple comparisons is closer than the Bonferroni test   It controls the family wise error rate when the comparisons are orthogal to each other
Dunnett's test   Used when one want to compare each treatment group mean with the mean of the control group  
Tukey honestly significant difference (HSD) test It is a very conservative pair wise comparison test as large number of groups would inflate Type I errors 1) When the number of groups is large. 2) When all pair wise comparisons are tested Post Hoc Conservative
Games and Howell's modification of Tukey's HSD Modified HSD test Suitable when the homogeneity of variances assumption is violated. Post HOC relatively liberal
Tukey's wholly significant difference (WSD) test Less conservative version of Tukey's HSD   Post HOC less conservative
Newman-Keuls test(Student-Newman-Keuls) Based on the q-statistic, which is used to evaluate partial null hypotheses 1) Recommended when the researcher wants to compare adjacent means. 2) only when the number of groups to be compared equals three  
Ryan test (REGWQ): 1) It is Modified version of Newman-Keuls test and in this test alpha level will decrease when stretch size decrease. 2)It controls the alpha rate at the desired level (ex., .05) even when the number of groups exceeds three but with less power than Newman-Keuls.3) It is a step-down procedure   Best choice as it Maintains good alpha control with 75% of the power
The Shaffer-Ryan test Modified Ryan test and also step-down test,   Best multiple comparison tests in terms of power
The least significant difference test (LSD)/ Fisher's LSD Based on the t-statistic 1) Suitale for both pair wise and nonpairwise comparisons. 2) It will not require equal sample sizes LSD is the most liberal of the post-hoc tests with Poor control of alpha
The Fisher-Hayter test Modified LSD with control on alpha It is suitable when all pair wise comparisons are done post-hoc, but power may be low for fewer comparisons  
The Scheff? test It checks whether overall null hypothesis is rejected. If rejected F values are computed for all possible comparisons. When the number of comparisons is large Mostly used method to control alpha with the cost of less power   More conservative than Tukey test
The Duncan's test Identical to Student Newman-keuls test.It uses error rate for collection of test.    
Hochberg's GT2 test Similar to Tukey's Honestly Significant test uses studentised maximum modules    
Gabriel test It is Pairwise comparison test uses studentised maximum modules.   Powerful than Hochberg's test when unequal sample size.Liberal when sample size vary largely
Waller-Duncan test Multiple comparison test based on t-statistic and uses bayesian approach    
Games-Howell GH   1) When we have unequal variances and unequal sample sizes this test can be used. 2) It is based on the q-statistic distribution Liberal test
Dunnett's T3 and Dunnett's C It maintain strict control over the alpha significance level Unequal variance and unequal sample size  
Tamhane's T2   Unequal variance and unequal sample size Conservative test
The Tukey-Kramer test   This is used when we have equal variances and unequal sample size  
The Miller-Winer test   When Equal variances assumed  
Multiple range (homogeneous subset) tests Based on q statistics    

Bonferroni-adjusted multiple t-tests (Dunn)

It is a simple type of multiple comparison tests. It keeps the family wise error to a fixed value. Family wise error is defined as the probability of having at least any one of the test results in a type 1 error out m series of test.
Positives
1.It used only when there is few number of comparisons
2.Suitable for non-pairwise as well as pair wise comparisons
Limitations
1.It does not have enough power to detect the difference as significant

Sidak Test

The alpha significance level for multiple comparisons is closer than the Bonferroni test
Positives

Limitations
1.It controls the family wise error rate when the controls are orthogonal to each other

R-code

group<-c(4,1,4,4,2,1,2,1,4,3,2,2,3,1,4,3,3,2,2,1,3,4,3,1,4,2,4,3,1,2,1,2,3,

4,4,2,2,1,4,3,1,2,4,1,3,2,3,1,2,3,1,4,1,2,3,3,2,3,3,2,4,2,4,3,4,4,1,2,1,4,2,3,3,3,2,4,2,1,1,3,3)


survival_in_month<-c(145,198,122,108,61,82,121,191,191,109,82,74,85,193,65,173,129,107,157,

175,173,169,104,2,129,181,189,9,190,70,7,190,86,141,64,134,121,200,177,59,26,137,64,

18,55,68,30,147,30,151,86,83,22,169,103,13,2,183,27,60,132,109,172,

13,44,124,87,31,130,90,69,37,57,195,4,195,62,144,35,187,193)


tapply(survival_in_month, group, mean)
pairwise.t.test(survival_in_month, group, p.adj = "bonferroni")
 
References

1.     Kucuk, U., Eyuboglu, M., Kucuk, H. O., & Degirmencioglu, G. (2016). Importance of using proper post hoc test with ANOVA. International journal of cardiology, 209, 346.

2.     Rice, W. R. (1989). Analyzing tables of statistical tests. Evolution, 43(1), 223-225.

3.     Wallenstein, S. Y. L. V. A. N., Zucker, C. L., & Fleiss, J. L. (1980). Some statistical methods useful in circulation research. Circulation Research,47(1), 1-9.

4.     Nakagawa, Shinichi. "A farewell to Bonferroni: the problems of low statistical power and publication bias." Behavioral Ecology 15, no. 6 (2004): 1044-1045.

5.     Elliott, A. C., & Hynan, L. S. (2011). A SAS macro implementation of a multiple comparison post hoc test for a Kruskal Wallis analysis. Computer methods and programs in biomedicine, 102(1), 75-80.

6.     Brown, A. M. (2005). A new software for carrying out one-way ANOVA post hoc tests. Computer methods and programs in biomedicine, 79(1), 89-95.

7.     Blakesley, R. E., Mazumdar, S., Dew, M. A., Houck, P. R., Tang, G., Reynolds III, C. F., & Butters, M. A. (2009). Comparisons of methods for multiple hypothesis testing in neuropsychological research. Neuropsychology, 23(2), 255.
8.      Dinno, A., & Dinno, M. A. (2015). Package dunn. test.