Making 'null effects' informative: statistical techniques and inferential frameworks
Being able to interpret ‘null effects’ is important for cumulative knowledge generation in science. To draw informative conclusions from null-effects, researchers need to move beyond the incorrect interpretation of a non-significant result in a null-hypothesis significance test as evidence of the absence of an effect. We explain how to statistically evaluate null-results using equivalence tests, Bayesian estimation, and Bayes factors. A worked example demonstrates how to apply these statistical tools and interpret the results. Finally, we explain how no statistical approach can actually prove that the null-hypothesis is true, and briefly discuss the philosophical differences between statistical approaches to examine null-effects. The increasing availability of easy-to-use software and online tools to perform equivalence tests, Bayesian estimation, and calculate Bayes factors make it timely and feasible to complement or move beyond traditional null-hypothesis tests, and allow researchers to draw more informative conclusions about null-effects.
Relevance for Patients: Conclusions based on clinical trial data often focus on demonstrating differences due to treatments, despite demonstrating the absence of differences is an equally important statistical question. Researchers commonly conclude the absence of an effect based on the incorrect use of traditional statistical methods. By providing an accessible overview of different approaches to exploring null-results, we hope researchers improve their statistical inferences. This should lead to a more accurate interpretation of studies, and facilitate knowledge generation about proposed treatments.
[1] R Core Team. R: A language and environment for statistical com puting. Vienna, Austria: R Foundation for Statistical Computing; 2017.
[2] JASP Team. JASP (Version 0.8.6) [Computer software]. 2018.
[3] jamovi project. jamovi (Version 0.8) [Computer Software]. 2018.
[4] EarpBD.Theneedforreportingnegativeresults- a 90 year update. J Clin Transl Res. 2017;3(S2):1–4.
[5] Kühberger A, Fritz A, Scherndl T. Publication Bias in Psychol ogy: A Diagnosis Based on the Correlation between Effect Size and Sample Size. PLoS ONE. 2014;9:e105825.
[6] Locascio JJ. Results blind science publishing. Basic and Applied Social Psychology. 2017;39:239–246.
[7] MeehlPE.WhySummariesofResearchonPsychologicalTheories are Often Uninterpretable. Psychological Reports. 1990;66:195.
[8] Abdel Shaheed C, Maher CG, Williams KA, Day R, McLachlan AJ. Efficacy, Tolerability, and Dose-Dependent Effects of Opi oid Analgesics for Low Back Pain. JAMA Internal Medicine. 2016;176:958.
[9] Machado GC, Maher CG, Ferreira PH, Pinheiro MB, Lin C-WC, Day RO, et al. Efficacy and safety of paracetamol for spinal pain and osteoarthritis: systematic review and meta-analysis of ran domised placebo controlled trials. BMJ. 2015;350:h1225.
[10] Hayes AF. A Primer on Multilevel Modeling. Human Communi cation Research. 2006;32:385–410.
[11] Lakens D, Adolfi FG, Albers CJ, Anvari F, Apps MAJ, Argamon SE, et al. Justify your alpha. Nature Human Behaviour. 2018 Feb;
[12] Neyman J, Pearson ES. On the Problem of the Most Efficient Tests of Statistical Hypotheses. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sci ences. 1933;231:289–337.
[13] Perezgonzalez JD. Fisher, Neyman-Pearson or NHST? A tutorial for teaching data testing. Frontiers in Psychology. 2015;6:1–11.
[14] Meyners M. Equivalence tests– A review. Food Quality and Pref erence. 2012;26:231–245.
[15] Schuirmann DJ. A comparison of the Two One-Sided Tests Pro cedure and the Power Approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Biophar maceutics. 1987;15:657–680.
[16] Rogers JL, Howard KI, Vessey JT. Using significance tests to eval uate equivalence between two experimental groups. Psychological Bulletin. 1993;113:553–565.
[17] Wandel S, Juni P, Tendal B, Nuesch E, Villiger PM, Welton NJ, et al. Effects of glucosamine, chondroitin, or placebo in patients with osteoarthritis of hip or knee: network meta-analysis. BMJ. 2010;341:c4675.
[18] Lakens D, Scheel AM, Isager PM. Equivalence testing for psycho logical research: A tutorial. PsyArXiv; 2018.
[19] LakensD.Equivalence Tests. Social Psychological and Personality Science. 2017;8:355–362.
[20] Morey RD, Hoekstra R, Rouder JN, Lee MD, Wagenmakers E-J. The fallacy of placing confidence in confidence intervals. Psycho nomic Bulletin Review. 2016;23:103–123.
[21] McElreath R. Statistical Rethinking: A Bayesian Course with Ex amples in R and Stan. CRC Press; 2016.
[22] Kruschke JK. Doing Bayesian Data Analysis: A tutorial with R, JAGS, and Stan. Academic Press; 2014.
[23] Gelman A. Induction and deduction in Bayesian data analysis. Ra tionality, Markets and Morals. 2011;2:67–78.
[24] Kruschke JK. Bayesian estimation supersedes the t test. Journal of Experimental Psychology: General. 2013;142:573–603.
[25] Kruschke JK, Meredith M. BEST: Bayesian estimation supersedes the t-test. 2017.
[26] Bååth R. Bayesian Estimation Supersedes the t-test (BEST)- on line. 2012.
[27] Kruschke JK, Liddell TM. The Bayesian New Statistics: Hypoth esis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin Review. 2017.
[28] Kruschke JK. Rejecting or accepting parameter values in Bayesian estimation. Advances in Methods and Practices in Psychological Science. in press.
[29] Greenwald AG. Consequences of Prejudice Against the Null Hy pothesis. Psychological Bulletin. 1975;82.
[30] Albers CJ, Kiers HAL, Ravenzwaaij D van. Credible Confidence: Apragmatic view on the frequentist vs Bayesian debate. 2018.
[31] Verhagen J, Wagenmakers E-J. Bayesian tests to quantify the re sult of a replication attempt. Journal of Experimental Psychology: General. 2014;143:1457–1475.
[32] Harms C. A Bayes Factor for Replications of ANOVA Results. ArXiv e-prints. 2016.
[33] DienesZ.UsingBayestogetthemostoutofnon-significantresults. Frontiers in Psychology. 2014;5:1–17.
[34] Rouder JN, Speckman PL, Sun D, MoreyRD,Iverson G. Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin Review. 2009;16:225–237.
[35] Kruschke JK, Liddell TM. Bayesian data analysis for newcomers. Psychonomic Bulletin Review. 2018;25(1):155–177.
[36] Hoffman BM, Papas RK, Chatkoff DK, Kerns RD. Meta-analysis of psychological interventions for chronic low back pain. Health Psychology. 2007;26:1–9.
[37] WagenmakersE-J,LodewyckxT,KuriyalH,GrasmanR.Bayesian hypothesis testing for psychologists: A tutorial on the Savage Dickey method. Cognitive Psychology. 2010;60:158–189.
[38] Gronau QF, Ly A, Wagenmakers E-J. Informed Bayesian t-tests. ArXiv e-prints. 2017.
[39] Jeffreys H. Theory of Probability. 3. ed. Oxford: Clarendon Press; 1961.
[40] SchönbrodtFD,WagenmakersE-J,ZehetleitnerM,PeruginiM.Se quential hypothesis testing with Bayes factors: Efficiently testing mean differences. Psychological Methods. 2017;22:322–339.
[41] Everson-Stewart S, Emerson SS. Bio-creep in non-inferiority clin ical trials. Statistics in Medicine. 2010;29:2769–2780.
[42] Aberegg SK, Hersh AM, Samore MH. Do non-inferiority trials of reduced intensity therapies show reduced effects? A descriptive analysis. BMJ Open. 2018;8:e019494.
[43] Beryl P, Vach W. Is there a danger of “biocreep” with non inferiority trials? Trials. 2011;12:A29.
[44] MoreyRD,RomeijnJ-W,RouderJN.ThephilosophyofBayesfac tors and the quantification of statistical evidence. Journal of Math ematical Psychology. 2016;72:6–18.