Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs

Daniël Lakens, Daniël Lakens

Abstract

Effect sizes are the most important outcome of empirical studies. Most articles on effect sizes highlight their importance to communicate the practical significance of results. For scientists themselves, effect sizes are most useful because they facilitate cumulative science. Effect sizes can be used to determine the sample size for follow-up studies, or examining effects across studies. This article aims to provide a practical primer on how to calculate and report effect sizes for t-tests and ANOVA's such that effect sizes can be used in a-priori power analyses and meta-analyses. Whereas many articles about effect sizes focus on between-subjects designs and address within-subjects designs only briefly, I provide a detailed overview of the similarities and differences between within- and between-subjects designs. I suggest that some research questions in experimental psychology examine inherently intra-individual effects, which makes effect sizes that incorporate the correlation between measures the best summary of the results. Finally, a supplementary spreadsheet is provided to make it as easy as possible for researchers to incorporate effect size calculations into their workflow.

Keywords: cohen's d; effect sizes; eta-squared; power analysis; sample size planning.

References

    1. Aberson C. L. (2010). Applied Power Analysis for the Behavioral Sciences. New York, NY: Routledge
    1. Bakeman R. (2005). Recommended effect size statistics for repeated measures designs. Behav. Res. Methods 37, 379–384 10.3758/BF03192707
    1. Bakker M., van Dijk A., Wicherts J. M. (2012). The rules of the game called psychological science. Perspect. Psychol. Sci. 7, 543–554 10.1177/1745691612459060
    1. Borenstein M., Hedges L. V., Higgins J. P., Rothstein H. R. (2011). Introduction to Meta-Analysis. Hoboken, NJ: Wiley
    1. Brand A., Bradley M. T., Best L. A., Stoica G. (2008). Accuracy of effect size estimates from published psychological research. Percept. Mot. Skills 106, 645–649 10.2466/pms.106.2.645-649
    1. Cohen J. (1965). Some statistical issues in psychological research,” in Handbook of Clinical Psychology, ed Wolman B. B. (New York, NY: McGraw-Hill; ), 95–121
    1. Cohen J. (1988). Statistical Power Analysis for the Behavioral Sciences. New York, NY: Routledge Academic
    1. Cohen J. (1995). The earth is round (p<. 05): Rejoinder. Am. Psychol. 50, 1103 10.1037/0003-066X.50.12.1103
    1. Cumming G. (2012). Understanding the New Statistics: Effect sizes, Confidence Intervals, and Meta-Analysis. New York, NY: Routledge
    1. Cumming G., Finch S. (2005). Inference by eye: confidence intervals and how to read pictures of data. Am. Psychol. 60, 170–180 10.1037/0003-066X.60.2.170
    1. Dunlap W. P., Cortina J. M., Vaslow J. B., Burke M. J. (1996). Meta-analysis of experiments with matched groups or repeated measures designs. Psychol. Methods 1, 170–177 10.1037/1082-989X.1.2.170
    1. Dutilh G., van Ravenzwaaij D., Nieuwenhuis S., van der Maas H. L., Forstmann B. U., Wagenmakers E. J. (2012). How to measure post-error slowing: a confound and a simple solution. J. Math. Psychol. 56, 208–216 10.1016/j.jmp.2012.04.001
    1. Ellis P. D. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge: Cambridge University Press; 10.1017/CBO9780511761676
    1. Faul F., Erdfelder E., Buchner A., Lang A.-G. (2009). Statistical power analyses using G*Power 3.1: tests for correlation and regression analyses. Behav. Res. Methods 41, 1149–1160 10.3758/BRM.41.4.1149
    1. Fidler F. (2002). The fifth edition of the APA Publication Manual: Why its statistics recommendations are so controversial. Educ. Psychol. Meas. 62, 749–770 10.1177/001316402236876
    1. Fiedler K., Kutzner F., Krueger J. I. (2012). The long way from α-error control to validity proper problems with a short-sighted false-positive debate. Perspect. Psychol. Sci. 7, 661–669 10.1177/1745691612462587
    1. Glass G. V., McGaw B., Smith M. L. (1981). Meta-Analysis in Social Research. Beverly Hills, CA: Sage
    1. Grissom R. J., Kim J. J. (2005). Effect Sizes for Research: A Broad Practical Approach. Mahwah, NJ: Lawrence Erlbaum Associates
    1. Hayes W. L. (1963). Statistics for Psychologists. New York, NY: Holt, Rinehart and Winston
    1. Hedges L. V., Olkin I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press
    1. Kelley K. (2005). The effects of nonnormal distributions on confidence intervals around the standardized mean difference: bootstrap and parametric confidence intervals. Educ. Psychol. Meas. 65, 51–69 10.1177/0013164404264850
    1. Keppel G. (1991). Design and Analysis: A researcher's handbook. Englewood Cliffs, NJ: Prentice Hall
    1. Kline R. B. (2004). Beyond Signi?cance Testing: Reforming Data Analysis Methods in Behavioral Research. Washington DC: American Psychological Association; 10.1037/10693-000
    1. Lane D. M., Dunlap W. P. (1978). Estimating effect size: bias resulting from the significance criterion in editorial decisions. Br. J. Math. Stat. Psychol. 31, 107–112 10.1111/j.2044-8317.1978.tb00578.x
    1. Loftus G. R., Masson M. E. (1994). Using confidence intervals in within-subjects designs. Psychon. Bull. Rev. 1, 476–490 10.3758/BF03210951
    1. Maxwell S. E., Delaney H. D. (2004). Designing experiments and analyzing data: A model comparison perspective, 2nd Edn Mahwah, NJ: Erlbaum
    1. Maxwell S. E., Kelley K., Rausch J. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annu. Rev. Psychol. 59, 537–563 10.1146/annurev.psych.59.103006.093735
    1. McGrath R. E., Meyer G. J. (2006). When effect sizes disagree: the case of r and d. Psychol. Methods 11, 386–401 10.1037/1082-989X.11.4.386
    1. McGraw K. O., Wong S. P. (1992). A common language effect size statistic. Psychol. Bull. 111, 361–365 10.1037/0033-2909.111.2.361
    1. Morris S. B., DeShon R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychol. Methods 7, 105–125 10.1037/1082-989X.7.1.105
    1. Murphy K., Myors B., Wolach A. (2012). Statistical Power Analysis: A Simple and General Model for Traditional and Modern Hypothesis Tests. New York, NY: Routledge Academic
    1. Murphy K. R., Myors B. (1999). Testing the hypothesis that treatments have negligible effects: minimum-effect tests in the general linear model. J. Appl. Psychol. 84, 234–248 10.1037/0021-9010.84.2.234
    1. Olejnik S., Algina J. (2000). Measures of effect size for comparative studies: applications, interpretations, and limitations. Contemp. Educ. Psychol. 25, 241–286 10.1006/ceps.2000.1040
    1. Olejnik S., Algina J. (2003). Generalized eta and omega squared statistics: measures of effect size for some common research designs. Psychol. Methods 8, 434–447 10.1037/1082-989X.8.4.434
    1. Poincaré H. (1952). Science and Method. New York, NY: Dover Publications
    1. Preacher K. J., Kelley K. (2011). Effect size measures for mediation models: quantitative strategies for communicating indirect effects. Psychol. Methods 16, 93–115 10.1037/a0022658
    1. Rabbit P. M. A. (1966). Errors and error correction in choice reaction tasks. J. Exp. Psychol. 71, 264–272 10.1037/h0022853
    1. Rosenthal R. (1991). Meta-analytic procedures for social research. Newbury Park, CA: SAGE Publications, Incorporated
    1. Rosenthal R. (1994). Parametric measures of effect size, in The hand-book of research synthesis, eds Cooper H., Hedges L. V. (New York, NY: Sage; ), 231–244
    1. Schmidt F. L. (1992). What do data really mean. Am. Psychol. 47, 1173–1181 10.1037/0003-066X.47.10.1173
    1. Smithson M. (2001). Correct confidence intervals for various regression effect sizes and parameters: the importance of noncentral distributions in computing intervals. Educ. Psychol. Meas. 61, 605–632 10.1177/00131640121971392
    1. Tabachnick B. G., Fidell L. S. (2001). Using Multivariate Statistics, 4th Edn. Boston: Allyn and Bacon
    1. Thompson B. (2006). Foundations of Behavioral Statistics: An Insight-Based Approach. New York, NY: Guilford
    1. Thompson B. (2007). Effect sizes, confidence intervals, and confidence intervals for effect sizes. Psychol. Sch. 44, 423–432 10.1002/pits.20234
    1. Winkler R. L., Hays W. L. (1975). Statistics: Probability, Inference, and Decision, 2nd Edn. New York, NY: Holt

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe