Improving the analysis of routine outcome measurement data: what a Bayesian approach can do for you

Rivka M de Vries, Rob R Meijer, Vincent van Bruggen, Richard D Morey, Rivka M de Vries, Rob R Meijer, Vincent van Bruggen, Richard D Morey

Abstract

Since recent decades, clinicians offering interventions against mental problems must systematically collect data on how clients change over time. Since these data typically contain measurement error, statistical tests have been developed which should disentangle true changes from random error. These statistical tests can be subdivided into two types: classical tests and Bayesian tests. Over the past, there has been much confusion among analysts regarding the questions that are answered by each of these tests. In this paper we discuss each type of test in detail and explain which questions are, and which are not, answered by each of the types of tests. We then apply a test of each type on an empirical data set and compare the results. Copyright © 2015 John Wiley & Sons, Ltd.

Keywords: Bayesian approach; classical approach; data analysis; evidence; hypothesis testing; routine outcome measurement.

Copyright © 2015 John Wiley & Sons, Ltd.

Figures

Figure 1
Figure 1
Two examples of the distribution of reliable change index (RCI) under the null and alternative hypothesis. In (A) the dot represents an RCI value of 1.3 and in (B) the dot represents an RCI value of 2.3.
Figure 2
Figure 2
Relation between Brief Symptom Inventory (BSI) scores and intervention time.
Figure 3
Figure 3
Bayes factors, which quantify the relative evidence in the data for zero mean change as compared to non‐zero mean change, for 188 persons plotted against the observed standardized mean difference d for each subject. Circles represent Bayes factors for subjects that have reliably changed according to the reliable change index (RCI), triangles represent Bayes factors for subjects that have not reliably changed. A Bayes factor of one implies that the data are equally likely under the null hypothesis as under the alternative hypothesis of change. A Bayes factor of, for example, 1/10 means that the data are 10 times more likely under the alternative hypothesis of change than under the null hypothesis of no change.

References

    1. Bauer S., Lambert M.J., Nielsen S.L. (2004) Clinical significance methods: a comparison of statistical techniques. Journal of Personality Assessment, 82, 60–70. DOI:10.1207/s15327752jpa820111.
    1. Buonaccorsi J.P. (2010) Measurement Error: Models, Methods, and Applications Chapman & Hall/CRC Interdisciplinary Statistics Series, Boca Raton, FL: CRC Press.
    1. Burgess P., Pirkis J., Coombs T. (2009) Modelling candidate effectiveness indicators for mental health services. Australian and New Zealand Journal of Psychiatry, 43(6), 531–538.
    1. Cohen J. (1992) A power primer. Psychological Bulletin, 112(1), 155–159.
    1. De Beurs E. (2010) De Genormaliseerde T‐score. Een “euro” voor testuitslagen [The normalized T‐score. A “euro” for test outcomes]. Maandblad Geestelijke volksgezondheid, 65(9), 684–695.
    1. Derogatis L.R. (2001) Brief Symptom Inventory (BSI)‐18. Administration, Scoring and Procedures Manual, Bloomington, MN: NCS Pearson, Inc.
    1. De Vries R.M., Morey R.D. (2013) Bayesian hypothesis testing for single‐subject designs. Psychological Methods, 18(1), 165–185.
    1. De Vries R.M., Hartogs B.M.A., Morey R.D. (in press) A tutorial on computing Bayes factors for single‐subject designs. Behavior Therapy. doi: 10.1016/j.beth.2014.09.013.
    1. De Vries R.M., Morey R.D., Tendeiro J.N. (in preparation) Bayesian hypothesis testing for routine outcome measurement data.
    1. Finch A.E., Lambert M.J., Schaalje B.G. (2001) Psychotherapy quality control: the statistical generation of expected recovery curves for integration into an early warning system. Clinical Psychology and Psychotherapy, 8, 231–242. DOI:10.1002/cpp.286.
    1. Good I.J. (1985) Weight of evidence: a brief survey In Bernardo J.M. (eds) Bayesian Statistics 2, pp. 249–270, Amsterdam, North‐Holland: Elsevier Science Publishers.
    1. Goodman S.N. (1999a) Toward evidence‐based medical statistics I. The P‐value fallacy. Annals of Internal Medicine, 130(12), 995–1004.
    1. Goodman S.N. (1999b) Toward evidence‐based medical statistics. 2: the Bayes factor. Annals of Internal Medicine, 130(12), 1005–1013.
    1. Jacobson N.S., Truax P. (1991) Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12–19.
    1. Johnson V.E. (2013) Revised standards for statistical evidence. Proceedings of the National Academy of Sciences, 110(48), 19313–19317.
    1. Morey R.D., Rouder J.N. (2011) Bayes factor approaches for testing interval null hypotheses. Psychological Methods, 16(4), 406–419.
    1. Lambert M.J., Whipple J.L., Bishop M.J., Vermeersch D.A., Gray G.V., Finch A.E. (2002) Comparison of empirically derived and rationally derived methods for identifying clients at risk for treatment failure. Clinical Psychology and Psychotherapy, 9(3), 149–164.
    1. Morey R.D., De Vries R.M. (2014) BayesSingleSub 0.6.3. Comprehensive R Archive Network. . [10 June 2015].
    1. Ogles B.M., Lunnen K.M., Bonesteel K. (2001) Clinical significance: history, application and current practice. Clinical Psychology Review, 21, 421–446. DOI:10.1016/S0272-7358(99)00058-6.
    1. Rouder J.N., Morey R.D., Verhagen J., Province J.M., Wagenmakers E.‐J. (submitted for publication) The p < 05 rule and the hidden costs of the free lunch in inference.
    1. Simpson S., Corney R., Fitzgerald P., Beecham J. (2003) A randomized controlled trial to evaluate the effectiveness and cost‐effectiveness of psychodynamic counselling for general practice patients with chronic depression. Psychological Medicine, 33(2), 229–239.
    1. Slade M. (2002a) Routine outcome assessment in mental health services. Psychological Medicine, 32(8), 1339–1343.
    1. Slade M. (2002b) What outcomes to measure in routine mental health services, and how to assess them: a systematic review. Australian and New Zealand Journal of Psychiatry, 36(6), 743–753.
    1. Spielmans G.I., Masters K.S., Lambert M.J. (2006) A comparison of rational versus empirical methods in the prediction of psychotherapy outcome. Clinical Psychology and Psychotherapy, 13(3), 202–214.
    1. Stiles W.B., Barkham M., Twigg E., Mellor‐Clark J., Cooper M. (2006) Effectiveness of cognitive‐behavioural, person‐centred and psychodynamic therapies as practised in UK National Health Service settings. Psychological Medicine, 36(4), 555–566.
    1. Van Hees S., Van der Vlist P., Mulder N. (2011) Van Weten naar Meten: ROM in de ggz [From knowledge to measurement: ROM in the ggz], Amsterdam: Uitgeverij Boom.
    1. Van Os J., Kahn R., Denys D., Schoevers R.A., Beekman A.T.F., Hoogendijk W.J.G., van Hemert A.M., Hodiamont P.P.G., Scheepers F., Delespaul Ph. A.E.G., Leentjens A.F.G. (2012) ROM: gedragsnorm of dwangmaatregel? Tijdschrift voor Psychiatrie, 54(3), 245–253.

Source: PubMed

Sponsorer og samarbejdspartnere

Medicinske tilstande

Narkotikainterventioner

3
Abonner