Predictors, Outcomes, and Statistical Solutions of Missing Cases in Web-Based Psychotherapy: Methodological Replication and Elaboration Study

Eyal Karin, Monique Frances Crane, Blake Farran Dear, Olav Nielssen, Gillian Ziona Heller, Rony Kayrouz, Nickolai Titov, Eyal Karin, Monique Frances Crane, Blake Farran Dear, Olav Nielssen, Gillian Ziona Heller, Rony Kayrouz, Nickolai Titov

Abstract

Background: Missing cases present a challenge to our ability to evaluate the effects of web-based psychotherapy trials. As missing cases are often lost to follow-up, less is known about their characteristics, their likely clinical outcomes, or the likely effect of the treatment being trialed.

Objective: The aim of this study is to explore the characteristics of missing cases, their likely treatment outcomes, and the ability of different statistical models to approximate missing posttreatment data.

Methods: A sample of internet-delivered cognitive behavioral therapy participants in routine care (n=6701, with 36.26% missing cases at posttreatment) was used to identify predictors of dropping out of treatment and predictors that moderated clinical outcomes, such as symptoms of psychological distress, anxiety, and depression. These variables were then incorporated into a range of statistical models that approximated replacement outcomes for missing cases, and the results were compared using sensitivity and cross-validation analyses.

Results: Treatment adherence, as measured by the rate of progress of an individual through the treatment modules, and higher pretreatment symptom scores were identified as the dominant predictors of missing cases probability (Nagelkerke R2=60.8%) and the rate of symptom change. Low treatment adherence, in particular, was associated with increased odds of presenting as missing cases during posttreatment assessment (eg, odds ratio 161.1:1) and, at the same time, attenuated the rate of symptom change across anxiety (up to 28% of the total symptom with 48% reduction effect), depression (up to 41% of the total with 48% symptom reduction effect), and psychological distress symptom outcomes (up to 52% of the total with 37% symptom reduction effect) at the end of the 8-week window. Reflecting this pattern of results, statistical replacement methods that overlooked the features of treatment adherence and baseline severity underestimated missing case symptom outcomes by as much as 39% at posttreatment.

Conclusions: The treatment outcomes of the cases that were missing at posttreatment were distinct from those of the remaining observed sample. Thus, overlooking the features of missing cases is likely to result in an inaccurate estimate of the effect of treatment.

Keywords: missing data; psychotherapy; statistical bias; treatment adherence and compliance; treatment evaluation.

Conflict of interest statement

Conflicts of Interest: None declared.

©Eyal Karin, Monique Frances Crane, Blake Farran Dear, Olav Nielssen, Gillian Ziona Heller, Rony Kayrouz, Nickolai Titov. Originally published in JMIR Mental Health (http://mental.jmir.org), 05.02.2021.

Figures

Figure 1
Figure 1
Probability for observing cases at posttreatment (inverse probability of missing cases) and treatment outcome trends associated with treatment completion; 95% CI is drawn around each effect in dotted lines. PHQ-9: Patient Health Questionnaire-9.
Figure 2
Figure 2
Probability for observing cases at posttreatment (inverse probability of missing cases) and treatment outcomes trends associated with depressive symptoms baseline severity; 95% CI is drawn around each effect in dotted lines. PHQ-9: Patient Health Questionnaire-9.

References

    1. Fernandez E, Salem D, Swift J, Ramtahal Nirvana. Meta-analysis of dropout from cognitive behavioral therapy: magnitude, timing, and moderators. J Consult Clin Psychol. 2015 Dec;83(6):1108–22. doi: 10.1037/ccp0000044.
    1. Karyotaki E, Kleiboer A, Smit F, Turner DT, Pastor AM, Andersson G, Berger T, Botella C, Breton JM, Carlbring P, Christensen H, de Graaf E, Griffiths K, Donker T, Farrer L, Huibers MJH, Lenndin J, Mackinnon A, Meyer B, Moritz S, Riper H, Spek V, Vernmark K, Cuijpers P. Predictors of treatment dropout in self-guided web-based interventions for depression: an ‘individual patient data’ meta-analysis. Psychol. Med. 2015 Apr 17;45(13):2717–26. doi: 10.1017/s0033291715000665.
    1. Little RJ, D'Agostino R, Cohen ML, Dickersin K, Emerson SS, Farrar JT, Frangakis C, Hogan JW, Molenberghs G, Murphy SA, Neaton J, Rotnitzky A, Scharfstein D, Shih WJ, Siegel JP, Stern H. The prevention and treatment of missing data in clinical trials. N Engl J Med. 2012 Oct 04;367(14):1355–60. doi: 10.1056/nejmsr1203730.
    1. Karin E, Dear B, Heller G, Crane M, Titov N. "Wish You Were Here": examining characteristics, outcomes, and statistical solutions for missing cases in web-based psychotherapeutic trials. JMIR Ment Health. 2018 Apr 19;5(2):e22. doi: 10.2196/mental.8363.
    1. Bell M, Fairclough DL. Practical and statistical issues in missing data for longitudinal patient-reported outcomes. Stat Methods Med Res. 2014 Oct;23(5):440–59. doi: 10.1177/0962280213476378.
    1. RUBIN DB. Inference and missing data. Biometrika. 1976;63(3):581–92. doi: 10.1093/biomet/63.3.581.
    1. Lang K, Little TD. Principled missing data treatments. Prev Sci. 2018 Apr;19(3):284–94. doi: 10.1007/s11121-016-0644-5.
    1. Mealli F, Rubin Db. Clarifying missing at random and related definitions, and implications when coupled with exchangeability: Table 1. Biometrika. 2015 Sep 22;102(4):995–1000. doi: 10.1093/biomet/asv035. doi: 10.1093/biomet/asv035.
    1. Bell M, Fiero M, Horton N, Hsu CH. Handling missing data in RCTs; a review of the top medical journals. BMC Med Res Methodol. 2014 Nov 19;14:118. doi: 10.1186/1471-2288-14-118.
    1. Little RJ, Rubin DB. Bayesian Analysis. Hoboken, NJ: John Wiley & Sons; 2014. Statistical Analysis with Missing Data.
    1. Little RJA. Modeling the drop-out mechanism in repeated-measures studies. J of the Amer Stati Asso. 1995 Sep;90(431):1112–21. doi: 10.1080/01621459.1995.10476615.
    1. Schafer J, Graham Jw. Missing data: Our view of the state of the art. Psychological Methods. 2002;7(2):147–177. doi: 10.1037/1082-989X.7.2.147. doi: 10.1037/1082-989x.7.2.147.
    1. Alfonsson S, Olsson E, Hursti T. Motivation and treatment credibility predicts dropout, treatment adherence, and clinical outcomes in an internet-based cognitive behavioral relaxation program: a randomized controlled trial. J Med Internet Res. 2016 Mar 08;18(3):e52. doi: 10.2196/jmir.5352.
    1. A digital mental health clinic for all Australians. MindSpot Clinic. [2021-01-06].
    1. Titov N, Dear BF, Staples LG, Bennett-Levy J, Klein B, Rapee RM, Andersson G, Purtell C, Bezuidenhout G, Nielssen OB. The first 30 months of the MindSpot Clinic: Evaluation of a national e-mental health service against project objectives. Aust N Z J Psychiatry. 2017 Dec;51(12):1227–39. doi: 10.1177/0004867416671598.
    1. Kroenke K, Spitzer R, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001 Sep;16(9):606–13. doi: 10.1046/j.1525-1497.2001.016009606.x.
    1. Staples LG, Dear BF, Gandy M, Fogliati V, Fogliati R, Karin E, Nielssen O, Titov N. Psychometric properties and clinical utility of brief measures of depression, anxiety, and general distress: The PHQ-2, GAD-2, and K-6. Gen Hosp Psychiatry. 2019;56:13–18. doi: 10.1016/j.genhosppsych.2018.11.003.
    1. Spitzer R, Kroenke K, Williams JBW, Löwe Bernd. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006 May 22;166(10):1092–7. doi: 10.1001/archinte.166.10.1092.
    1. Kessler RC, Andrews G, Colpe LJ, Hiripi E, Mroczek DK, Normand SLT, Walters EE, Zaslavsky AM. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol Med. 2002 Aug;32(6):959–76. doi: 10.1017/s0033291702006074.
    1. Sieverink F, Kelders S, van Gemert-Pijnen Julia Ewc. Clarifying the concept of adherence to eHealth technology: systematic review on when usage becomes adherence. J Med Internet Res. 2017 Dec 06;19(12):e402. doi: 10.2196/jmir.8578.
    1. R Core Team . R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2021. [2020-06-06]. The R Project for Statistical Computing.
    1. Donohue MC, Gamst AC, Edland SD. Sample Size Calculations for L. CRAN; 2020. Apr 21, [2021-01-06]. Package ‘longpower’. .
    1. Harrell JF. Springer Series in Statistics. New York: Springer; 2015. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis.
    1. Nagelkerke NJD. A note on a general definition of the coefficient of determination. Biometrika. 1991;78(3):691–2. doi: 10.1093/BIOMET/78.3.691. doi: 10.1093/biomet/78.3.691.
    1. LIANG K, ZEGER SL. Longitudinal data analysis using generalized linear models. Biometrika. 1986;73(1):13–22. doi: 10.1093/biomet/73.1.13.
    1. Karin E, Dear B, Heller G, Gandy M, Titov N. Measurement of symptom change following web-based psychotherapy: statistical characteristics and analytical methods for measuring and interpreting change. JMIR Ment Health. 2018 Jul 12;5(3):e10200. doi: 10.2196/10200.
    1. Cook JA, Hislop J, Altman DG, Fayers P, Briggs AH, Ramsay CR, Norrie JD, Harvey IM, Buckley B, Fergusson D, Ford I, Vale LD, DELTA group Specifying the target difference in the primary outcome for a randomised controlled trial: guidance for researchers. Trials. 2015 Jan 15;16:12. doi: 10.1186/s13063-014-0526-8.
    1. Greene C, Morland L, Durkalski V, Frueh BC. Noninferiority and equivalence designs: issues and implications for mental health research. J Trauma Stress. 2008 Oct;21(5):433–9. doi: 10.1002/jts.20367.
    1. Delgadillo J, McMillan D, Lucock M, Leach C, Ali S, Gilbody S. Early changes, attrition, and dose-response in low intensity psychological interventions. Br J Clin Psychol. 2014 Mar;53(1):114–30. doi: 10.1111/bjc.12031. doi: 10.1111/bjc.12031.
    1. Robinson L, Delgadillo J, Kellett S. The dose-response effect in routinely delivered psychological therapies: a systematic review. Psychother Res. 2020 Jan 20;30(1):79–96. doi: 10.1080/10503307.2019.1566676. doi: 10.1080/10503307.2019.1566676.
    1. Cornelisz I, Cuijpers P, Donker T, van Klaveren C. Addressing missing data in randomized clinical trials: a causal inference perspective. PLoS One. 2020 Jul 6;15(7):e0234349. doi: 10.1371/journal.pone.0234349.
    1. Levis B, Benedetti A, Ioannidis JPA, Sun Y, Negeri Z, He C, Wu Y, Krishnan A, Bhandari PM, Neupane D, Imran M, Rice DB, Riehm KE, Saadat N, Azar M, Boruff J, Cuijpers P, Gilbody S, Kloda LA, McMillan D, Patten SB, Shrier I, Ziegelstein RC, Alamri SH, Amtmann D, Ayalon L, Baradaran HR, Beraldi A, Bernstein CN, Bhana A, Bombardier CH, Carter G, Chagas MH, Chibanda D, Clover K, Conwell Y, Diez-Quevedo C, Fann JR, Fischer FH, Gholizadeh L, Gibson LJ, Green EP, Greeno CG, Hall BJ, Haroz EE, Ismail K, Jetté N, Khamseh ME, Kwan Y, Lara MA, Liu SI, Loureiro SR, Löwe B, Marrie RA, Marsh L, McGuire A, Muramatsu K, Navarrete L, Osório FL, Petersen I, Picardi A, Pugh SL, Quinn TJ, Rooney AG, Shinn EH, Sidebottom A, Spangenberg L, Tan PLL, Taylor-Rowan M, MF A, van Weert HC, Vöhringer PA, Wagner LI, White J, Winkley K, Thombs BD. Patient Health Questionnaire-9 scores do not accurately estimate depression prevalence: individual participant data meta-analysis. J Clin Epidemiol. 2020 Jun;122(1):115–28. doi: 10.1016/j.jclinepi.2020.02.002. doi: 10.1016/j.jclinepi.2020.02.002.
    1. Falkenström F, Josefsson A, Berggren T, Holmqvist R. How much therapy is enough? Comparing dose-effect and good-enough models in two different settings. Psychotherapy (Chic) 2016 Mar;53(1):130–9. doi: 10.1037/pst0000039. doi: 10.1037/pst0000039.
    1. DiMatteo M, Lepper H, Croghan T W. Depression is a risk factor for noncompliance with medical treatment: meta-analysis of the effects of anxiety and depression on patient adherence. Arch Intern Med. 2000 Jul 24;160(14):2101–7. doi: 10.1001/archinte.160.14.2101.
    1. Fernández-Álvarez J, Díaz-García A, González-Robles A, Baños R, García-Palacios A, Botella C. Dropping out of a transdiagnostic online intervention: a qualitative analysis of client's experiences. Internet Interv. 2017 Dec;10:29–38. doi: 10.1016/j.invent.2017.09.001.

Source: PubMed

Sponsorer och medarbetare

Medicinska tillstånd

Läkemedelsinterventioner

3
Prenumerera