Compliance and Retention With the Experience Sampling Method Over the Continuum of Severe Mental Disorders: Meta-Analysis and Recommendations

Hugo Vachon, Wolfgang Viechtbauer, Aki Rintala, Inez Myin-Germeys, Hugo Vachon, Wolfgang Viechtbauer, Aki Rintala, Inez Myin-Germeys

Abstract

Background: Despite the growing interest in the experience sampling method (ESM) as a data collection tool for mental health research, the absence of methodological guidelines related to its use has resulted in a large heterogeneity of designs. Concomitantly, the potential effects of the design on the response behavior of the participants remain largely unknown.

Objective: The objective of this meta-analysis was to investigate the associations between various sample and design characteristics and the compliance and retention rates of studies using ESM in mental health research.

Methods: ESM studies investigating major depressive disorder, bipolar disorder, and psychotic disorder were considered for inclusion. Besides the compliance and retention rates, a number of sample and design characteristics of the selected studies were collected to assess their potential relationships with the compliance and retention rates. Multilevel random/mixed effects models were used for the analyses.

Results: Compliance and retention rates were lower for studies with a higher proportion of male participants (P<.001) and individuals with a psychotic disorder (P<.001). Compliance was positively associated with the use of a fixed sampling scheme (P=.02), higher incentives (P=.03), higher time intervals between successive evaluations (P=.02), and fewer evaluations per day (P=.008), while no significant associations were observed with regard to the mean age of the sample, the study duration, or other design characteristics.

Conclusions: The findings demonstrate that ESM studies can be carried out in mental health research, but the quality of the data collection depends upon a number of factors related to the design of ESM studies and the samples under study that need to be considered when designing such protocols.

Keywords: compliance; ecological momentary assessment; experience sampling; retention; severe mental disorders.

Conflict of interest statement

Conflicts of Interest: None declared.

©Hugo Vachon, Wolfgang Viechtbauer, Aki Rintala, Inez Myin-Germeys. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 06.12.2019.

Figures

Figure 1
Figure 1
Equations.
Figure 2
Figure 2
Flow chart of study inclusion protocol.
Figure 3
Figure 3
Funnel plots for compliance and retention.
Figure 4
Figure 4
Graphical representation of the relationship between the compliance of experience sampling method studies and the frequency of daily self-evaluations.

References

    1. Larson R, Csikszentmihalyi M. The experience sampling method. New Dir Methodology Soc Behav Sci. 1983:41–56.
    1. Stone AA, Shiffman S. Capturing momentary, self-report data: a proposal for reporting guidelines. Ann Behav Med. 2002;24(3):236–43. doi: 10.1207/S15324796ABM2403_09.
    1. Levin RL, Heller W, Mohanty A, Herrington JD, Miller GA. Cognitive deficits in depression and functional specificity of regional brain activity. Cogn Ther Res. 2007 Apr 6;31(2):211–33. doi: 10.1007/s10608-007-9128-z.
    1. Ohmuro N, Matsumoto K, Katsura M, Obara C, Kikuchi T, Hamaie Y, Sakuma A, Iizuka K, Ito F, Matsuoka H. The association between cognitive deficits and depressive symptoms in at-risk mental state: a comparison with first-episode psychosis. Schizophr Res. 2015 Mar;162(1-3):67–73. doi: 10.1016/j.schres.2015.01.008.
    1. Myin-Germeys I, Delespaul PA, deVries MW. Schizophrenia patients are more emotionally active than is assumed based on their behavior. Schizophr Bull. 2000;26(4):847–54. doi: 10.1093/oxfordjournals.schbul.a033499.
    1. Thompson RJ, Mata J, Jaeggi SM, Buschkuehl M, Jonides J, Gotlib IH. The everyday emotional experience of adults with major depressive disorder: examining emotional instability, inertia, and reactivity. J Abnorm Psychol. 2012 Nov;121(4):819–29. doi: 10.1037/a0027978.
    1. aan het Rot M, Hogenelst K, Schoevers RA. Mood disorders in everyday life: a systematic review of experience sampling and ecological momentary assessment studies. Clin Psychol Rev. 2012 Aug;32(6):510–23. doi: 10.1016/j.cpr.2012.05.007.
    1. Myin-Germeys I, Oorschot M, Collip D, Lataster J, Delespaul P, van Os J. Experience sampling research in psychopathology: opening the black box of daily life. Psychol Med. 2009 Sep;39(9):1533–47. doi: 10.1017/S0033291708004947.
    1. Colombo D, Palacios AG, Alvarez JF, Patané A, Semonella M, Cipresso P, Kwiatkowska M, Riva G, Botella C. Current state and future directions of technology-based ecological momentary assessments and interventions for major depressive disorder: protocol for a systematic review. Syst Rev. 2018 Dec 13;7(1):233. doi: 10.1186/s13643-018-0899-y.
    1. Myin-Germeys I, Kasanova Z, Vaessen T, Vachon H, Kirtley O, Viechtbauer W, Reininghaus U. Experience sampling methodology in mental health research: new insights and technical developments. World Psychiatry. 2018 Jun;17(2):123–32. doi: 10.1002/wps.20513. doi: 10.1002/wps.20513.
    1. Delespaul P. Assessing Schizophrenia in Daily Life: The Experience Sampling Method. Maastricht: Datawyse / Universitaire Pers Maastricht; 1995.
    1. Fuller-Tyszkiewicz M, Skouteris H, Richardson B, Blore J, Holmes M, Mills J. Does the burden of the experience sampling method undermine data quality in state body image research? Body Image. 2013 Sep;10(4):607–13. doi: 10.1016/j.bodyim.2013.06.003.
    1. ALLIGER GM, WILLIAMS KJ. Using signal-contingent experience sampling methodology to study work in the field: a discussion and illustration examining task perceptions and mood. Pers Psychol. 1993 Sep;46(3):525–49. doi: 10.1111/j.1744-6570.1993.tb00883.x.
    1. Messiah A, Grondin O, Encrenaz G. Factors associated with missing data in an experience sampling investigation of substance use determinants. Drug Alcohol Depend. 2011 Apr 01;114(2-3):153–8. doi: 10.1016/j.drugalcdep.2010.09.016.
    1. Sokolovsky AW, Mermelstein RJ, Hedeker D. Factors predicting compliance to ecological momentary assessment among adolescent smokers. Nicotine Tob Res. 2014 Mar;16(3):351–8. doi: 10.1093/ntr/ntt154.
    1. Morren M, van Dulmen S, Ouwerkerk J, Bensing J. Compliance with momentary pain measurement using electronic diaries: a systematic review. Eur J Pain. 2009 Apr;13(4):354–65. doi: 10.1016/j.ejpain.2008.05.010.
    1. Stone AA, Broderick JE, Schwartz JE, Shiffman S, Litcher-Kelly L, Calvanese P. Intensive momentary reporting of pain with an electronic diary: reactivity, compliance, and patient satisfaction. Pain. 2003 Jul;104(1-2):343–51. doi: 10.1016/s0304-3959(03)00040-x.
    1. Bentall RP, Myin-Germeys I, Smith A, Knowles R, Jones SH, Smith T, Tai SJ. Hypomanic personality, stability of self-esteem and response styles to negative mood. Clin Psychol Psychother. 2011;18(5):397–410. doi: 10.1002/cpp.780.
    1. Kuppens P, Oravecz Z, Tuerlinckx F. Feelings change: accounting for individual differences in the temporal dynamics of affect. J Pers Soc Psychol. 2010 Dec;99(6):1042–60. doi: 10.1037/a0020962.
    1. van Roekel E, Bennik EC, Bastiaansen JA, Verhagen M, Ormel J, Engels RC, Oldehinkel AJ. Depressive symptoms and the experience of pleasure in daily life: an exploration of associations in early and late adolescence. J Abnorm Child Psychol. 2016 Dec;44(5):999–1009. doi: 10.1007/s10802-015-0090-z.
    1. Ruscio AM, Gentes EL, Jones JD, Hallion LS, Coleman ES, Swendsen J. Rumination predicts heightened responding to stressful life events in major depressive disorder and generalized anxiety disorder. J Abnorm Psychol. 2015 Feb;124(1):17–26. doi: 10.1037/abn0000025.
    1. Audrain-McGovern J, Wileyto EP, Ashare R, Cuevas J, Strasser AA. Reward and affective regulation in depression-prone smokers. Biol Psychiatry. 2014 Nov 01;76(9):689–97. doi: 10.1016/j.biopsych.2014.04.018.
    1. Aldinger M, Stopsack M, Ulrich I, Appel K, Reinelt E, Wolff S, Grabe HJ, Lang S, Barnow S. Neuroticism developmental courses--implications for depression, anxiety and everyday emotional experience; a prospective study from adolescence to young adulthood. BMC Psychiatry. 2014 Aug 06;14:210. doi: 10.1186/s12888-014-0210-2.
    1. Vachon H, Bourbousson M, Deschamps T, Doron J, Bulteau S, Sauvaget A, Thomas-Ollivier V. Repeated self-evaluations may involve familiarization: an exploratory study related to ecological momentary assessment designs in patients with major depressive disorder. Psychiatry Res. 2016 Nov 30;245:99–104. doi: 10.1016/j.psychres.2016.08.034.
    1. Barge-Schaapveld DQ, Nicolson NA. Effects of antidepressant treatment on the quality of daily life: an experience sampling study. J Clin Psychiatry. 2002 Jun;63(6):477–85.
    1. Demiralp E, Thompson RJ, Mata J, Jaeggi SM, Buschkuehl M, Barrett LF, Ellsworth PC, Demiralp M, Hernandez-Garcia L, Deldin PJ, Gotlib IH, Jonides J. Feeling blue or turquoise? Emotional differentiation in major depressive disorder. Psychol Sci. 2012;23(11):1410–6. doi: 10.1177/0956797612444903.
    1. Myin-Germeys I, Marcelis M, Krabbendam L, Delespaul P, van Os J. Subtle fluctuations in psychotic phenomena as functional states of abnormal dopamine reactivity in individuals at risk. Biol Psychiatry. 2005 Jul 15;58(2):105–10. doi: 10.1016/j.biopsych.2005.02.012.
    1. Blum LH, Vakhrusheva J, Saperstein A, Khan S, Chang RW, Hansen MC, Zemon V, Kimhy D. Depressed mood in individuals with schizophrenia: a comparison of retrospective and real-time measures. Psychiatry Res. 2015 Jun 30;227(2-3):318–23. doi: 10.1016/j.psychres.2015.03.008.
    1. Havermans R, Nicolson NA, Berkhof J, deVries MW. Patterns of salivary cortisol secretion and responses to daily events in patients with remitted bipolar disorder. Psychoneuroendocrinology. 2011 Feb;36(2):258–65. doi: 10.1016/j.psyneuen.2010.07.016.
    1. Sagar KA, Dahlgren MK, Racine MT, Dreman MW, Olson DP, Gruber SA. Joint effects: a pilot investigation of the impact of bipolar disorder and marijuana use on cognitive function and mood. PLoS One. 2016;11(6):e0157060. doi: 10.1371/journal.pone.0157060.
    1. Janssens KA, Bos EH, Rosmalen JG, Wichers MC, Riese H. A qualitative approach to guide choices for designing a diary study. BMC Med Res Methodol. 2018 Nov 16;18(1):140. doi: 10.1186/s12874-018-0579-6.
    1. Rintala A, Wampers M, Myin-Germeys I, Viechtbauer W. Response compliance and predictors thereof in studies using the experience sampling method. Psychol Assess. 2019 Feb;31(2):226–35. doi: 10.1037/pas0000662.
    1. Jones A, Remmerswaal D, Verveer I, Robinson E, Franken IH, Wen CK, Field M. Compliance with ecological momentary assessment protocols in substance users: a meta-analysis. Addiction. 2019 Apr;114(4):609–19. doi: 10.1111/add.14503.
    1. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA, PRISMA-P Group Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015 Jan 01;4:1. doi: 10.1186/2046-4053-4-1.
    1. Vachon H, Rintala A, Viechtbauer W, Myin-Germeys I. Data quality and feasibility of the Experience Sampling Method across the spectrum of severe psychiatric disorders: a protocol for a systematic review and meta-analysis. Syst Rev. 2018 Jan 18;7(1):7. doi: 10.1186/s13643-018-0673-1.
    1. Nakagawa S, Poulin R, Mengersen K, Reinhold K, Engqvist L, Lagisz M, Senior AM. Meta-analysis of variation: ecological and evolutionary applications and beyond. Methods Ecol Evol. 2014 Dec 17;6(2):143–152. doi: 10.1111/2041-210X.12309.
    1. Rücker G, Schwarzer G, Carpenter J, Olkin I. Why add anything to nothing? The arcsine difference as a measure of treatment effect in meta-analysis with zero cells. Stat Med. 2009 Feb 28;28(5):721–38. doi: 10.1002/sim.3511.
    1. Konstantopoulos S. Fixed effects and variance components estimation in three-level meta-analysis. Res Synth Methods. 2011 Mar;2(1):61–76. doi: 10.1002/jrsm.35.
    1. Viechtbauer W. Conducting Meta-Analyses in R with the metafor Package. J Stat Soft. 2010;36(3):1–48. doi: 10.18637/jss.v036.i03.
    1. Raudenbush S. Random effects models. In: Cooper L, Hedges LV, editors. The Handbook of Research Synthesis. New York: Russell Sage Foundation; 1994. pp. 301–21.
    1. Cochran WG. Some methods for strengthening the common χ2 tests. Biometrics. 1954 Dec;10(4):417. doi: 10.2307/3001616.
    1. Viechtbauer W, Cheung MW. Outlier and influence diagnostics for meta-analysis. Res Synth Methods. 2010 Apr;1(2):112–25. doi: 10.1002/jrsm.11.
    1. Kuepper R, Oorschot M, Myin-Germeys I, Smits M, van Os J, Henquet C. Is psychotic disorder associated with increased levels of craving for cannabis? An Experience Sampling study. Acta Psychiatr Scand. 2013 Dec;128(6):448–56. doi: 10.1111/acps.12078.
    1. Ramsey AT, Wetherell JL, Depp C, Dixon D, Lenze E. Feasibility and acceptability of smartphone assessment in older adults with cognitive and emotional difficulties. J Technol Hum Serv. 2016;34(2):209–223. doi: 10.1080/15228835.2016.1170649.
    1. Wichers M, Peeters F, Geschwind N, Jacobs N, Simons CJ, Derom C, Thiery E, Delespaul PH, van Os J. Unveiling patterns of affective responses in daily life may improve outcome prediction in depression: a momentary assessment study. J Affect Disord. 2010 Jul;124(1-2):191–5. doi: 10.1016/j.jad.2009.11.010.
    1. Lee-Flynn SC, Pomaki G, Delongis A, Biesanz JC, Puterman E. Daily cognitive appraisals, daily affect, and long-term depressive symptoms: the role of self-esteem and self-concept clarity in the stress process. Pers Soc Psychol Bull. 2011 Feb;37(2):255–68. doi: 10.1177/0146167210394204.
    1. Silvia PJ, Kwapil TR, Eddington KM, Brown LH. Missed beeps and missing data: dispositional and situational predictors of nonresponse in experience sampling research. Soc Sci Comput Rev. 2013 Mar 13;31(4):471–81. doi: 10.1177/0894439313479902.
    1. Johnson EI, Grondin O, Barrault M, Faytout M, Helbig S, Husky M, Granholm EL, Loh C, Nadeau L, Wittchen H, Swendsen J. Computerized ambulatory monitoring in psychiatry: a multi-site collaborative study of acceptability, compliance, and reactivity. Int J Methods Psychiatr Res. 2009;18(1):48–57. doi: 10.1002/mpr.276.
    1. Fleming SK, Blasey C, Schatzberg AF. Neuropsychological correlates of psychotic features in major depressive disorders: a review and meta-analysis. J Psychiatr Res. 2004 Jan;38(1):27–35. doi: 10.1016/s0022-3956(03)00100-6.
    1. Bora E, Yucel M, Pantelis C. Theory of mind impairment in schizophrenia: meta-analysis. Schizophr Res. 2009 Apr;109(1-3):1–9. doi: 10.1016/j.schres.2008.12.020.
    1. Torous J, Friedman R, Keshavan M. Smartphone ownership and interest in mobile applications to monitor symptoms of mental health conditions. JMIR Mhealth Uhealth. 2014 Jan 21;2(1):e2. doi: 10.2196/mhealth.2994.
    1. Granholm E, Loh C, Swendsen J. Feasibility and validity of computerized ecological momentary assessment in schizophrenia. Schizophr Bull. 2008 May;34(3):507–14. doi: 10.1093/schbul/sbm113.
    1. Borsboom D, Cramer AO. Network analysis: an integrative approach to the structure of psychopathology. Annu Rev Clin Psychol. 2013;9:91–121. doi: 10.1146/annurev-clinpsy-050212-185608.
    1. Husky MM, Gindre C, Mazure CM, Brebant C, Nolen-Hoeksema S, Sanacora G, Swendsen J. Computerized ambulatory monitoring in mood disorders: feasibility, compliance, and reactivity. Psychiatry Res. 2010 Jul 30;178(2):440–2. doi: 10.1016/j.psychres.2010.04.045.
    1. Piasecki TM, Hufford MR, Solhan M, Trull TJ. Assessing clients in their natural environments with electronic diaries: rationale, benefits, limitations, and barriers. Psychol Assess. 2007 Mar;19(1):25–43. doi: 10.1037/1040-3590.19.1.25.
    1. Mata J, Thompson RJ, Jaeggi SM, Buschkuehl M, Jonides J, Gotlib IH. Walk on the bright side: physical activity and affect in major depressive disorder. J Abnorm Psychol. 2012 May;121(2):297–308. doi: 10.1037/a0023533.
    1. So SH, Peters ER, Swendsen J, Garety PA, Kapur S. Detecting improvements in acute psychotic symptoms using experience sampling methodology. Psychiatry Res. 2013 Nov 30;210(1):82–8. doi: 10.1016/j.psychres.2013.05.010.
    1. Husen K, Rafaeli E, Rubel JA, Bar-Kalifa E, Lutz W. Daily affect dynamics predict early response in CBT: feasibility and predictive validity of EMA for outpatient psychotherapy. J Affect Disord. 2016 Dec;206:305–314. doi: 10.1016/j.jad.2016.08.025.
    1. Husky M, Olié E, Guillaume S, Genty C, Swendsen J, Courtet P. Feasibility and validity of ecological momentary assessment in the investigation of suicide risk. Psychiatry Res. 2014 Dec 15;220(1-2):564–70. doi: 10.1016/j.psychres.2014.08.019.

Source: PubMed

Patrocinadores y Colaboradores

Condiciones médicas

Intervenciones de drogas

3
Suscribir