Measuring the Implementation of Behavioral Intervention Technologies: Recharacterization of Established Outcomes

Eric DA Hermes, Aaron R Lyon, Stephen M Schueller, Joseph E Glass, Eric DA Hermes, Aaron R Lyon, Stephen M Schueller, Joseph E Glass

Abstract

Behavioral intervention technologies (BITs) are websites, software, mobile apps, and sensors designed to help users address or change behaviors, cognitions, and emotional states. BITs have the potential to transform health care delivery, and early research has produced promising findings of efficacy. BITs also favor new models of health care delivery and provide novel data sources for measurement. However, there are few examples of successful BIT implementation and a lack of consensus on as well as inadequate descriptions of BIT implementation measurement. The aim of this viewpoint paper is to provide an overview and characterization of implementation outcomes for the study of BIT use in routine practice settings. Eight outcomes for the evaluation of implementation have been previously described: acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability. In a proposed recharacterization of these outcomes with respect to BIT implementation, definitions are clarified, expansions to the level of analysis are identified, and unique measurement characteristics are discussed. Differences between BIT development and implementation, an increased focus on consumer-level outcomes, the expansion of providers who support BIT use, and the blending of BITs with traditional health care services are specifically discussed. BITs have the potential to transform health care delivery. Realizing this potential, however, will hinge on high-quality research that consistently and accurately measures how well such technologies have been integrated into health services. This overview and characterization of implementation outcomes support BIT research by identifying and proposing solutions for key theoretical and practical measurement challenges.

Keywords: behavior therapy; behavioral intervention technology; diffusion of innovation; implementation; internet; mobile applications; outcome assessment (health care); review; technology; telemedicine; translational medical research.

Conflict of interest statement

Conflicts of Interest: None declared.

©Eric DA Hermes, Aaron R Lyon, Stephen M Schueller, Joseph E Glass. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 25.01.2019.

Figures

Figure 1
Figure 1
Continuum of support for delivering behavioral intervention technologies (BITs).

References

    1. Fortney JC, Burgess JF, Bosworth HB, Booth BM, Kaboli PJ. A re-conceptualization of access for 21st century healthcare. J Gen Intern Med. 2011 Nov;26(Suppl 2):639–47. doi: 10.1007/s11606-011-1806-6.
    1. Marsch LA, Lord SE, Dallery J, editors. Behavioral healthcare and technology: using science-based innovations to transform practice. Oxford, UK: Oxford University Press; 2014. p. 368.
    1. England MJ, Butler AS, Gonzalez ML, editors. Psychosocial interventions for mental and substance use disorders: A framework for establishing evidence-based standards. Washington, DC: National Academy Press; 2015. p. 209.
    1. Fleming T, Bavin L, Lucassen M, Stasiak K, Hopkins S, Merry S. Beyond the trial: systematic review of real-world uptake and engagement with digital self-help interventions for depression, low mood, or anxiety. J Med Internet Res. 2018 Jun 6;20(6):e199. doi: 10.2196/jmir.9275. v20i6e199
    1. Sander L, Rausch L, Baumeister H. Effectiveness of internet-based interventions for the prevention of mental disorders: a systematic review and meta-analysis. J Med Internet Res Ment Health. 2016;3(3):e38. doi: 10.2196/mental.6061. v3i3e38
    1. Quinn CC, Clough SS, Minor JM, Lender D, Okafor MC, Gruber-Baldini A. WellDoc™ mobile diabetes management randomized controlled trial: change in clinical and behavioral outcomes and patient and physician satisfaction. Diabetes Technol Ther. 2008 May 12;10(3):160–8. doi: 10.1089/dia.2008.0283. doi: 10.1089/dia.2008.0283.
    1. Calear AL, Christensen H, Mackinnon A, Griffiths KM. Adherence to the MoodGYM program: outcomes and predictors for an adolescent school-based population. J Affect Disord. 2013 May;147(1-3):338–44. doi: 10.1016/j.jad.2012.11.036.S0165-0327(12)00803-8
    1. Schneider J, Sarrami Foroushani P, Grime P, Thornicroft G. Acceptability of online self-help to people with depression: users' views of MoodGYM versus informational websites. J Med Internet Res. 2014 Mar 28;16(3):e90. doi: 10.2196/jmir.2871. v16i3e90
    1. Christensen H, Batterham PJ, Gosling JA, Ritterband LM, Griffiths KM, Thorndike FP, Glozier N, O'Dea B, Hickie IB, Mackinnon AJ. Effectiveness of an online insomnia program (SHUTi) for prevention of depressive episodes (the GoodNight Study): a randomised controlled trial. Lancet Psychiatry. 2016 Apr;3(4):331–41. doi: 10.1016/S2215-0366(15)00536-2.S2215-0366(15)00536-2
    1. Ritterband LM, Thorndike FP, Gonder-Frederick LA, Magee JC, Bailey ET, Saylor DK, Morin CM. Efficacy of an internet-based behavioral intervention for adults with insomnia. Arch Gen Psychiatry. 2009 Jul;66(7):692–8. doi: 10.1001/archgenpsychiatry.2009.66. 66/7/692
    1. Ritterband LM, Thorndike FP, Ingersoll KS, Lord HR, Gonder-Frederick L, Frederick C, Quigg MS, Cohn WF, Morin CM. Effect of a web-based cognitive behavior therapy for insomnia intervention with 1-year follow-up: a randomized clinical trial. J Am Med Assoc Psychiatry. 2017 Jan 01;74(1):68–75. doi: 10.1001/jamapsychiatry.2016.3249.2589161
    1. Kuhn E, Greene C, Hoffman J, Nguyen T, Wald L, Schmidt J, Ramsey KM, Ruzek J. Preliminary evaluation of PTSD Coach, a smartphone app for post-traumatic stress symptoms. Mil Med. 2014 Jan;179(1):12–8. doi: 10.7205/MILMED-D-13-00271.
    1. Possemato K, Kuhn E, Johnson E, Hoffman JE, Owen JE, Kanuri N, De Stefano SL, Brooks E. Using PTSD Coach in primary care with and without clinician support: a pilot randomized controlled trial. Gen Hosp Psychiatry. 2016;38:94–8. doi: 10.1016/j.genhosppsych.2015.09.005.S0163-8343(15)00223-6
    1. Reger GM, Hoffman J, Riggs D, Rothbaum BO, Ruzek J, Holloway KM, Kuhn E. The "PE coach" smartphone application: an innovative approach to improving implementation, fidelity, and homework adherence during prolonged exposure. Psychol Serv. 2013 Aug;10(3):342–9. doi: 10.1037/a0032774.2013-28609-002
    1. Campbell AN, Nunes EV, Matthews AG, Stitzer M, Miele GM, Polsky D, Turrigiano E, Walters S, McClure EA, Kyle TL, Wahle A, Van Veldhuisen P, Goldman B, Babcock D, Stabile PQ, Winhusen T, Ghitza UE. Internet-delivered treatment for substance abuse: a multisite randomized controlled trial. Am J Psychiatry. 2014 Jun;171(6):683–90. doi: 10.1176/appi.ajp.2014.13081055. 1859474
    1. Warmerdam L, Smit F, van Straten A, Riper H, Cuijpers P. Cost-utility and cost-effectiveness of internet-based treatment for adults with depressive symptoms: randomized trial. J Med Internet Res. 2010;12(5):e53. doi: 10.2196/jmir.1436. v12i5e53
    1. Murphy SM, Campbell AN, Ghitza UE, Kyle TL, Bailey GL, Nunes EV, Polsky D. Cost-effectiveness of an internet-delivered treatment for substance abuse: data from a multisite randomized controlled trial. Drug Alcohol Depend. 2016 Apr 1;161:119–26. doi: 10.1016/j.drugalcdep.2016.01.021. S0376-8716(16)00049-1
    1. Syrowatka A, Krömker D, Meguerditchian AN, Tamblyn R. Features of computer-based decision aids: systematic review, thematic synthesis, and meta-analyses. J Med Internet Res. 2016 Jan 26;18(1):e20. doi: 10.2196/jmir.4982. v18i1e20
    1. Sheikh A, Sood HS, Bates DW. Leveraging health information technology to achieve the "triple aim" of healthcare reform. J Am Med Inform Assoc. 2015 Jul;22(4):849–56. doi: 10.1093/jamia/ocv022. ocv022
    1. Rogers MA, Lemmen K, Kramer R, Mann J, Chopra V. Internet-delivered health interventions that work: systematic review of meta-analyses and evaluation of website availability. J Med Internet Res. 2017 Mar 24;19(3):e90. doi: 10.2196/jmir.7111. v19i3e90
    1. Gilbody S, Brabyn S, Lovell K, Kessler D, Devlin T, Smith L, Araya R, Barkham M, Bower P, Cooper C, Knowles S, Littlewood E, Richards DA, Tallon D, White D, Worthy G, REEACT collaborative Telephone-supported computerised cognitive-behavioural therapy: REEACT-2 large-scale pragmatic randomised controlled trial. Br J Psychiatry. 2017 May;210(5):362–7. doi: 10.1192/bjp.bp.116.192435.bjp.bp.116.192435
    1. Gilbody S, Littlewood E, Hewitt C, Brierley G, Tharmanathan P, Araya R, Barkham M, Bower P, Cooper C, Gask L, Kessler D, Lester H, Lovell K, Parry G, Richards DA, Andersen P, Brabyn S, Knowles S, Shepherd C, Tallon D, White D. Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial): large scale pragmatic randomised controlled trial. Br Med J. 2016 Jan 12;352:i195.
    1. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011 Mar;38(2):65–76. doi: 10.1007/s10488-010-0319-7.
    1. Glasgow RE, Dickinson P, Fisher L, Christiansen S, Toobert DJ, Bender BG, Dickinson LM, Jortberg B, Estabrooks PA. Use of RE-AIM to develop a multi-media facilitation tool for the patient-centered medical home. Implement Sci. 2011 Oct 21;6:118. doi: 10.1186/1748-5908-6-118. 1748-5908-6-118
    1. Glasgow RE, Phillips SM, Sanchez MA. Implementation science approaches for integrating eHealth research into practice and policy. Int J Med Inform. 2014 Jul;83(7):e1–11. doi: 10.1016/j.ijmedinf.2013.07.002.S1386-5056(13)00157-3
    1. Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res. 2009 Apr 24;11(2):e13. doi: 10.2196/jmir.1194. v11i2e13
    1. Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A systematic review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res. 2011;13(3):e52. doi: 10.2196/jmir.1772. v13i3e52
    1. Wozney L, McGrath PJ, Gehring ND, Bennett K, Huguet A, Hartling L, Dyson MP, Soleimani A, Newton AS. eMental healthcare technologies for anxiety and depression in childhood and adolescence: systematic review of studies reporting implementation outcomes. J Med Internet Res Ment Health. 2018 Jun 26;5(2):e48. doi: 10.2196/mental.9655. v5i2e48
    1. Baumeister H, Reichler L, Munzinger M, Lin J. The impact of guidance on internet-based mental health interventions? A systematic review. Internet Interv. 2014 Jan;1(4):205–15. doi: 10.1016/j.invent.2014.08.003. doi: 10.1016/j.invent.2014.08.003.
    1. Kuhn E, Weiss BJ, Taylor KL, Hoffman JE, Ramsey KM, Manber R, Gehrman P, Crowley JJ, Ruzek JI, Trockel M. CBT-I coach: a description and clinician perceptions of a mobile app for cognitive behavioral therapy for insomnia. J Clin Sleep Med. 2016;12(4):597–606. doi: 10.5664/jcsm.5700.jc-00087-15
    1. Muñoz RF. The efficiency model of support and the creation of digital apothecaries. Clin Psychol Sci Pract. 2016 Nov 17;24(1):46–9. doi: 10.1111/cpsp.12174. doi: 10.1111/cpsp.12174.
    1. International Organization for Standardization. 2010. Mar, [2018-12-10]. Ergonomics of human-system interaction-part 210: Human- centered design for interactive systems .
    1. Mares ML, Gustafson DH, Glass JE, Quanbeck A, McDowell H, McTavish F, Atwood AK, Marsch LA, Thomas C, Shah D, Brown R, Isham A, Nealon MJ, Ward V. Implementing an mHealth system for substance use disorders in primary care: a mixed methods study of clinicians' initial expectations and first year experiences. BMC Med Inform Decis Mak. 2016 Sep 29;16(1):126. doi: 10.1186/s12911-016-0365-5. 10.1186/s12911-016-0365-5
    1. Milward J, Deluca P, Drummond C, Watson R, Dunne J, Kimergård A. Usability testing of the BRANCH smartphone app designed to reduce harmful drinking in young adults. J Med Internet Res Mhealth Uhealth. 2017 Aug 8;5(8):e109. doi: 10.2196/mhealth.7836. v5i8e109
    1. Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: toward a unified view. MIS Q. 2003 Sep;27(3):425–78. doi: 10.2307/30036540.
    1. Lyon AR, Ludwig K, Wasse JK, Bergstrom A, Hendrix E, McCauley E. Determinants and functions of standardized assessment use among school mental health clinicians: a mixed methods evaluation. Adm Policy Ment Health. 2016 Jan;43(1):122–34. doi: 10.1007/s10488-015-0626-0. 10.1007/s10488-015-0626-0
    1. Kumar D, Tully LM, Iosif AM, Zakskorn LN, Nye KE, Zia A, Niendam TA. A mobile health platform for clinical monitoring in early psychosis: implementation in community-based outpatient early psychosis care. J Med Internet Res Ment Health. 2018 Feb 27;5(1):e15. doi: 10.2196/mental.8551. v5i1e15
    1. Sineath A, Lambert L, Verga C, Wagstaff M, Wingo BC. Monitoring intervention fidelity of a lifestyle behavioral intervention delivered through telehealth. Mhealth. 2017;3:35. doi: 10.21037/mhealth.2017.07.04. doi: 10.21037/-03-2017.07.04
    1. Quanbeck A, Gustafson DH, Marsch LA, Chih MY, Kornfield R, McTavish F, Johnson R, Brown RT, Mares ML, Shah DV. Implementing a mobile health system to integrate the treatment of addiction into primary care: a hybrid implementation-effectiveness study. J Med Internet Res. 2018 Jan 30;20(1):e37. doi: 10.2196/jmir.8928. v20i1e37
    1. Titov N, Dear BF, Staples LG, Bennett-Levy J, Klein B, Rapee RM, Shann C, Richards D, Andersson G, Ritterband L, Purtell C, Bezuidenhout G, Johnston L, Nielssen OB. MindSpot clinic: an accessible, efficient, and effective online treatment service for anxiety and depression. Psychiatr Serv. 2015 Oct;66(10):1043–50. doi: 10.1176/appi.ps.201400477.
    1. Brooke J. Usability evaluation in industry. Boca Raton: CRC Press; 1996. Jun, SUS-A quick and dirty usability scale.
    1. Schröder J, Sautier L, Kriston L, Berger T, Meyer B, Späth C, Köther U, Nestoriuc Y, Klein JP, Moritz S. Development of a questionnaire measuring attitudes towards psychological online interventions-the APOI. J Affect Disord. 2015 Nov 15;187:136–41. doi: 10.1016/j.jad.2015.08.044.S0165-0327(15)30148-8
    1. Schröder J, Berger T, Meyer B, Lutz W, Hautzinger M, Späth C, Eichenberg C, Klein JP, Moritz S. Attitudes towards internet interventions among psychotherapists and individuals with mild to moderate depression symptoms. Cogn Ther Res. 2017 Oct;41(5):745–56. doi: 10.1007/s10608-017-9850-0.
    1. Grudin J, Pruitt J. Personas, participatory design and product development: An Infrastructure for Engagement. Proceedings of Participation and Design Conference (PDC2002); Participation and Design Conference (PDC2002); June 23-25, 2002; Sweden. 2002. pp. 144–61.
    1. Lyon AR, Wasse JK, Ludwig K, Zachry M, Bruns EJ, Unützer J, McCauley E. The contextualized technology adaptation process (CTAP): optimizing health information technology to improve mental health systems. Adm Policy Ment Health. 2016 May;43(3):394–409. doi: 10.1007/s10488-015-0637-x.10.1007/s10488-015-0637-x
    1. Zheng K, Haftel HM, Hirschl RB, O'Reilly M, Hanauer DA. Quantifying the impact of health IT implementations on clinical workflow: a new methodological perspective. J Am Med Inform Assoc. 2010;17(4):454–61. doi: 10.1136/jamia.2010.004440. 17/4/454
    1. Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, Boynton MH, Halko H. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017 Dec 29;12(1):108. doi: 10.1186/s13012-017-0635-3. 10.1186/s13012-017-0635-3
    1. Rubin J, Chisnell D, Spool J. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. New York, NY: Wiley, John & Sons; 2008. p. 384.
    1. Zhou X, Qiu Y, Hua G, Wang H, Ruan X. A feasible approach to the integration of CAD and CAPP. Comput Aided Des. 2007 Apr;39(4):324–38. doi: 10.1016/j.cad.2007.01.005. doi: 10.1016/j.cad.2007.01.005.
    1. Lappalainen P, Kaipainen K, Lappalainen R, Hoffrén H, Myllymäki T, Kinnunen ML, Mattila E, Happonen AP, Rusko H, Korhonen I. Feasibility of a personal health technology-based psychological intervention for men with stress and mood problems: randomized controlled pilot trial. J Med Inernet Res Res Protoc. 2013 Jan 9;2(1):e1. doi: 10.2196/resprot.2389. v2i1e1
    1. Morrison C, Doherty G. Analyzing engagement in a web-based intervention platform through visualizing log-data. J Med Internet Res. 2014;16(11):e252. doi: 10.2196/jmir.3575. v16i11e252
    1. Christensen H, Griffiths K, Groves C, Korten A. Free range users and one hit wonders: community users of an internet-based cognitive behaviour therapy program. Aust N Z J Psychiatry. 2006 Jan;40(1):59–62. doi: 10.1111/j.1440-1614.2006.01743.x.ANP1743
    1. Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M. Behavioral intervention technologies: evidence review and recommendations for future research in mental health. Gen Hosp Psychiatry. 2013 Aug;35(4):332–8. doi: 10.1016/j.genhosppsych.2013.03.008. S0163-8343(13)00069-8
    1. Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, Glass JE, York JL. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012 Apr;69(2):123–57. doi: 10.1177/1077558711430690. 1077558711430690
    1. Muñoz RF, Bunge EL, Chen K, Schueller SM, Bravin JI, Shaughnessy EA, Pérez-Stable EJ. Massive open online interventions: a novel model for delivering behavioral-health services worldwide. Clin Psychol Sci. 2016;4(2):194–205. doi: 10.1177/2167702615583840. doi: 10.1177/2167702615583840.
    1. Mohr DC, Cheung K, Schueller SM, Hendricks Brown C, Duan N. Continuous evaluation of evolving behavioral intervention technologies. Am J Prev Med. 2013 Oct;45(4):517–23. doi: 10.1016/j.amepre.2013.06.006. S0749-3797(13)00387-5
    1. Tang PC, Smith MD. Democratization of health care. J Am Me Assoc. 2016 Oct 25;316(16):1663–4. doi: 10.1001/jama.2016.14060.2556028
    1. Chamberlain P, Brown CH, Saldana L. Observational measure of implementation progress in community based settings: the stages of implementation completion (SIC) Implement Sci. 2011 Oct 6;6:116. doi: 10.1186/1748-5908-6-116. 1748-5908-6-116
    1. Hobfoll SE, Blais RK, Stevens NR, Walt L, Gengler R. Vets prevail online intervention reduces PTSD and depression in veterans with mild-to-moderate symptoms. J Consult Clin Psychol. 2016 Jan;84(1):31–42. doi: 10.1037/ccp0000041.2015-39661-001
    1. Lyon. Bruns E. User-centered redesign of evidence-based psychosocial interventions to enhance implementation-hospitable soil or better seeds? JAMA Psychiatry. 2018 Nov 14;:-. doi: 10.1001/jamapsychiatry.2018.3060. Epub ahead of print(forthcoming)2712978
    1. Lewis CC, Fischer S, Weiner BJ, Stanick C, Kim M, Martinez RG. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. 2015 Nov 4;10(1):155. doi: 10.1186/s13012-015-0342-x. 10.1186/s13012-015-0342-x
    1. Donker T, Petrie K, Proudfoot J, Clarke J, Birch MR, Christensen H. Smartphones for smarter delivery of mental health programs: a systematic review. J Med Internet Res. 2013 Nov 15;15(11):e247. doi: 10.2196/jmir.2791. v15i11e247
    1. Arnhold M, Quade M, Kirch W. Mobile applications for diabetics: a systematic review and expert-based usability evaluation considering the special requirements of diabetes patients age 50 years or older. J Med Internet Res. 2014;16(4):e104. doi: 10.2196/jmir.2968. v16i4e104
    1. Chung GH, Choi JN, Du J. Tired of innovations? Learned helplessness and fatigue in the context of continuous streams of innovation implementation. J Organ Behav. 2017 Sep;38(7):1130–48. doi: 10.1002/job.2191. doi: 10.1002/job.2191.
    1. Schueller SM, Tomasino KN, Mohr DC. Integrating human support into behavioral intervention technologies: the efficiency model of support. Clin Psychol. 2017;24(1):27–45. doi: 10.1111/cpsp.12173. doi: 10.1111/cpsp.12173.
    1. Katon W, Von Korff M, Lin E, Simon G, Walker E, Unützer J, Bush T, Russo J, Ludman E. Stepped collaborative care for primary care patients with persistent symptoms of depression: a randomized trial. Arch Gen Psychiatry. 1999 Dec;56(12):1109–15.
    1. Almirall D, Compton SN, Gunlicks-Stoessel M, Duan N, Murphy SA. Designing a pilot sequential multiple assignment randomized trial for developing an adaptive treatment strategy. Stat Med. 2012 Jul 30;31(17):1887–902. doi: 10.1002/sim.4512.
    1. Rozental A, Boettcher J, Andersson G, Schmidt B, Carlbring P. Negative effects of internet interventions: a qualitative content analysis of patients' experiences with treatments delivered online. Cogn Behav Ther. 2015;44(3):223–36. doi: 10.1080/16506073.2015.1008033.
    1. Mamede S, Schmidt HG. The twin traps of overtreatment and therapeutic nihilism in clinical practice. Med Educ. 2014 Jan;48(1):34–43. doi: 10.1111/medu.12264.
    1. Glass JE, McKay JR, Gustafson DH, Kornfield R, Rathouz PJ, McTavish FM, Atwood AK, Isham A, Quanbeck A, Shah D. Treatment seeking as a mechanism of change in a randomized controlled trial of a mobile health intervention to support recovery from alcohol use disorders. J Subst Abuse Treat. 2017 Jun;77:57–66. doi: 10.1016/j.jsat.2017.03.011.S0740-5472(16)30343-9

Source: PubMed

Sponsorit ja yhteistyökumppanit

Sairaudet

Huumeiden interventiot

3
Tilaa