Accelerating Digital Mental Health Research From Early Design and Creation to Successful Implementation and Sustainment

David C Mohr, Aaron R Lyon, Emily G Lattie, Madhu Reddy, Stephen M Schueller, David C Mohr, Aaron R Lyon, Emily G Lattie, Madhu Reddy, Stephen M Schueller

Abstract

Mental health problems are common and pose a tremendous societal burden in terms of cost, morbidity, quality of life, and mortality. The great majority of people experience barriers that prevent access to treatment, aggravated by a lack of mental health specialists. Digital mental health is potentially useful in meeting the treatment needs of large numbers of people. A growing number of efficacy trials have shown strong outcomes for digital mental health treatments. Yet despite their positive findings, there are very few examples of successful implementations and many failures. Although the research-to-practice gap is not unique to digital mental health, the inclusion of technology poses unique challenges. We outline some of the reasons for this gap and propose a collection of methods that can result in sustainable digital mental health interventions. These methods draw from human-computer interaction and implementation science and are integrated into an Accelerated Creation-to-Sustainment (ACTS) model. The ACTS model uses an iterative process that includes 2 basic functions (design and evaluate) across 3 general phases (Create, Trial, and Sustain). The ultimate goal in using the ACTS model is to produce a functioning technology-enabled service (TES) that is sustainable in a real-world treatment setting. We emphasize the importance of the service component because evidence from both research and practice has suggested that human touch is a critical ingredient in the most efficacious and used digital mental health treatments. The Create phase results in at least a minimally viable TES and an implementation blueprint. The Trial phase requires evaluation of both effectiveness and implementation while allowing optimization and continuous quality improvement of the TES and implementation plan. Finally, the Sustainment phase involves the withdrawal of research or donor support, while leaving a functioning, continuously improving TES in place. The ACTS model is a step toward bringing implementation and sustainment into the design and evaluation of TESs, public health into clinical research, research into clinics, and treatment into the lives of our patients.

Keywords: eHealth; mHealth; methodology.

Conflict of interest statement

Conflicts of Interest: Dr Mohr has received an honorarium from Optum Behavioral Health. The remaining authors declare they have no potential conflicts of interest.

©David C. Mohr, Aaron R Lyon, Emily G Lattie, Madhu Reddy, Stephen M Schueller. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 10.05.2017.

Figures

Figure 1
Figure 1
Aims for each development target in each phase.
Figure 2
Figure 2
Iterative evaluative and design functions at each phase.

References

    1. Kessler RC, Chiu WT, Demler O, Merikangas KR, Walters EE. Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005 Jun;62(6):617–627. doi: 10.1001/archpsyc.62.6.617.
    1. Hu T. Perspectives: an international review of the national cost estimates of mental illness, 1990-2003. J Ment Health Policy Econ. 2006 Mar;9(1):3–13.
    1. Mathers CD, Loncar D. Projections of global mortality and burden of disease from 2002 to 2030. PLoS Med. 2006 Nov;3(11):e442. doi: 10.1371/journal.pmed.0030442.
    1. Brody DS, Khaliq AA, Thompson TL. Patients' perspectives on the management of emotional distress in primary care settings. J Gen Intern Med. 1997 Jul;12(7):403–406.
    1. Dwight-Johnson M, Sherbourne CD, Liao D, Wells KB. Treatment preferences among depressed primary care patients. J Gen Intern Med. 2000 Aug;15(8):527–534.
    1. Bedi N, Chilvers C, Churchill R, Dewey M, Duggan C, Fielding K, Gretton V, Miller P, Harrison G, Lee A, Williams I. Assessing effectiveness of treatment of depression in primary care: partially randomised preference trial. Br J Psychiatry. 2000 Oct;177:312–318.
    1. Priest RG, Vize C, Roberts A, Roberts M, Tylee A. Lay people's attitudes to treatment of depression: results of opinion poll for Defeat Depression Campaign just before its launch. BMJ. 1996 Oct 5;313(7061):858–859.
    1. Churchill R, Khaira M, Gretton V, Chilvers C, Dewey M, Duggan C, Lee A, Nottingham Counselling and Antidepressants in Primary Care (CAPC) Study Group Treating depression in general practice: factors affecting patients' treatment preferences. Br J Gen Pract. 2000 Nov;50(460):905–906.
    1. Mohr DC, Ho J, Duffecy J, Baron KG, Lehman KA, Jin L, Reifler D. Perceived barriers to psychological treatments and their relationship to depression. J Clin Psychol. 2010 Apr;66(4):394–409. doi: 10.1002/jclp.20659.
    1. Mohr DC, Hart SL, Howard I, Julian L, Vella L, Catledge C, Feldman MD. Barriers to psychotherapy among depressed and nondepressed primary care patients. Ann Behav Med. 2006 Dec;32(3):254–258. doi: 10.1207/s15324796abm3203_12.
    1. Kazdin A, Blase S. Rebooting psychotherapy research and practice to reduce the burden of mental illness. Perspect Psychol Sci. 2011 Jan;6(1):21–37. doi: 10.1177/1745691610393527.
    1. Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M. Behavioral intervention technologies: evidence review and recommendations for future research in mental health. Gen Hosp Psychiatry. 2013 Aug;35(4):332–338. doi: 10.1016/j.genhosppsych.2013.03.008.
    1. Wagner B, Horn AB, Maercker A. Internet-based versus face-to-face cognitive-behavioral intervention for depression: a randomized controlled non-inferiority trial. J Affect Disord. 2014 Jan;152-154:113–121. doi: 10.1016/j.jad.2013.06.032.
    1. Andersson G, Cuijpers P, Carlbring P, Riper H, Hedman E. Guided Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta-analysis. World Psychiatry. 2014 Oct;13(3):288–295. doi: 10.1002/wps.20151. doi: 10.1002/wps.20151.
    1. Cuijpers P, Marks IM, van Straten A, Cavanagh K, Gega L, Andersson G. Computer-aided psychotherapy for anxiety disorders: a meta-analytic review. Cogn Behav Ther. 2009;38(2):66–82. doi: 10.1080/16506070802694776.
    1. Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev. 2012 Jun;32(4):329–342. doi: 10.1016/j.cpr.2012.02.004.
    1. Donker T, Petrie K, Proudfoot J, Clarke J, Birch M, Christensen H. Smartphones for smarter delivery of mental health programs: a systematic review. J Med Internet Res. 2013;15(11):e247. doi: 10.2196/jmir.2791.
    1. Kuester A, Niemeyer H, Knaevelsrud C. Internet-based interventions for posttraumatic stress: a meta-analysis of randomized controlled trials. Clin Psychol Rev. 2016 Feb;43:1–16. doi: 10.1016/j.cpr.2015.11.004.
    1. Gilbody S, Littlewood E, Hewitt C, Brierley G, Tharmanathan P, Araya R, Barkham M, Bower P, Cooper C, Gask L, Kessler D, Lester H, Lovell K, Parry G, Richards DA, Andersen P, Brabyn S, Knowles S, Shepherd C, Tallon D, White D. Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial): large scale pragmatic randomised controlled trial. BMJ. 2015;351:h5627.
    1. Mohr D, Weingardt K, Reddy M, Schueller S. Three problems with current digital mental health research—and three things we can do about it. Psychiatr Serv (in press) 2017:1. (forthcoming)
    1. Laing BY, Mangione CM, Tseng C, Leng M, Vaisberg E, Mahida M, Bholat M, Glazier E, Morisky DE, Bell DS. Effectiveness of a smartphone application for weight loss compared with usual care in overweight primary care patients: a randomized, controlled trial. Ann Intern Med. 2014 Nov 18;161(10 Suppl):S5–S12. doi: 10.7326/M13-3005.
    1. Jakicic JM, Davis KK, Rogers RJ, King WC, Marcus MD, Helsel D, Rickman AD, Wahed AS, Belle SH. Effect of wearable technology combined with a lifestyle intervention on long-term weight loss: the IDEA randomized clinical trial. JAMA. 2016 Sep 20;316(11):1161–1171. doi: 10.1001/jama.2016.12858.
    1. Balas E, Boren S. Managing clinical knowledge for healthcare improvements. In: van Bemmel JH, McCray AT, editors. Yearbook of Medical Informatics. Stuttgart: Schattauer; 2000. pp. 2000–2070.
    1. Onken LS, Carroll KM, Shoham V, Cuthbert BN, Riddle M. Reenvisioning clinical science: unifying the discipline to improve the public health. Clin Psychol Sci. 2014 Jan 1;2(1):22–34. doi: 10.1177/2167702613497932.
    1. Rounsaville B, Carroll KM, Onken LS. A stage model of behavioral therapies research: getting started and moving on from stage I. Clin Psychol Sci Pract. 2001;8(2):133–142. doi: 10.1093/clipsy.8.2.133.
    1. Weisz J, Jensen A, McLeod B, Hibbs E, Jensen P. Milestones and methods in the development and dissemination of child and adolescent psychotherapies: a new deployment-focused model. In: Hibbs ED, editor. Psychosocial Treatments for Child and Adolescent Disorders: Empirically Based Strategies for Clinical Practice, 2nd edition. Washington: American Psychological Association; 2005. pp. 9–39.
    1. Bennett GG, Glasgow RE. The delivery of public health interventions via the Internet: actualizing their potential. Annu Rev Public Health. 2009;30:273–292. doi: 10.1146/annurev.publhealth.031308.100235.
    1. Glasgow RE, Davidson KW, Dobkin PL, Ockene J, Spring B. Practical behavioral trials to advance evidence-based behavioral medicine. Ann Behav Med. 2006 Feb;31(1):5–13. doi: 10.1207/s15324796abm3101_3.
    1. Glasgow RE, Lichtenstein E, Marcus AC. Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003 Aug;93(8):1261–1267.
    1. Institute of Medicine . Envisioning a Transformed Clinical Trials Enterprise in the United States: Establishing an Agenda for 2020. Washington: National Academies Press; 2012.
    1. Patrick K, Hekler E, Estrin D, Mohr D, Riper H, Crane D. Rapid rate of technological development and its implications for research on digital health interventions. Am J Prev Med (in press) 2017:1.
    1. Mohr DC, Schueller SM, Riley WT, Brown CH, Cuijpers P, Duan N, Kwasny MJ, Stiles-Shields C, Cheung K. Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. J Med Internet Res. 2015;17(7):e166. doi: 10.2196/jmir.4391.
    1. Riley WT, Glasgow RE, Etheredge L, Abernethy AP. Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise. Clin Transl Med. 2013;2(1):10. doi: 10.1186/2001-1326-2-10.
    1. Smith A. Record shares of Americans now own smartphones, have home broadband. Washington: Pew Internet and American Life Project; 2017. [2017-04-28].
    1. Lyon AR, Wasse JK, Ludwig K, Zachry M, Bruns EJ, Unützer J, McCauley E. The Contextualized Technology Adaptation Process (CTAP): optimizing health information technology to improve mental health systems. Adm Policy Ment Health. 2016 May;43(3):394–409. doi: 10.1007/s10488-015-0637-x.
    1. Proudfoot J, Goldberg D, Mann A, Everitt B, Marks I, Gray JA. Computerized, interactive, multimedia cognitive-behavioural program for anxiety and depression in general practice. Psychol Med. 2003 Feb;33(2):217–227.
    1. Christensen H, Griffiths KM, Korten A. Web-based cognitive behavior therapy: analysis of site usage and changes in depression and anxiety scores. J Med Internet Res. 2002;4(1):e3. doi: 10.2196/jmir.4.1.e3.
    1. Mohr DC, Duffecy J, Jin L, Ludman EJ, Lewis A, Begale M, McCarthy M. Multimodal e-mental health treatment for depression: a feasibility trial. J Med Internet Res. 2010 Dec 19;12(5):e48. doi: 10.2196/jmir.1370.
    1. Alfonsson S, Olsson E, Linderman S, Winnerhed S, Hursti T. Is online treatment adherence affected by presentation and therapist support? A randomized controlled trial. Comput Hum Behav. 2016 Jul;60:550–558. doi: 10.1016/j.chb.2016.01.035.
    1. Lyon A, Koerner K. User-centered design for psychosocial intervention development and implementation. Clin Psychol Sci Pract. 2016;23(2):180–200.
    1. Buntrock C, Ebert DD, Lehr D, Smit F, Riper H, Berking M, Cuijpers P. Effect of a Web-based guided self-help intervention for prevention of major depression in adults with subthreshold depression: a randomized clinical trial. JAMA. 2016 May 3;315(17):1854–1863. doi: 10.1001/jama.2016.4326.
    1. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012 Mar;50(3):217–226. doi: 10.1097/MLR.0b013e3182408812.
    1. Mummah SA, Robinson TN, King AC, Gardner CD, Sutton S. IDEAS (Integrate, Design, Assess, and Share): a framework and toolkit of strategies for the development of more effective digital interventions to change health behavior. J Med Internet Res. 2016 Dec 16;18(12):e317. doi: 10.2196/jmir.5927.
    1. Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008;14(2):117–123. doi: 10.1097/.
    1. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011 Jan;38(1):4–23. doi: 10.1007/s10488-010-0327-7.
    1. Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117. doi: 10.1186/1748-5908-8-117.
    1. Brownson RC, Jacobs JA, Tabak RG, Hoehner CM, Stamatakis KA. Designing for dissemination among public health researchers: findings from a national survey in the United States. Am J Public Health. 2013 Sep;103(9):1693–1699. doi: 10.2105/AJPH.2012.301165.
    1. Ben-Zeev D, Drake R, Marsch L. Clinical technology specialists. BMJ. 2015 Feb 19;350:1.
    1. Schuler D, Namioka A, editors. Participatory Design: Principles and Practices. Hillsdale: Lawrence Erlbaum Associates; 1993.
    1. Muller M. The Human-Computer Interaction Handbook. Hillsdale: Lawrence Erlbaum Associates; 2003. Participatory design: the third space in HCI; pp. 1051–1068.
    1. Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc. 2015 Nov;22(6):1179–1182. doi: 10.1093/jamia/ocv050.
    1. Norman D, Draper S, editors. User-Centered System Design: New Perspectives on Human-Computer Interaction, 1st edition. Hillsdale: Lawrence Erlbaum Associates; 1986.
    1. Abras C, Maloney-Krichmar D, Preece J. User-centered design. In: Bainbridge W, editor. Encyclopedia of Human-Computer Interaction. Thousand Oaks: Sage Publications; 2004. pp. 445–456.
    1. McCurdie T, Taneva S, Casselman M, Yeung M, McDaniel C, Ho W, Cafazzo J. mHealth consumer apps: the case for user-centered design. Biomed Instrum Technol. 2012:49–56. doi: 10.2345/0899-8205-46.s2.49.
    1. Vredenburg K, Mao J, Smith P, Carey T. A survey of user-centered design practice. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; 2002; Minneapolis. 2002.
    1. Lazar J, Feng J, Hochheiser H. Research methods in human-computer interaction. Hoboken: John Wiley & Sons; 2010. p. 0470723378.
    1. Krueger R, Casey M. Focus Groups: A Practical Guide for Applied Research. Thousand Oaks: Sage Publications; 2014.
    1. Reddy M, Shabot M, Bradner E. Evaluating collaborative features of critical care systems: a methodological study of information technology in surgical intensive care units. J Biomed Inform. 2008 Jun;41(3):479–487. doi: 10.1016/j.jbi.2008.01.004.
    1. Reddy MC, McDonald DW, Pratt W, Shabot MM. Technology, work, and information flows: lessons from the implementation of a wireless alert pager system. J Biomed Inform. 2005 Jun;38(3):229–238. doi: 10.1016/j.jbi.2004.11.010.
    1. Bessant J, Maher L. Developing radical service innovations in healthcare? The role of design methods. Int J Innov Manage. 2009;13(04):555–568.
    1. Mackay W. The interactive thread: exploring methods for multi-disciplinary design. Proceedings of the 5th conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques; 2004; Cambridge. 2004.
    1. King S, Conley M, Latimer B, Ferrari D. Co-Design: A Process of Design Participation. New York: Van Nostrand Reinhold Company; 1989.
    1. Sefelin R, Tscheligi M, Giller V. Paper prototyping—what is it good for? A comparison of paper- and computer-based low-fidelity prototyping. CHI '03 extended abstracts on Human factors in computing systems; 2003; Ft. Lauderdale. 2003.
    1. Spinuzzi C. The methodology of participatory design. Tech Commun. 2005;52(2):163–174.
    1. Diaper D, Stanton N. The Handbook of Task Analysis for Human-Computer Interaction. Mahwah: Lawrence Erlbausm Assoc; 2003.
    1. Nielsen J. Finding usability problems through heuristic evaluation. CHI '92 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; 1992; Monterey. 1992.
    1. Militello LG, Hutton RJ. Applied cognitive task analysis (ACTA): a practitioner's toolkit for understanding cognitive task demands. Ergonomics. 1998 Nov;41(11):1618–1641. doi: 10.1080/001401398186108.
    1. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2009 Jan;36(1):24–34. doi: 10.1007/s10488-008-0197-4.
    1. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011 Mar;38(2):65–76. doi: 10.1007/s10488-010-0319-7.
    1. Eysenbach G. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res. 2011;13(4):e126. doi: 10.2196/jmir.1923.
    1. Kristensson P, Bi X, Howes A, Oulasvirta A, Murray-Smith R, Thimbleby H. Principles, techniques, and perspectives on optimization. CHI ' 15 Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems; 2015; Seoul. 2015.
    1. Yen P, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc. 2012 Jun;19(3):413–422. doi: 10.1136/amiajnl-2010-000020.
    1. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2009 Jan;36(1):24–34. doi: 10.1007/s10488-008-0197-4.
    1. Kenter RM, van de Ven PM, Cuijpers P, Koole G, Niamat S, Gerrits RS, Willems M, van Straten A. Costs and effects of Internet cognitive behavioral treatment blended with face-to-face treatment: Results from a naturalistic study. Internet Interv. 2015 Mar;2(1):77–83. doi: 10.1016/j.invent.2015.01.001.
    1. Hekler E, Klasnja P, Froelich JE, Buman M. Mind the theoretical gap: interpreting, using, and developing behavioral theory in HCI research. SIGCHI Conference on Human Factors in Computing Systems; 2013; Paris. 2013.
    1. Mohr DC, Cheung K, Schueller SM, Hendricks BC, Duan N. Continuous evaluation of evolving behavioral intervention technologies. Am J Prev Med. 2013 Oct;45(4):517–523. doi: 10.1016/j.amepre.2013.06.006.
    1. Collins LM, Murphy SA, Nair VN, Strecher VJ. A strategy for optimizing and evaluating behavioral interventions. Ann Behav Med. 2005 Aug;30(1):65–73. doi: 10.1207/s15324796abm3001_8.
    1. Hekler EB, Klasnja P, Riley WT, Buman MP, Huberty J, Rivera DE, Martin CA. Agile science: creating useful products for behavior change in the real world. Transl Behav Med. 2016 Jun;6(2):317–328. doi: 10.1007/s13142-016-0395-7.
    1. Wiltsey SS, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012 Mar 14;7:17. doi: 10.1186/1748-5908-7-17.
    1. Fixsen DL, Blase KA, Naoom SF, Wallace F. Core Implementation Components. Res Soc Work Prac. 2009 May 27;19(5):531–540. doi: 10.1177/1049731509335549.
    1. Glisson C, Schoenwald SK. The ARC organizational and community intervention strategy for implementing evidence-based children's mental health treatments. Ment Health Serv Res. 2005 Dec;7(4):243–259. doi: 10.1007/s11020-005-7456-1.
    1. Chambers DA, Feero WG, Khoury MJ. Convergence of implementation science, precision medicine, and the learning health care system: a new model for biomedical research. JAMA. 2016 May 10;315(18):1941–1942. doi: 10.1001/jama.2016.3867.
    1. Novins DK, Green AE, Legha RK, Aarons GA. Dissemination and implementation of evidence-based practices for child and adolescent mental health: a systematic review. J Am Acad Child Adolesc Psychiatry. 2013 Oct;52(10):1009–1025. doi: 10.1016/j.jaac.2013.07.012.
    1. Gallo C, Pantin H, Villamar J, Prado G, Tapia M, Ogihara M, Cruden G, Brown CH. Blending qualitative and computational linguistics methods for fidelity assessment: experience with the Familias Unidas preventive intervention. Adm Policy Ment Health. 2015 Sep;42(5):574–585. doi: 10.1007/s10488-014-0538-4.
    1. Boehm B, Basili V. Top 10 list [software development] Computer. 2001;34(1):135–137. doi: 10.1109/2.962984.
    1. Aarons GA, Green AE, Trott E, Willging CE, Torres EM, Ehrhart MG, Roesch SC. The roles of system and organizational leadership in system-wide evidence-based intervention sustainment: a mixed-method study. Adm Policy Ment Health. 2016 Nov;43(6):991–1008. doi: 10.1007/s10488-016-0751-4.
    1. Ford JH, Alagoz E, Dinauer S, Johnson KA, Pe-Romashko K, Gustafson DH. Successful organizational strategies to sustain use of A-CHESS: a mobile intervention for individuals with alcohol use disorders. J Med Internet Res. 2015 Aug 18;17(8):e201. doi: 10.2196/jmir.3965.
    1. Ebert DD, Donkin L, Andersson G, Andrews G, Berger T, Carlbring P, Rozenthal A, Choi I, Laferton JA, Johansson R, Kleiboer A, Lange A, Lehr D, Reins JA, Funk B, Newby J, Perini S, Riper H, Ruwaard J, Sheeber L, Snoek FJ, Titov N, Ünlü IB, Vernmark K, Warmerdam L, Salsman N, Cuijpers P. Does Internet-based guided-self-help for depression cause harm? An individual participant data meta-analysis on deterioration rates and its moderators in randomized controlled trials. Psychol Med. 2016 Oct;46(13):2679–2693. doi: 10.1017/S0033291716001562.
    1. Rozental A, Andersson G, Boettcher J, Ebert D, Cuijpers P, Knaevelsrud C. Consensus statement on defining and measuring negative effects of Internet interventions. Internet Interv. 2014;1:12–9.
    1. Stecklein J, Dabney J, Dick B, Haskins B, Lovell R, Moroney G. Error cost escalation through the project life cycle. International Council on Systems Engineering (INCOSE): 14th Annual International Symposium; 2004; Toulouse. 2004.

Source: PubMed

Patrocinadores y Colaboradores

Condiciones médicas

Intervenciones de drogas

3
Suscribir