REinforcement learning to improve non-adherence for diabetes treatments by Optimising Response and Customising Engagement (REINFORCE): study protocol of a pragmatic randomised trial

Julie C Lauffenburger, Elad Yom-Tov, Punam A Keller, Marie E McDonnell, Lily G Bessette, Constance P Fontanet, Ellen S Sears, Erin Kim, Kaitlin Hanken, J Joseph Buckley, Renee A Barlev, Nancy Haff, Niteesh K Choudhry, Julie C Lauffenburger, Elad Yom-Tov, Punam A Keller, Marie E McDonnell, Lily G Bessette, Constance P Fontanet, Ellen S Sears, Erin Kim, Kaitlin Hanken, J Joseph Buckley, Renee A Barlev, Nancy Haff, Niteesh K Choudhry

Abstract

Introduction: Achieving optimal diabetes control requires several daily self-management behaviours, especially adherence to medication. Evidence supports the use of text messages to support adherence, but there remains much opportunity to improve their effectiveness. One key limitation is that message content has been generic. By contrast, reinforcement learning is a machine learning method that can be used to identify individuals' patterns of responsiveness by observing their response to cues and then optimising them accordingly. Despite its demonstrated benefits outside of healthcare, its application to tailoring communication for patients has received limited attention. The objective of this trial is to test the impact of a reinforcement learning-based text messaging programme on adherence to medication for patients with type 2 diabetes.

Methods and analysis: In the REinforcement learning to Improve Non-adherence For diabetes treatments by Optimising Response and Customising Engagement (REINFORCE) trial, we are randomising 60 patients with suboptimal diabetes control treated with oral diabetes medications to receive a reinforcement learning intervention or control. Subjects in both arms will receive electronic pill bottles to use, and those in the intervention arm will receive up to daily text messages. The messages will be individually adapted using a reinforcement learning prediction algorithm based on daily adherence measurements from the pill bottles. The trial's primary outcome is average adherence to medication over the 6-month follow-up period. Secondary outcomes include diabetes control, measured by glycated haemoglobin A1c, and self-reported adherence. In sum, the REINFORCE trial will evaluate the effect of personalising the framing of text messages for patients to support medication adherence and provide insight into how this could be adapted at scale to improve other self-management interventions.

Ethics and dissemination: This study was approved by the Mass General Brigham Institutional Review Board (IRB) (USA). Findings will be disseminated through peer-reviewed journals, clinicaltrials.gov reporting and conferences.

Trial registration number: Clinicaltrials.gov (NCT04473326).

Keywords: clinical trials; diabetes & endocrinology; public health.

Conflict of interest statement

Competing interests: EYT is an employee of Microsoft. RAB is now an employee at Vytalize Health. There are no other reported competing interests.

© Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Figures

Figure 1
Figure 1
Overall trial design. HbA1c, haemoglobin A1c.
Figure 2
Figure 2
Timeline of study procedures.
Figure 3
Figure 3
Reinforcement learning platform.

References

    1. American Diabetes Association . Economic costs of diabetes in the U.S. in 2017. Diabetes Care 2018;41:917–28. 10.2337/dci18-0007
    1. Garber AJ, Abrahamson MJ, Barzilay JI, et al. . Consensus statement by the American association of clinical endocrinologists and American college of endocrinology on the comprehensive type 2 diabetes management algorithm - 2018 executive summary. Endocr Pract 2018;24:91–121. 10.4158/CS-2017-0153
    1. Gonzalez JS, Tanenbaum ML, Commissariat PV. Psychosocial factors in medication adherence and diabetes self-management: implications for research and practice. Am Psychol 2016;71:539–51. 10.1037/a0040388
    1. Lauffenburger JC, Mahesri M, Choudhry NK. Not there yet: using data-driven methods to predict who becomes costly among low-cost patients with type 2 diabetes. BMC Endocr Disord 2020;20:125. 10.1186/s12902-020-00609-1
    1. Whelton PK, Carey RM, Aronow WS, et al. . 2017 ACC/AHA/AAPA/ABC/ACPM/AGS/APhA/ASH/ASPC/NMA/PCNA guideline for the prevention, detection, evaluation, and management of high blood pressure in adults: a report of the American College of Cardiology/American heart association Task force on clinical practice guidelines. J Am Coll Cardiol 2018;71:e127–248. 10.1016/j.jacc.2017.11.006
    1. Carls GS, Tuttle E, Tan R-D, et al. . Understanding the gap between efficacy in randomized controlled trials and effectiveness in real-world use of GLP-1 RA and DPP-4 therapies in patients with type 2 diabetes. Diabetes Care 2017;40:1469–78. 10.2337/dc16-2725
    1. Viswanathan M, Golin CE, Jones CD, et al. . Interventions to improve adherence to self-administered medications for chronic diseases in the United States: a systematic review. Ann Intern Med 2012;157:785–95. 10.7326/0003-4819-157-11-201212040-00538
    1. Schellenberg ES, Dryden DM, Vandermeer B, et al. . Lifestyle interventions for patients with and at risk for type 2 diabetes: a systematic review and meta-analysis. Ann Intern Med 2013;159:543–51. 10.7326/0003-4819-159-8-201310150-00007
    1. Powers MA, Bardsley J, Cypress M, et al. . Diabetes self-management education and support in type 2 diabetes. Diabetes Educ 2017;43:40–53. 10.1177/0145721716689694
    1. Lauffenburger JC, Lewey J, Jan S, et al. . Effectiveness of targeted Insulin-Adherence interventions for glycemic control using predictive analytics among patients with type 2 diabetes: a randomized clinical trial. JAMA Netw Open 2019;2:e190657. 10.1001/jamanetworkopen.2019.0657
    1. Thakkar J, Kurup R, Laba T-L, et al. . Mobile telephone text messaging for medication adherence in chronic disease: a meta-analysis. JAMA Intern Med 2016;176:340–9. 10.1001/jamainternmed.2015.7667
    1. Lauffenburger JC, Choudhry NK. Text messaging and patient engagement in an increasingly mobile world. Circulation 2016;133:555–6. 10.1161/CIRCULATIONAHA.116.021182
    1. Fischer HH, Fischer IP, Pereira RI, et al. . Text message support for weight loss in patients with prediabetes: a randomized clinical trial. Diabetes Care 2016;39:1364–70. 10.2337/dc15-2137
    1. Pandey A, Krumme AA, Patel T, et al. . The impact of text messaging on medication adherence and exercise among Postmyocardial infarction patients: randomized controlled pilot trial. JMIR Mhealth Uhealth 2017;5:e110. 10.2196/mhealth.7144
    1. Dobson R, Whittaker R, Pfaeffli Dale L, et al. . The effectiveness of text message-based self-management interventions for poorly-controlled diabetes: a systematic review. Digit Health 2017;3:2055207617740315. 10.1177/2055207617740315
    1. Sahin C, Courtney KL, Naylor PJ, et al. . Tailored mobile text messaging interventions targeting type 2 diabetes self-management: a systematic review and a meta-analysis. Digit Health 2019;5:2055207619845279. 10.1177/2055207619845279
    1. Farmer AJ, McSharry J, Rowbotham S, et al. . Effects of interventions promoting monitoring of medication use and brief messaging on medication adherence for people with type 2 diabetes: a systematic review of randomized trials. Diabet Med 2016;33:565–79. 10.1111/dme.12987
    1. Affect KPA. Framing, and Persuasian. J Market Res 2003;40:54–64.
    1. Gong J, Zhang Y, Yang Z, et al. . The framing effect in medical decision-making: a review of the literature. Psychol Health Med 2013;18:645–53. 10.1080/13548506.2013.766352
    1. Yokum D, Lauffenburger JC, Ghazinouri R, et al. . Letters designed with behavioural science increase influenza vaccination in Medicare beneficiaries. Nat Hum Behav 2018;2:743–9. 10.1038/s41562-018-0432-2
    1. Gonul S, Namli T, Huisman S, et al. . An expandable approach for design and personalization of digital, just-in-time adaptive interventions. J Am Med Inform Assoc 2019;26:198–210. 10.1093/jamia/ocy160
    1. He P, Jagannathan S. Reinforcement learning-based output feedback control of nonlinear systems with input constraints. IEEE Trans Syst Man Cybern B Cybern 2005;35:150–4. 10.1109/tsmcb.2004.840124
    1. Shteingart H, Loewenstein Y. Reinforcement learning and human behavior. Curr Opin Neurobiol 2014;25:93–8. 10.1016/j.conb.2013.12.004
    1. Hochberg I, Feraru G, Kozdoba M, et al. . Encouraging physical activity in patients with diabetes through automatic personalized feedback via reinforcement learning improves glycemic control. Diabetes Care 2016;39:e59–60. 10.2337/dc15-2340
    1. Yom-Tov E, Feraru G, Kozdoba M, et al. . Encouraging physical activity in patients with diabetes: intervention using a reinforcement learning system. J Med Internet Res 2017;19:e338. 10.2196/jmir.7994
    1. Krakow EF, Hemmer M, Wang T, et al. . Tools for the precision medicine era: how to develop highly personalized treatment recommendations from cohort and registry data using Q-Learning. Am J Epidemiol 2017;186:160–72. 10.1093/aje/kwx027
    1. Paredes P G-BR, Czerwinski M, Roseway A, et al. . PopTherapy: coping with stress through pop-culture. paper presented at: 8th International Conference on pervasive computing technologies for Healthcare2014; Oldenberg, Germany.
    1. Guez A VRD, Avoli, M P. Treatment of epilepsy via batch-mode reinforcement learning. Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence 2008:1671–8.
    1. Klasnja P, Hekler EB, Shiffman S. Micro-randomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol 2015;34:1220–8.
    1. Komorowski M, Celi LA, Badawi O, et al. . The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nat Med 2018;24:1716–20. 10.1038/s41591-018-0213-5
    1. Lauffenburger JC, Fontanet CP, Isaac T, et al. . Comparison of a new 3-item self-reported measure of adherence to medication with pharmacy claims data in patients with cardiometabolic disease. Am Heart J 2020;228:36–43. 10.1016/j.ahj.2020.06.012
    1. U.S. smartphone use in 2015, 2015. Available:
    1. Baptista S, Wadley G, Bird D, et al. . My diabetes coach research G. user experiences with a type 2 diabetes coaching APP: qualitative study. JMIR Diabetes 2020;5:e16692.
    1. Aguilera A, Figueroa CA, Hernandez-Ramos R, et al. . mHealth app using machine learning to increase physical activity in diabetes and depression: clinical trial protocol for the DIAMANTE study. BMJ Open 2020;10:e034723. 10.1136/bmjopen-2019-034723
    1. Harris PA, Taylor R, Minor BL, et al. . The REDCap Consortium: building an international community of software platform partners. J Biomed Inform 2019;95:103208. 10.1016/j.jbi.2019.103208
    1. Harris PA, Taylor R, Thielke R, et al. . Research electronic data capture (REDCap)-a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42:377–81. 10.1016/j.jbi.2008.08.010
    1. Volpp KG, Troxel AB, Mehta SJ, et al. . Effect of electronic reminders, financial incentives, and social support on outcomes after myocardial infarction: the HeartStrong randomized clinical trial. JAMA Intern Med 2017;177:1093–101. 10.1001/jamainternmed.2017.2449
    1. Mehta SJ, Asch DA, Troxel AB. Comparison of pharmacy claims and electronic pill bottles for measurement of medication adherence among myocardial infarction patients. Med Care 2018.
    1. American Diabetes Association . Pharmacologic approaches to glycemic treatment: standards of medical care in diabetes-2018. Diabetes Care 2018;41:S73–85. 10.2337/dc18-S008
    1. Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev 1977;84:191–215. 10.1037//0033-295x.84.2.191
    1. Gintis H. A framework for the unification of the behavioral sciences. Behav Brain Sci 2007;30:1–16. 10.1017/S0140525X07000581
    1. Tzeng OC, Jackson JW. Common methodological framework for theory construction and evaluation in the social and behavioral sciences. Genet Soc Gen Psychol Monogr 1991;117:49–76.
    1. Lauffenburger JC, Barlev RA, Sears ES, et al. . Preferences for mHealth technology and text messaging communication in patients with type 2 diabetes: qualitative interview study. J Med Internet Res 2021;23:e25958. 10.2196/25958
    1. Baron RM. Social reinforcement effects as a function of social reinforcement history. Psychol Rev 1966;73:527–39. 10.1037/h0023896
    1. Lauffenburger JC, Khan NF, Brill G, et al. . Quantifying social reinforcement among family members on adherence to medications for chronic conditions: a US-Based retrospective cohort study. J Gen Intern Med 2019;34:855–61. 10.1007/s11606-018-4654-9
    1. Choudhry NK, Isaac T, Lauffenburger JC, et al. . Effect of a remotely delivered tailored multicomponent approach to enhance medication taking for patients with hyperlipidemia, hypertension, and diabetes: the STIC2IT cluster randomized clinical trial. JAMA Intern Med 2018;178:1182–9. 10.1001/jamainternmed.2018.3189
    1. Lewey J, Wei W, Lauffenburger JC, et al. . Targeted adherence intervention to reach glycemic control with insulin therapy for patients with diabetes (TARGIT-Diabetes): rationale and design of a pragmatic randomised clinical trial. BMJ Open 2017;7:e016551. 10.1136/bmjopen-2017-016551
    1. Delahanty LM, Chang Y, Levy DE, et al. . Design and participant characteristics of a primary care adaptation of the look ahead lifestyle intervention for weight loss in type 2 diabetes: the REAL HEALTH-diabetes study. Contemp Clin Trials 2018;71:9–17. 10.1016/j.cct.2018.05.018
    1. Personalizer. Available:
    1. Kianercy A, Galstyan A. Dynamics of Boltzmann Q learning in two-player two-action games. Phys Rev E Stat Nonlin Soft Matter Phys 2012;85:041145. 10.1103/PhysRevE.85.041145
    1. Laber EB, Linn KA, Stefanski LA. Interactive model building for Q-learning. Biometrika 2014;101:831–47. 10.1093/biomet/asu043
    1. Nahum-Shani I, Qian M, Almirall D, et al. . Q-learning: a data analysis method for constructing adaptive interventions. Psychol Methods 2012;17:478–94. 10.1037/a0029373
    1. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health 2013;103:e38–46. 10.2105/AJPH.2013.301299
    1. Lauffenburger JC, Ghazinouri R, Jan S, et al. . Impact of a novel pharmacist-delivered behavioral intervention for patients with poorly-controlled diabetes: the enhancing outcomes through goal assessment and generating engagement in diabetes mellitus (ENGAGE-DM) pragmatic randomized trial. PLoS One 2019;14:e0214754. 10.1371/journal.pone.0214754
    1. Sterne JAC, White IR, Carlin JB, et al. . Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ 2009;338:b2393. 10.1136/bmj.b2393
    1. McNutt L-A, Wu C, Xue X, et al. . Estimating the relative risk in cohort studies and clinical trials of common outcomes. Am J Epidemiol 2003;157:940–3. 10.1093/aje/kwg074
    1. Shipe ME, Deppen SA, Farjah F, et al. . Developing prediction models for clinical use using logistic regression: an overview. J Thorac Dis 2019;11:S574–84. 10.21037/jtd.2019.01.25
    1. Adams MA, Sallis JF, Norman GJ, et al. . An adaptive physical activity intervention for overweight adults: a randomized controlled trial. PLoS One 2013;8:e82901. 10.1371/journal.pone.0082901
    1. Zhou M, Fukuoka Y, Mintz Y, et al. . Evaluating machine Learning-Based automated personalized daily step goals delivered through a mobile phone APP: randomized controlled trial. JMIR Mhealth Uhealth 2018;6:e28. 10.2196/mhealth.9117
    1. Arambepola C, Ricci-Cabello I, Manikavasagam P, et al. . The impact of automated brief messages promoting lifestyle changes delivered via mobile devices to people with type 2 diabetes: a systematic literature review and meta-analysis of controlled trials. J Med Internet Res 2016;18:e86. 10.2196/jmir.5425
    1. Arora S, Peters AL, Burner E, et al. . Trial to examine text message-based mHealth in emergency department patients with diabetes (TExT-MED): a randomized controlled trial. Ann Emerg Med 2014;63:e746:745–54. 10.1016/j.annemergmed.2013.10.012
    1. Bobrow K, Farmer AJ, Springer D, et al. . Mobile Phone Text Messages to Support Treatment Adherence in Adults With High Blood Pressure (SMS-Text Adherence Support [StAR]): A Single-Blind, Randomized Trial. Circulation 2016;133:592–600. 10.1161/CIRCULATIONAHA.115.017530
    1. de Jongh T, Gurol-Urganci I, Vodopivec-Jamsek V, et al. . Mobile phone messaging for facilitating self-management of long-term illnesses. Cochrane Database Syst Rev 2012;12:CD007459. 10.1002/14651858.CD007459.pub2
    1. Fischer HH, Moore SL, Ginosar D, et al. . Care by cell phone: text messaging for chronic disease management. Am J Manag Care 2012;18:e42–7.
    1. Hamine S, Gerth-Guyette E, Faulx D, et al. . Impact of mHealth chronic disease management on treatment adherence and patient outcomes: a systematic review. J Med Internet Res 2015;17:e52. 10.2196/jmir.3951
    1. Hartz J, Yingling L, Powell-Wiley TM. Use of mobile health technology in the prevention and management of diabetes mellitus. Curr Cardiol Rep 2016;18:130. 10.1007/s11886-016-0796-8
    1. Arnsten JH, Demas PA, Farzadegan H, et al. . Antiretroviral therapy adherence and viral suppression in HIV-infected drug users: comparison of self-report and electronic monitoring. Clin Infect Dis 2001;33:1417–23. 10.1086/323201
    1. Troxel AB, Asch DA, Mehta SJ, et al. . Rationale and design of a randomized trial of automated hovering for post-myocardial infarction patients: the HeartStrong program. Am Heart J 2016;179:166–74. 10.1016/j.ahj.2016.06.006
    1. Mehta SJ, Asch DA, Troxel AB, et al. . Comparison of pharmacy claims and electronic pill bottles for measurement of medication adherence among myocardial infarction patients. Med Care 2019;57:e9–14. 10.1097/MLR.0000000000000950
    1. Wilson IB, Lee Y, Michaud J, et al. . Validation of a new Three-Item self-report measure for medication adherence. AIDS Behav 2016;20:2700–8. 10.1007/s10461-016-1406-x

Source: PubMed

Patrocinadores e Colaboradores

Condições médicas

Intervenções de drogas

3
Se inscrever