Adaptive learning algorithms to optimize mobile applications for behavioral health: guidelines for design decisions

Abstract

Objective: Providing behavioral health interventions via smartphones allows these interventions to be adapted to the changing behavior, preferences, and needs of individuals. This can be achieved through reinforcement learning (RL), a sub-area of machine learning. However, many challenges could affect the effectiveness of these algorithms in the real world. We provide guidelines for decision-making.

Materials and methods: Using thematic analysis, we describe challenges, considerations, and solutions for algorithm design decisions in a collaboration between health services researchers, clinicians, and data scientists. We use the design process of an RL algorithm for a mobile health study "DIAMANTE" for increasing physical activity in underserved patients with diabetes and depression. Over the 1.5-year project, we kept track of the research process using collaborative cloud Google Documents, Whatsapp messenger, and video teleconferencing. We discussed, categorized, and coded critical challenges. We grouped challenges to create thematic topic process domains.

Results: Nine challenges emerged, which we divided into 3 major themes: 1. Choosing the model for decision-making, including appropriate contextual and reward variables; 2. Data handling/collection, such as how to deal with missing or incorrect data in real-time; 3. Weighing the algorithm performance vs effectiveness/implementation in real-world settings.

Conclusion: The creation of effective behavioral health interventions does not depend only on final algorithm performance. Many decisions in the real world are necessary to formulate the design of problem parameters to which an algorithm is applied. Researchers must document and evaulate these considerations and decisions before and during the intervention period, to increase transparency, accountability, and reproducibility.

Trial registration: clinicaltrials.gov, NCT03490253.

Keywords: algorithms; behavioral medicine; implementation science; machine learning; telemedicine.

© The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com.

Figures

Figure 1.

The DIAMANTE study is a randomized controlled trial with 3 groups (uniform random, reinforcement learning, and a control group). In the reinforcement learning group each morning, the algorithm evaluates which messages, delivered at what time period, will likely increase steps for every participant in the upcoming day. The algorithm training data consist of the historical data of all participants (contextual variables), which include time-varying variables, and select clinical/demographic data to improve prediction abilities. Our action space is defined by the 5x4x4x2 DIAMANTE factorial design (5 intervention options for a “feedback” message and 4 intervention options for a “motivational” message, including the “no-message” category, 4 options for the time frame, and 2 social categories [individual or family]).

Figure 2.

Challenges and considerations when using reinforcement learning in clinical mobile health studies.