Evaluating the use of a recommender system for selecting optimal messages for smoking cessation: patterns and effects of user-system engagement

Jinying Chen, Thomas K Houston, Jamie M Faro, Catherine S Nagawa, Elizabeth A Orvek, Amanda C Blok, Jeroan J Allison, Sharina D Person, Bridget M Smith, Rajani S Sadasivam, Jinying Chen, Thomas K Houston, Jamie M Faro, Catherine S Nagawa, Elizabeth A Orvek, Amanda C Blok, Jeroan J Allison, Sharina D Person, Bridget M Smith, Rajani S Sadasivam

Abstract

Background: Motivational messaging is a frequently used digital intervention to promote positive health behavior changes, including smoking cessation. Typically, motivational messaging systems have not actively sought feedback on each message, preventing a closer examination of the user-system engagement. This study assessed the granular user-system engagement around a recommender system (a new system that actively sought user feedback on each message to improve message selection) for promoting smoking cessation and the impact of engagement on cessation outcome.

Methods: We prospectively followed a cohort of current smokers enrolled to use the recommender system for 6 months. The system sent participants motivational messages to support smoking cessation every 3 days and used machine learning to incorporate user feedback (i.e., user's rating on the perceived influence of each message, collected on a 5-point Likert scale with 1 indicating strong disagreement and 5 indicating strong agreement on perceiving the influence on quitting smoking) to improve the selection of the following message. We assessed user-system engagement by various metrics, including user response rate (i.e., the percent of times a user rated the messages) and the perceived influence of messages. We compared retention rates across different levels of user-system engagement and assessed the association between engagement and the 7-day point prevalence abstinence (missing outcome = smoking) by using multiple logistic regression.

Results: We analyzed data from 731 participants (13% Black; 73% women). The user response rate was 0.24 (SD = 0.34) and user-perceived influence was 3.76 (SD = 0.84). The retention rate positively increased with the user response rate (trend test P < 0.001). Compared with non-response, six-month cessation increased with the levels of response rates: low response rate (odds ratio [OR] = 1.86, 95% confidence interval [CI]: 1.07-3.23), moderate response rate (OR = 2.30, 95% CI: 1.36-3.88), high response rate (OR = 2.69, 95% CI: 1.58-4.58). The association between perceived message influence and the outcome showed a similar pattern.

Conclusions: High user-system engagement was positively associated with both high retention rate and smoking cessation, suggesting that investigation of methods to increase engagement may be crucial to increase the impact of the recommender system for smoking cessation.

Trial registration: Registration Identifier: NCT03224520 . Registration date: July 21, 2017.

Keywords: Computer-tailored health communication; Digital health intervention; Engagement; Motivational messaging; Recommender system; Smoking cessation.

Conflict of interest statement

None declared.

© 2021. The Author(s).

Figures

Fig. 1
Fig. 1
Trend of responses over time, for users with different levels of 6-month response rate. High-response rate: > 0.6 & ≤1, Medium-response rate: > 0.1 & ≤0.6, Low-response rate: > 0 & ≤0.1
Fig. 2
Fig. 2
Retention rate by level of (a) response rate (P < 0.001) and (b) rating score of perceived influence
Fig. 3
Fig. 3
Response rate by level of rating score of perceived influence (P < 0.001)

References

    1. Taylor GM, Dalili MN, Semwal M, Civljak M, Sheikh A, Car J. Internet-based interventions for smoking cessation. Cochrane Database Syst Rev. 2017;9(9). 10.1002/14651858.CD007078.pub5.
    1. Whittaker R, McRobbie H, Bullen C, Rodgers A, Gu Y. Mobile phone-based interventions for smoking cessation. Cochrane Database Syst Rev. 2016;4. 10.1002/14651858.CD006611.pub4.
    1. Scott-Sheldon LA, Lantini R, Jennings EG, Thind H, Rosen RK, Salmoirago-Blotcher E, et al. Text messaging-based interventions for smoking cessation: a systematic review and Meta-analysis. JMIR Mhealth Uhealth. 2016;4(2):e49. doi: 10.2196/mhealth.5436.
    1. Whittaker R, McRobbie H, Bullen C, Rodgers A, Gu Y, Dobson R. Mobile phone text messaging and app-based interventions for smoking cessation. Cochrane Database Syst Rev. 2019;10. 10.1002/14651858.CD006611.pub5.
    1. Abroms LC, Carroll P, Boal AL, Mendel J, Carpenter KM. Integrated phone counselling and text messaging services at quitlines: an acceptability study. J Smok Cessat. 2016;11(1):5–11. doi: 10.1017/jsc.2014.11.
    1. Kreuter MW, Farrell DW, Olevitch LR, Brennan LK. Tailoring health messages: Customizing communication with computer technology. England: Routledge; 2013.
    1. DiClemente CC, Prochaska JO, Fairhurst SK, Velicer WF, Velasquez MM, Rossi JS. The process of smoking cessation: an analysis of precontemplation, contemplation, and preparation stages of change. J Consult Clin Psychol. 1991;59(2):295–304. doi: 10.1037/0022-006X.59.2.295.
    1. Eldredge LKB, Markham CM, Ruiter RA, Fernández ME, Kok G, Parcel GS. Planning health promotion programs: an intervention mapping approach. Hoboken: Wiley; 2016.
    1. Bandura A. Social foundations of thought and action: a social cognitive theory. Englewood Cliffs, NJ: Prentice Hall; 1986.
    1. Brug J, Steenhuis I, van Assema P, de Vries H. The impact of a computer-tailored nutrition intervention. Prev Med. 1996;25(3):236–242. doi: 10.1006/pmed.1996.0052.
    1. Campbell MK, DeVellis BM, Strecher VJ, Ammerman AS, DeVellis RF, Sandler RS. Improving dietary behavior: the effectiveness of tailored messages in primary care settings. Am J Public Health. 1994;84(5):783–787. doi: 10.2105/AJPH.84.5.783.
    1. Bull FC, Kreuter MW, Scharff DP. Effects of tailored, personalized and general health messages on physical activity. Patient Educ Couns. 1999;36(2):181–192. doi: 10.1016/S0738-3991(98)00134-7.
    1. Marcus BH, Emmons KM, Simkin-Silverman LR, Linnan LA, Taylor ER, Bock BC, Roberts MB, Rossi JS, Abrams DB. Evaluation of motivationally tailored vs. standard self-help physical activity interventions at the workplace. Am J Health Promot. 1998;12(4):246–253. doi: 10.4278/0890-1171-12.4.246.
    1. Kreuter MW, Strecher VJ. Do tailored behavior change messages enhance the effectiveness of health risk appraisal? Results from a randomized trial. Health Educ Res. 1996;11(1):97–105. doi: 10.1093/her/11.1.97.
    1. Prochaska JO, DiClemente CC, Velicer WF, Rossi JS. Standardized, individualized, interactive, and personalized self-help programs for smoking cessation. Health Psychol. 1993;12(5):399–405. doi: 10.1037/0278-6133.12.5.399.
    1. Strecher VJ, McClure JB, Alexander GL, Chakraborty B, Nair VN, Konkel JM, et al. Web-based smoking-cessation programs: results of a randomized trial. Am J Prev Med. 2008;34(5):373–381. doi: 10.1016/j.amepre.2007.12.024.
    1. Houston TK, Sadasivam RS, Allison JJ, Ash AS, Ray MN, English TM, Hogan TP, Ford DE. Evaluating the QUIT-PRIMO clinical practice ePortal to increase smoker engagement with online cessation interventions: a national hybrid type 2 implementation study. Implement Sci. 2015;10(1):154. doi: 10.1186/s13012-015-0336-8.
    1. Rodgers A, Corbett T, Bramley D, Riddell T, Wills M, Lin R-B, Jones M. Do u smoke after txt? Results of a randomised trial of smoking cessation using mobile phone text messaging. Tob Control. 2005;14(4):255–261. doi: 10.1136/tc.2005.011577.
    1. Free C, Knight R, Robertson S, Whittaker R, Edwards P, Zhou W, Rodgers A, Cairns J, Kenward MG, Roberts I. Smoking cessation support delivered via mobile phone text messaging (txt2stop): a single-blind, randomised trial. Lancet. 2011;378(9785):49–55. doi: 10.1016/S0140-6736(11)60701-0.
    1. Te Poel F, Bolman C, Reubsaet A, de Vries H. Efficacy of a single computer-tailored e-mail for smoking cessation: results after 6 months. Health Educ Res. 2009;24(6):930–940. doi: 10.1093/her/cyp036.
    1. Sadasivam RS, Cutrona SL, Kinney RL, Marlin BM, Mazor KM, Lemon SC, Houston TK. Collective-intelligence recommender systems: advancing computer tailoring for health behavior change into the 21st century. J Med Internet Res. 2016;18(3):e42. doi: 10.2196/jmir.4448.
    1. Marlin BM, Adams RJ, Sadasivam R, Houston TK. Towards collaborative filtering recommender systems for tailored health communications. AMIA Annual Symposium Proceedings: American Medical Informatics Association; 2013. p. 1600–7.
    1. Adams RJ, Sadasivam RS, Balakrishnan K, Kinney RL, Houston TK, Marlin BM. PERSPeCT: collaborative filtering for tailored health communications. Proceedings of the 8th ACM Conference on Recommender systems; 2014. p. 329–32.
    1. Hors-Fraile S, Schneider F, Fernandez-Luque L, Luna-Perejon F, Civit A, Spachos D, Bamidis P, de Vries H. Tailoring motivational health messages for smoking cessation using an mHealth recommender system integrated with an electronic health record: a study protocol. BMC Public Health. 2018;18(1):1–10. doi: 10.1186/s12889-018-5612-5.
    1. Hors-Fraile S, Malwade S, Spachos D, Fernandez-Luque L, Su C-T, Jeng W-L, Syed-Abdul S, Bamidis P, Li YC(J) A recommender system to quit smoking with mobile motivational messages: study protocol for a randomized controlled trial. Trials. 2018;19(1):1–12. doi: 10.1186/s13063-018-3000-1.
    1. Sadasivam RS, Borglund EM, Adams R, Marlin BM, Houston TK. Impact of a collective intelligence tailored messaging system on smoking cessation: the Perspect randomized experiment. J Med Internet Res. 2016;18(11):e285. doi: 10.2196/jmir.6465.
    1. Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A systematic review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res. 2011;13(3):e52. doi: 10.2196/jmir.1772.
    1. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med. 2017;7(2):254–267. doi: 10.1007/s13142-016-0453-1.
    1. Michie S, Yardley L, West R, Patrick K, Greaves F. Developing and evaluating digital interventions to promote behavior change in health and health care: recommendations resulting from an international workshop. J Med Internet Res. 2017;19(6):e232. doi: 10.2196/jmir.7126.
    1. Short CE, DeSmet A, Woods C, Williams SL, Maher C, Middelweerd A, Müller AM, Wark PA, Vandelanotte C, Poppe L, Hingle MD, Crutzen R. Measuring engagement in eHealth and mHealth behavior change interventions: viewpoint of methodologies. J Med Internet Res. 2018;20(11):e292. doi: 10.2196/jmir.9397.
    1. Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, Merchant GC, Naughton F, Blandford A. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. 2016;51(5):833–842. doi: 10.1016/j.amepre.2016.06.015.
    1. Faro JM, Orvek EA, Blok AC, Nagawa CS, McDonald AJ, Seward G, et al. Dissemination and effectiveness of the peer marketing and messaging of a web-assisted tobacco intervention: protocol for a hybrid effectiveness trial. JMIR Res Protoc. 2019;8(7):e14814. doi: 10.2196/14814.
    1. Decide2Quit. Available from: .
    1. Coley HL, Sadasivam RS, Williams JH, Volkman JE, Schoenberger Y-M, Kohler CL, Sobko H, Ray MN, Allison JJ, Ford DE, Gilbert GH, Houston TK, National Dental PBRN and QUITPRIMO Collaborative Group Crowdsourced peer-versus expert-written smoking-cessation messages. Am J Prev Med. 2013;45(5):543–550. doi: 10.1016/j.amepre.2013.07.004.
    1. Berglund F, Bernstein D, Eisinger R, Hochbaum G, Lichtenstein E, Schwartz J, et al. Guidelines for research on the effectiveness of smoking cessation programs. Chicago: American Dental Association, National Interagency Council on Smoking & Health; 1974.
    1. Velicer WF, Prochaska JO. A comparison of four self-report smoking cessation outcome measures. Addict Behav. 2004;29(1):51–60. doi: 10.1016/S0306-4603(03)00084-4.
    1. West R, Hajek P, Stead L, Stapleton J. Outcome criteria in smoking cessation trials: proposal for a common standard. Addiction. 2005;100(3):299–303. doi: 10.1111/j.1360-0443.2004.00995.x.
    1. Lichtenstein E, Glasgow RE. Smoking cessation: what have we learned over the past decade? J Consult Clin Psychol. 1992;60(4):518–527. doi: 10.1037/0022-006X.60.4.518.
    1. StataCorp. Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC. 2017.
    1. Westmaas JL, Bontemps-Jones J, Hendricks PS, Kim J, Abroms LC. Randomised controlled trial of stand-alone tailored emails for smoking cessation. Tob Control. 2018;27(2):136–146. doi: 10.1136/tobaccocontrol-2016-053056.
    1. Wangberg SC, Nilsen O, Antypas K, Gram IT. Effect of tailoring in an internet-based intervention for smoking cessation: randomized controlled trial. J Med Internet Res. 2011;13(4):e1605. doi: 10.2196/jmir.1605.
    1. Abroms LC, Boal AL, Simmens SJ, Mendel JA, Windsor RA. A randomized trial of Text2Quit: a text messaging program for smoking cessation. Am J Prev Med. 2014;47(3):242–250. doi: 10.1016/j.amepre.2014.04.010.
    1. Brendryen H, Drozd F, Kraft P. A digital smoking cessation program delivered through internet and cell phone without nicotine replacement (happy ending): randomized controlled trial. J Med Internet Res. 2008;10(5):e51. doi: 10.2196/jmir.1005.
    1. Gram IT, Larbi D, Wangberg SC. Comparing the efficacy of an identical, tailored smoking cessation intervention delivered by mobile text messaging versus email: randomized controlled trial. JMIR mHealth and uHealth. 2019;7(9):e12137. doi: 10.2196/12137.
    1. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol. 2008;4(1):1–32. doi: 10.1146/annurev.clinpsy.3.022806.091415.
    1. Eysenbach G. The law of attrition. J Med Internet Res. 2005;7(1):e11. doi: 10.2196/jmir.7.1.e11.
    1. Cole-Lewis H, Augustson E, Sanders A, Schwarz M, Geng Y, Coa K, Hunt Y. Analysing user-reported data for enhancement of SmokefreeTXT: a national text message smoking cessation intervention. Tob Control. 2017;26(6):683–689. doi: 10.1136/tobaccocontrol-2016-052945.
    1. Mason D, Gilbert H, Sutton S. Effectiveness of web-based tailored smoking cessation advice reports (i Q uit): a randomized trial. Addiction. 2012;107(12):2183–2190. doi: 10.1111/j.1360-0443.2012.03972.x.
    1. Strecher VJ, Shiffman S, West R. Randomized controlled trial of a web-based computer-tailored smoking cessation program as a supplement to nicotine patch therapy. Addiction. 2005;100(5):682–688. doi: 10.1111/j.1360-0443.2005.01093.x.
    1. Danaher BG, Tyler MS, Crowley RC, Brendryen H, Seeley JR. Outcomes and device usage for fully automated internet interventions designed for a smartphone or personal computer: the MobileQuit smoking cessation randomized controlled trial. J Med Internet Res. 2019;21(6):e13290. doi: 10.2196/13290.
    1. Curry SJ, Sporer AK, Pugach O, Campbell RT, Emery S. Use of tobacco cessation treatments among young adult smokers: 2005 National Health Interview Survey. Am J Public Health. 2007;97(8):1464–1469. doi: 10.2105/AJPH.2006.103788.
    1. Solberg LI, Boyle RG, McCarty M, Asche SE, Thoele MJ. Young adult smokers: are they different? Am J Manag Care. 2007;13(11):626–632.
    1. Strecher V, McClure J, Alexander G, Chakraborty B, Nair V, Konkel J, Greene S, Couper M, Carlier C, Wiese C, Little R, Pomerleau C, Pomerleau O. The role of engagement in a tailored web-based smoking cessation program: randomized controlled trial. J Med Internet Res. 2008;10(5):e36. doi: 10.2196/jmir.1002.
    1. Cantrell J, Ilakkuvan V, Graham AL, Richardson A, Xiao H, Mermelstein RJ, Curry SJ, Sporer AK, Vallone DM. Young adult utilization of a smoking cessation website: an observational study comparing young and older adult patterns of use. JMIR research protocols. 2016;5(3):e142. doi: 10.2196/resprot.4881.
    1. Faro JM, Nagawa CS, Allison JA, Lemon SC, Mazor KM, Houston TK, Sadasivam RS. Comparison of a collective intelligence tailored messaging system on smoking cessation between African American and white people who smoke: quasi-experimental design. JMIR mHealth and uHealth. 2020;8(4):e18064. doi: 10.2196/18064.

Source: PubMed

Sponsorer og samarbejdspartnere

Medicinske tilstande

Narkotikainterventioner

3
Abonner