Using multiple imputations to accommodate time-outs in online interventions

Susan M Shortreed, Andy Bogart, Jennifer B McClure, Susan M Shortreed, Andy Bogart, Jennifer B McClure

Abstract

Background: Accurately estimating the period of time that individuals are exposed to online intervention content is important for understanding program engagement. This can be calculated from time-stamped data reflecting navigation to and from individual webpages. Prolonged periods of inactivity are commonly handled with a time-out feature and assigned a prespecified exposure duration. Unfortunately, this practice can lead to biased results describing program exposure.

Objective: The aim of the study was to describe how multiple imputations can be used to better account for the time spent viewing webpages that result in a prolonged period of inactivity or a time-out.

Methods: To illustrate this method, we present data on time-outs collected from the Q(2) randomized smoking cessation trial. For this analysis, we evaluate the effects on intervention exposure of receiving content written in a prescriptive versus motivational tone. Using multiple imputations, we created five complete datasets in which the time spent viewing webpages that resulted in a time-out were replaced with values estimated with imputation models. We calculated standard errors using Rubin's formulas to account for the variability due to the imputations. We also illustrate how current methods of accounting for time-outs (excluding timed-out page views or assigning an arbitrary viewing time) can influence conclusions about participant engagement.

Results: A total of 63.00% (1175/1865) of participants accessed the online intervention in the Q(2) trial. Of the 6592 unique page views, 683 (10.36%, 683/6592) resulted in a time-out. The median time spent viewing webpages that did not result in a time-out was 1.07 minutes. Assuming participants did not spend any time viewing a webpage that resulted in a time-out, no difference between the two message tones was observed (ratio of mean time online: 0.87, 95% CI 0.75-1.02). Assigning 30 minutes of viewing time to all page views that resulted in a time-out concludes that participants who received content in a motivational tone spent less time viewing content (ratio of mean time online: 0.86, 95% CI 0.77-0.98) than those participants who received content in a prescriptive tone. Using multiple imputations to account for time-outs concludes that there is no difference in participant engagement between the two message tones (ratio of mean time online: 0.87; 95% CI 0.75-1.01).

Conclusions: The analytic technique chosen can significantly affect conclusions about online intervention engagement. We propose a standardized methodology in which time spent viewing webpages that result in a time-out is treated as missing information and corrected with multiple imputations.

Trial registration: Clinicaltrials.gov NCT00992264; https://ichgcp.net/clinical-trials-registry/NCT00992264 (Archived by WebCite at http://www.webcitation.org/6Kw5m8EkP).

Keywords: Internet; automatic time-out; behavioral research; engagement; multiple imputations; online interventions; smoking cessation; time spent online; utilization.

Conflict of interest statement

Conflicts of Interest: Dr Shortreed has received funding from research grants awarded to Group Health Research Institute by Bristol Meyers Squibb. Mr Bogart and Dr McClure have no conflicts of interest to declare.

Figures

Figure 1
Figure 1
Distribution of minutes spent viewing an intervention page, excluding page views that resulted in an automatic time-out.
Figure 2
Figure 2
Sensitivity of model results to assigning an arbitrary time spent online to page views that resulted in a time-out (estimate from the zero-inflated Poisson model for the ratio of the mean time spent online comparing individuals who received content in a prescriptive [RX] tone versus a motivational tone [MI]).

References

    1. Bennett GG, Glasgow RE. The delivery of public health interventions via the Internet: actualizing their potential. Annu Rev Public Health. 2009;30:273–92. doi: 10.1146/annurev.publhealth.031308.100235.
    1. Baker TB, Gustafson DH, Shaw B, Hawkins R, Pingree S, Roberts L, Strecher V. Relevance of CONSORT reporting criteria for research on eHealth interventions. Patient Educ Couns. 2010 Dec;81 Suppl:S77–86. doi: 10.1016/j.pec.2010.07.040.
    1. Proudfoot J, Klein B, Barak A, Carlbring P, Cuijpers P, Lange A, Ritterband L, Andersson G. Establishing guidelines for executing and reporting Internet intervention research. Cognitive Behaviour Therapy. 2011 Jun 12;40(2):82–97.
    1. Peterson ET. Web Site Measurement Hacks. Sebastopol, CA: O'Reilly Media; 2005.
    1. Danaher BG, Boles SM, Akers L, Gordon JS, Severson HH. Defining participant exposure measures in Web-based health behavior change programs. J Med Internet Res. 2006;8(3):e15. doi: 10.2196/jmir.8.3.e15.
    1. Danaher BG, Seeley JR. Methodological issues in research on web-based behavioral interventions. Ann Behav Med. 2009 Aug;38(1):28–39. doi: 10.1007/s12160-009-9129-0.
    1. Crenshaw K, Curry W, Salanitro AH, Safford MM, Houston TK, Allison JJ, Estrada CA. Is physician engagement with Web-based CME associated with patients' baseline hemoglobin A1c levels? The Rural Diabetes Online Care study. Acad Med. 2010 Sep;85(9):1511–7. doi: 10.1097/ACM.0b013e3181eac036.
    1. Crutzen R, Roosjen JL, Poelman J. Using Google Analytics as a process evaluation method for Internet-delivered interventions: an example on sexual health. Health Promot Int. 2013 Mar;28(1):36–42. doi: 10.1093/heapro/das008.
    1. Graham AL, Cha S, Papandonatos GD, Cobb NK, Mushro A, Fang Y, Niaura RS, Abrams DB. Improving adherence to web-based cessation programs: a randomized controlled trial study protocol. Trials. 2013;14:48. doi: 10.1186/1745-6215-14-48.
    1. McClure JB, Shortreed SM, Bogart A, Derry H, Riggs K, St John J, Nair V, An L. The effect of program design on engagement with an internet-based smoking intervention: randomized factorial trial. J Med Internet Res. 2013;15(3):e69. doi: 10.2196/jmir.2508.
    1. Richardson A, Graham AL, Cobb N, Xiao H, Mushro A, Abrams D, Vallone D. Engagement promotes abstinence in a web-based cessation intervention: cohort study. J Med Internet Res. 2013;15(1):e14. doi: 10.2196/jmir.2277.
    1. Glasgow RE, Christiansen SM, Kurz D, King DK, Woolley T, Faber AJ, Estabrooks PA, Strycker L, Toobert D, Dickman J. Engagement in a diabetes self-management website: usage patterns and generalizability of program use. J Med Internet Res. 2011;13(1):e9. doi: 10.2196/jmir.1391.
    1. Zbikowski SM, Jack LM, McClure JB, Deprey M, Javitz HS, McAfee TA, Catz SL, Richards J, Bush T, Swan GE. Utilization of services in a randomized trial testing phone- and web-based interventions for smoking cessation. Nicotine Tob Res. 2011 May;13(5):319–27. doi: 10.1093/ntr/ntq257.
    1. Allison PD. Missing Data. Thousand Oaks, CA. USA: Sage; 2002.
    1. Little RJA, Rubin DB. Statistical Analysis with Missing Data. Second ed. New York, NY. USA: J Wiley & Sons; 2002.
    1. National Research Council . The Prevention and Treatment of Missing Data in Clinical Trials. Panel on Handling Missing Data in Clinical Trials, Committee on National Statistics, Division of Behavioral and Social Sciences and Education, editors. Washington, DC: The National Academies Press; 2010.
    1. McClure JB, Derry H, Riggs KR, Westbrook EW, St John J, Shortreed SM, Bogart A, An LC. Questions about quitting (Q2): design and methods of a Multiphase Optimization Strategy (MOST) randomized screening experiment for an online, motivational smoking cessation intervention. Contemp Clin Trials. 2012 Sep;33(5):1094–102. doi: 10.1016/j.cct.2012.06.009.
    1. Wu CFJ, Hamada M. Experiments: Planning, Analysis, and Parameter Design Optimization. New York: Wiley and Sons, Inc; 2000.
    1. Miller WR, Rollnick S. Motivational Interviewing: Preparing People for Change. 2nd ed. New York: The Guilford Press; 2002.
    1. Little RJ, D'Agostino R, Cohen ML, Dickersin K, Emerson SS, Farrar JT, Frangakis C, Hogan JW, Molenberghs G, Murphy SA, Neaton JD, Rotnitzky A, Scharfstein D, Shih WJ, Siegel JP, Stern H. The prevention and treatment of missing data in clinical trials. N Engl J Med. 2012 Oct 4;367(14):1355–60. doi: 10.1056/NEJMsr1203730.
    1. Blankers M, Koeter MW, Schippers GM. Missing data approaches in eHealth research: simulation study and a tutorial for nonmathematically inclined researchers. J Med Internet Res. 2010;12(5):e54. doi: 10.2196/jmir.1448.
    1. Schafer JL. Analysis of Incomplete Multivariate Data. London, UK: Chapman & Hall; 1997.
    1. Schafer JL. Imputation of missing covariates under a multivariate linear mixed model. Technical report, Dept. of Statistics, Penn State University. 1997:1–24.
    1. Raghunathan TE, Lepkowski JM, Van Hoewyk J, Solenberger P. A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology. 2001;27(1):85–95.
    1. Van Buuren S, Brand JPL, Groothuis-Oudshoorn CGM, Rubin DB. Fully conditional specification in multivariate imputation. Journal of Statistical Computation and Simulation. 2006;76(12):1049–64.
    1. van Buuren S. Multiple imputations of discrete and continuous data by fully conditional specification. Stat Methods Med Res. 2007 Jun;16(3):219–42. doi: 10.1177/0962280206074463.
    1. Raghunathan TE, Solenberger P, Van Hoewyk J. IVEware: Imputation and Variance Estimation software user guide. Survey Research Center, Institute for Social Research, University of Michigan. 2001:1–73.
    1. Horton NJ, Kleinman KP. Much ado about nothing: A comparison of missing data methods and software to fit incomplete data regression models. Am Stat. 2007 Feb;61(1):79–90. doi: 10.1198/000313007X172556.
    1. van Buuren S, Groothuis-Oudshoorn K. Mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software. 2011;45(3):1–67.
    1. Royston P, White IR. Multiple Imputations by Chained Equations (mice): Implementation in Stata. Journal of Statistical Software. 2011;45(1):1–20.
    1. Stuart EA, Azur M, Frangakis C, Leaf P. Multiple imputations with large data sets: a case study of the Children's Mental Health Initiative. Am J Epidemiol. 2009 May 1;169(9):1133–9. doi: 10.1093/aje/kwp026.
    1. Shortreed SM, Laber EB, Pineau J, Murphy SA. Imputations methods for the clinical antipsychotic trials of intervention and effectiveness study. School of Computer Science, McGill University. 2010:1–26.
    1. Schafer JL, Graham JW. Missing data: our view of the state of the art. Psychol Methods. 2002 Jun;7(2):147–77.
    1. Lee KJ, Carlin JB. Multiple imputations for missing data: fully conditional specification versus multivariate normal imputation. Am J Epidemiol. 2010 Mar 1;171(5):624–32. doi: 10.1093/aje/kwp425.
    1. Raghunathan TE, Siscovick DS. A multiple imputation analysis of a case-control study of the risk of primary cardiac arrest among pharmacologically treated hypertensives. Applied Statistics. 1996;45:335–52.
    1. Barnard J, Meng XL. Applications of multiple imputation in medical studies: from AIDS to NHANES. Stat Methods Med Res. 1999 Mar;8(1):17–36.
    1. Nevalainen J, Kenward MG, Virtanen SM. Missing values in longitudinal dietary data: a multiple imputation approach based on a fully conditional specification. Stat Med. 2009 Dec 20;28(29):3657–69. doi: 10.1002/sim.3731.
    1. Rubin DB, Schenker N. Multiple imputation for interval estimation from simple random samples with ignorable nonresponse. J Am Stat Assoc. 1986;81(394):366–74.
    1. Rubin DB. Multiple Imputation for Nonresponse in Surveys. New York: J Wiley & Sons; 1987.
    1. Böhning D, Dietz E, Schlattmann P, Mendonca L, Kirchner U. The zero-inflated poisson model the decayed, MissingFilled teeth index in dental epidemiology. J R Statist Soc A. 1999;162:195–209.
    1. Preisser JS, Stamm JW, Long DL, Kincade ME. Review and recommendations for zero-inflated count regression modeling of dental caries indices in epidemiological studies. Caries Res. 2012;46(4):413–23. doi: 10.1159/000338992.
    1. StataPress . Stata Statistical Software: Release 12. College Station, TX: Stata Corp; 2011.

Source: PubMed

Sponzoři a spolupracovníci

Zdravotní podmínky

Drogové intervence

3
Předplatit