The law of attrition

Abstract

In an ongoing effort of this Journal to develop and further the theories, models, and best practices around eHealth research, this paper argues for the need for a "science of attrition", that is, a need to develop models for discontinuation of eHealth applications and the related phenomenon of participants dropping out of eHealth trials. What I call "law of attrition" here is the observation that in any eHealth trial a substantial proportion of users drop out before completion or stop using the application. This feature of eHealth trials is a distinct characteristic compared to, for example, drug trials. The traditional clinical trial and evidence-based medicine paradigm stipulates that high dropout rates make trials less believable. Consequently eHealth researchers tend to gloss over high dropout rates, or not to publish their study results at all, as they see their studies as failures. However, for many eHealth trials, in particular those conducted on the Internet and in particular with self-help applications, high dropout rates may be a natural and typical feature. Usage metrics and determinants of attrition should be highlighted, measured, analyzed, and discussed. This also includes analyzing and reporting the characteristics of the subpopulation for which the application eventually "works", ie, those who stay in the trial and use it. For the question of what works and what does not, such attrition measures are as important to report as pure efficacy measures from intention-to-treat (ITT) analyses. In cases of high dropout rates efficacy measures underestimate the impact of an application on a population which continues to use it. Methods of analyzing attrition curves can be drawn from survival analysis methods, eg, the Kaplan-Meier analysis and proportional hazards regression analysis (Cox model). Measures to be reported include the relative risk of dropping out or of stopping the use of an application, as well as a "usage half-life", and prediction models reporting demographic usage discontinuation in a population. Differential dropout or usage rates between two interventions could be a standard metric for the "usability efficacy" of a system. A "run-in and withdrawal" trial design is suggested as a methodological innovation for Internet-based trials with a high number of initial dropouts/nonusers and a stable group of hardcore users.

Figures

Figure 1

Nonusage attrition curves for two studies [1,2] published in this issue of the Journal of Medical Internet Research. Plotted are the number of completed modules from two Web-based interventions against the proportion of participants completing them. From the two Christensen/Moodgym curves, the upper one refers to a trial setting, while the other (lower one) refers to an “open” situation with casual Internet visitors.

Figure 2

An example for logarithmic “attrition curves” in a hypothetical eHealth trial. In the intervention group (INTV), a proportion of participants will be lost to follow-up (INTV dropout), as will be in the control group (CTRL dropout). In addition, even within those not lost to follow up, there might be a proportion of nonusers

Figure 3

Attrition curves from Figure 1 on a logarithmic scale (y-axis is the natural logarithm of the proportion of users completing a module)

Figure 4

A (hypothetical) sigmoid attrition curve

Figure 5

A proposed “run-in and withdrawal” design