Comparing Amazon Mechanical Turk with unpaid internet resources in online clinical trials

Eduardo Bunge, Haley M Cook, Melissa Bond, Rachel E Williamson, Monique Cano, Alinne Z Barrera, Yan Leykin, Ricardo F Muñoz, Eduardo Bunge, Haley M Cook, Melissa Bond, Rachel E Williamson, Monique Cano, Alinne Z Barrera, Yan Leykin, Ricardo F Muñoz

Abstract

Internet interventions face significant challenges in recruitment and attrition rates are typically high and problematic. Finding innovative yet scientifically valid avenues for attaining and retaining participants is therefore of considerable importance. The main goal of this study was to compare recruitment process and participants characteristics between two similar randomized control trials of mood management interventions. One of the trials (Bunge et al., 2016) was conducted with participants recruited from Amazon's Mechanical Turk (AMT), and the other trial recruited via Unpaid Internet Resources (UIR).

Methods: The AMT sample (Bunge et al., 2016) consisted of 765 adults, and the UIR sample (recruited specifically for this study) consisted of 329 adult US residents. Participants' levels of depression, anxiety, confidence, motivation, and perceived usefulness of the intervention were assessed. The AMT sample was financially compensated whereas the UIR was not.

Results: AMT yielded higher recruitment rates per month (p < .05). At baseline, the AMT sample reported significantly lower depression and anxiety scores (p < .001 and p < .005, respectively) and significantly higher mood, motivation, and confidence (all p < .001) compared to the UIR sample. AMT participants spent significantly less time on the site (p < .05) and were more likely to complete follow-ups than the UIR sample (p < .05). Both samples reported a significant increase in their level of confidence and motivation from pre- to post-intervention. AMT participants showed a significant increase in perceived usefulness of the intervention (p < .0001), whereas the UIR sample did not (p = .1642).

Conclusions: By using AMT, researchers can recruit very rapidly and obtain higher retention rates; however, these participants may not be representative of the general online population interested in clinical interventions. Considering that AMT and UIR participants differed in most baseline variables, data from clinical studies resulting from AMT samples should be interpreted with caution.

Keywords: Amazon Mechanical Turk; Clinical trials; Depression; Internet interventions; Recruitment.

Figures

Fig. 1
Fig. 1
Progression of participants recruited from Amazon Mechanical Turk (AMT) and Unpaid Internet Resources (UIR). The Consort flow chart represents the AMT sample from Bunge et al. (2016) and the UIR sample.
Fig. 2
Fig. 2
Usefulness ratings by sample at pre-intervention and post-intervention. AMT: Amazon Mechanical Turk; UIR: Unpaid internet resources.

References

    1. Andersson G., Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn. Behav. Ther. 2009;38(4):196–205.
    1. Andersson G., Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn. Behav. Ther. 2009;38(4):196–205.
    1. Ashery R.S., McAuliffe W.E. Implementation issues and techniques in randomized trials of outpatient psychosocial treatments for drug abusers: recruitment of subjects. Am. J. Drug Alcohol Abuse. 1992;18(3):305–329.
    1. Baumeister H., Reichler L., Munzinger M., Lin J. The impact of guidance on internet-based mental health interventions — a systematic review. Internet Intervent. 2014;1(4):205–215.
    1. Buhrmester M., Kwang T., Gosling S.D. Amazon's mechanical Turk: a new source of inexpensive, yet high-quality, data? Perspect. Psychol. Sci. 2011;6(1):3–5.
    1. Bunge E.L., Williamson R.E., Cano M., Leykin Y., Muñoz R.F. Mood management effects of brief unsupported internet interventions. Internet Interventions. 2016;5:36–43.
    1. Clarke G., Reid D.E., O'Connor E., DeBar L.L., Kelleher C., Lynch F., Nunley S. Overcoming depression on the internet (ODIN): a randomized controlled trial of an internet depression skills intervention program. J. Med. Internet Res. 2002;4(3) Retrieved from.
    1. Cunningham J.A., Godinho A., Kushnir V. Can Amazon's Mechanical Turk be used to recruit participants for internet intervention trials? A pilot study involving a randomized controlled trial of a brief online intervention for hazardous alcohol use. Internet Intervent. 2017;10(Supplement C):12–16.
    1. Eysenbach G. The law of attrition. J. Med. Internet Res. 2005;7(1)
    1. Fenner Y., Garland S.M., Moore E.E., Jayasinghe Y., Fletcher A., Tabrizi S.N.…Wark J.D. Web-based recruiting for Health Research using a social networking site: an exploratory study. J. Med. Internet Res. 2012;14(1)
    1. Gosling S.D., Mason W. Internet research in psychology. Annu. Rev. Psychol. 2015;66(1):877–902.
    1. Gosling S.D., Vazire S., Srivastava S., John O.P. Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. Am. Psychol. 2004;59(2):93–104.
    1. Graham A.L., Fang Y., Moreno J.L., Streiff S.L., Villegas J., Muñoz R.F.…Vallone D.M. Online advertising to reach and recruit Latino smokers to an internet cessation program: impact and costs. J. Med. Internet Res. 2012;14(4)
    1. Gross M.S., Liu N.H., Contreras O., Muñoz R.F., Leykin Y. Using Google AdWords for international multilingual recruitment to Health Research websites. J. Med. Internet Res. 2014;16(1)
    1. Hoerger M. Participant dropout as a function of survey length in internet-mediated university studies: implications for study design and voluntary participation in psychological research. Cyberpsychol. Behav. Soc. Netw. 2010;13(6):697–700.
    1. Kayrouz R., Dear B.F., Karin E., Titov N. Facebook as an effective recruitment strategy for mental health research of hard to reach populations. Internet Interventions. 2016;4:1–10.
    1. Kroenke K., Spitzer R.L. The PHQ-9: A new depression diagnostic and severity measure. Psychiatr. Ann. 2002;32:509–515.
    1. Leykin Y., Muñoz R.F., Contreras O., Latham M.D. Results from a trial of an unsupported internet intervention for depressive symptoms. Internet Intervent. 2014;1(4):175–181.
    1. Mason W., Suri S. Conducting behavioral research on Amazon's Mechanical Turk. Behav. Res. Methods. 2012;44(1):1–23.
    1. Miller P.G., Sønderlund A.L. Using the internet to research hidden populations of illicit drug users: a review. Addiction. 2010;105(9):1557–1567.
    1. Muñoz R.F. Using evidence-based internet interventions to reduce health disparities worldwide. J. Med. Internet Res. 2010;12(5)
    1. Nosek B.A., Banaji M.R., Greenwald A.G. E-research: ethics, security, design, and control in psychological research on the internet. J. Soc. Issues. 2002;58(1):161–176.
    1. Patel M.X. Challenges in recruitment of research participants. Adv. Psychiatr. Treat. 2003;9(3):229–238.
    1. Paolacci G., Chandler J., Ipeirotis P.G. Running experiments on Amazon Mechanical Turk. Judgm. Decis. Mak. 2010;5(5):411–419.
    1. Prince K.R., Litovsky A.R., Friedman-Wheeler D.G. Research forum-internet-mediated research: beware of bots. Behav. Therapist. 2012;35(5):85.
    1. Ramo D.E., Prochaska J.J. Broad reach and targeted recruitment using Facebook for an online survey of young adult substance use. J. Med. Internet Res. 2012;14(1)
    1. Richards D., Richardson T., Timulak L., McElvaney J. The efficacy of internet-delivered treatment for generalized anxiety disorder: a systematic review and meta-analysis. Internet Intervent. 2015;2(3):272–282.
    1. Ritterband L.M., Tate D.F. The science of internet interventions. Ann. Behav. Med. 2009;38(1):1–3.
    1. Ryan G.S. Online social networks for patient involvement and recruitment in clinical research. Nurs. Res. 2013;21(1):35–39.
    1. Shapiro D.N., Chandler J., Mueller P.A. Using Mechanical Turk to study clinical populations. Clin. Psychol. Sci. 2013;1(2):213–220.
    1. Spitzer R.L., Kroenke K., Williams J.B., Löwe B. A brief measure for assessing generalized anxiety disorder: The GAD-7. Arch. Intern. Med. 2006;166:1092–1097.
    1. Whitaker C., Stevelink S., Fear N. The use of Facebook in recruiting participants for Health Research purposes: a systematic review. J. Med. Internet Res. 2017;19(8)

Source: PubMed

Patrocinadores e Colaboradores

Condições médicas

Intervenções de drogas

3
Se inscrever