A usability study of two formats of a shortened systematic review for clinicians

Laure Perrier, M Ryan Kealey, Sharon E Straus, Laure Perrier, M Ryan Kealey, Sharon E Straus

Abstract

Objective: The aim of this study was to evaluate the usability of two formats of a shortened systematic review for clinicians.

Materials and methods: Usability of the prototypes was assessed using three cycles of iterative testing. 10 participants were asked to complete tasks of locating information or items within two prototypes and 'think aloud' while being audio taped. Interviews were also audio recorded and participants completed a systematic usability scale.

Results: Revisions were made between each iteration in order to address issues identified by participants. Finding information relating to the number of studies in the meta-analysis, and locating the number of studies in the entire systematic review were revealed as areas needing attention during the usability evaluation.

Conclusions: Iterative testing combined with a multifaceted approach to usability testing offered essential insight into aspects of the prototypes that required modifications. Alterations were made in order to create finalised versions of the two shortened systematic review formats.

Keywords: Evidence Based Practice; PRIMARY CARE; Review Literature as Topic.

Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

References

    1. Cochrane Collaboration. What are Cochrane Reviews? (accessed 7 Apr 2014).
    1. Guyatt GH, Sackett DL, Sinclair JC et al. . ‘Users’ guides to the medical literature: IX. A method for grading health care recommendations. Evidence-Based Medicine Working Group . JAMA 1995;274:1800–4. 10.1001/jama.1995.03530220066035
    1. Laupacis A, Straus S. Systematic reviews: time to address clinical and policy relevance as well as methodological rigor. Ann Intern Med 2007;147:273–4. 10.7326/0003-4819-147-4-200708210-00180
    1. De Vito C, Nobile CG, Furnari G et al. . Physicians’ knowledge, attitudes and professional use of RCTs and meta-analyses: a cross-sectional survey. Eur J Public Health 2009;19:297–302. 10.1093/eurpub/ckn134
    1. Ely JW, Osheroff JA, Ebell MH et al. . Obstacles to answering doctors’ questions about patient care with evidence: qualitative study. BMJ 2002;324:710 10.1136/bmj.324.7339.710
    1. Ely JW, Osheroff JA, Chambliss ML et al. . Answering physicians’ clinical questions: obstacles and potential solutions. J Am Med Inform Assoc 2005;12:217–24. 10.1197/jamia.M1608
    1. Fozi K, Teng CL, Krishnan R et al. . A study of clinical questions in primary care. Med J Malaysia 2000;55:486–92.
    1. D'Alessandro DM, Kreiter CD, Peterson MW. An evaluation of information seeking behaviors of general pediatricians. Pediatrics 2004;113:64–9. 10.1542/peds.113.1.64
    1. Coumou HC, Meijman FJ. How do primary care physicians seek answers to clinical questions? A literature review. J Med Libr Assoc 2006;94:55–60.
    1. Andrews JE, Pearce KA, Ireson C et al. . Information-seeking behaviors of practitioners in a primary care practice-based research network (PBRN). J Med Libr Assoc 2005;93:206–12.
    1. Perrier L, Mrklas K, Shepperd S et al. . Interventions encouraging the use of systematic reviews in clinical decision-making: a systematic review. J Gen Intern Med 2011;26:419–26. 10.1007/s11606-010-1506-7
    1. Rosenbaum SE, Glenton C, Nylund HK et al. . User testing and stakeholder feedback contributed to the development of understandable and useful summary of findings tables for Cochrane reviews. J Clin Epidemiol 2010;63:607–19. 10.1016/j.jclinepi.2009.12.013
    1. Rosenbaum SE, Glenton C, Oxman AD. Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information. J Clin Epidemiol 2010;63:620–6. 10.1016/j.jclinepi.2009.12.014
    1. Perrier L, Persaud N, Ko A et al. . Development of two shortened systematic review formats for clinicians. Implement Sci 2013;8:68 10.1186/1748-5908-8-68
    1. Perrier L, Kealey MK, Straus SE. An iterative evaluation of two shortened systematic review formats for clinicians: a focus group study. J Am Med Inform Assoc 2014;21(e2):e341–6. 10.1136/amiajnl-2014-002660
    1. Atkinson RA, Derry SJ, Renkl A et al. . Learning from examples: Instructional principles from worked examples research. Rev Educ Res 2000;70:181–214. 10.3102/00346543070002181
    1. Lee AY, Hutchison L. Improving learning from examples through reflection. J Exp Psychol Appl 1998;4:187–210. 10.1037/1076-898X.4.3.187
    1. Robertson I, Kahney H. The use of examples in expository texts: outline of an interpretation theory for text analysis. Instruc Sci 1996;24:93–123. 10.1007/BF00120485
    1. Moreno R, Mayer RE. Engaging students in active learning: the case for personalized multi-media messages. J Educ Psych 2000;92:724–33. 10.1037/0022-0663.92.4.724
    1. Czuchry M, Dansereau DF. The generation and recall of personally relevant information. J Exp Info 1998;66:293–315.
    1. Thomas JC.. Story-based mechanisms of tacit knowledge transfer. 2002. ECSCW 2001 Workshop on Managing Tacit Knowledge.
    1. Sackett DL, Rosenberg WM, Gray JA et al. . Evidence based medicine: what it is and what it isn't. BMJ 1996;312:71–2. 10.1136/bmj.312.7023.71
    1. El Dib RP, Atallah AN, Andriolo RB. Mapping the Cochrane evidence for decision making in health care. J Eval Clin Pract 2007;13:689–92. 10.1111/j.1365-2753.2007.00886.x
    1. Lottridge DM, Chignell M, Danicic-Mizdrak R et al. . Group differences in physician responses to handheld presentation of clinical evidence: a verbal protocol analysis. BMC Med Inform Decis Mak 2007;7:22 10.1186/1472-6947-7-22
    1. Health Information Research Unit. McMaster PLUS. (accessed 15 Feb 2013).
    1. van Zuuren EJ, Gupta AK, Gover MD et al. . Systematic review of rosacea treatments. J Am Acad Dermatol 2007;56:107–15.
    1. U.S. Department of Health & Human Services. Glossary. (accessed 7 Apr 2014).
    1. Biernacki P, Waldorf D. Snowball sampling: problems and techniques of chain referral sampling. Sociol Methods Res 1981;10:141–63.
    1. Snodgrass A, Coyne R. Models, metaphors and the hermeneuticals of designing. Des Issues 1992;9:56–74. 10.2307/1511599
    1. Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. Proceedings of ACM INTERCHI'93 Conference; Amsterdam, The Netherlands, 24–29 April 1993:206–13.
    1. MySQL. Mathematical Functions. (accessed 7 Apr 2014).
    1. Brooke J. SUS—a quick and dirty usability scale 1986. (accessed 7 Feb 2014).
    1. System Usability Scale. (accessed 7 Apr 2014).
    1. Kushniruk AW, Triola MM, Borycki EM et al. . Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform 2005;74:519–26. 10.1016/j.ijmedinf.2005.01.003
    1. Lincoln Y, Guba EG. Naturalistic inquiry. Newbury Park, CA: Sage Publications, 1985.
    1. Nelson AM. Addressing the threat of evidence-based practice to qualitative inquiry though increasing attention to quality: a discussion paper. Int J Nurs Stud 2008;45:316–22. 10.1016/j.ijnurstu.2007.01.012
    1. Mays N, Pope C. Qualitative research in health care. Assessing quality in qualitative research. BMJ 2000;320:50–2. 10.1136/bmj.320.7226.50
    1. Wegwarth O, Schwartz LM, Woloshin S et al. . Do physicians understand cancer screening statistics? A national survey of primary care physicians in the United States. Ann Intern Med 2012;156:340–9. 10.7326/0003-4819-156-5-201203060-00005
    1. Baldwin LM, Trivers KF, Matthews B et al. . Vignette-based study of ovarian cancer screening: do U.S. physicians report adhering to evidence-based recommendations? Ann Intern Med 2012;156:182–94. 10.7326/0003-4819-156-3-201202070-00006
    1. Novack L, Jotkowitz A, Knyazer B et al. . Evidence-based medicine: assessment of knowledge of basic epidemiological and research methods among medical doctors. Postgrad Med J 2006;82:817–22. 10.1136/pgmj.2006.049262
    1. Raina PS, Brehaut JC, Platt RW et al. . The influence of display and statistical factors on the interpretation of metaanalysis results by physicians. Med Care 2005;43:1242–9. 10.1097/01.mlr.0000185710.62071.7f
    1. Nielsen J. How to conduct a heuristic evaluation. (accessed 7 Apr 2014).
    1. Gustafson DH, Wyatt JC. Evaluation of ehealth systems and services. BMJ 2004;328:1150 10.1136/bmj.328.7449.1150
    1. Kushniruk A. Evaluation in the design of health information systems: application of approaches emerging from usability engineering. Comput Biol Med 2002;32:141–9. 10.1016/S0010-4825(02)00011-2

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe