Developing questionnaires for educational research: AMEE Guide No. 87

Anthony R Artino Jr, Jeffrey S La Rochelle, Kent J Dezee, Hunter Gehlbach, Anthony R Artino Jr, Jeffrey S La Rochelle, Kent J Dezee, Hunter Gehlbach

Abstract

In this AMEE Guide, we consider the design and development of self-administered surveys, commonly called questionnaires. Questionnaires are widely employed in medical education research. Unfortunately, the processes used to develop such questionnaires vary in quality and lack consistent, rigorous standards. Consequently, the quality of the questionnaires used in medical education research is highly variable. To address this problem, this AMEE Guide presents a systematic, seven-step process for designing high-quality questionnaires, with particular emphasis on developing survey scales. These seven steps do not address all aspects of survey design, nor do they represent the only way to develop a high-quality questionnaire. Instead, these steps synthesize multiple survey design techniques and organize them into a cohesive process for questionnaire developers of all levels. Addressing each of these steps systematically will improve the probabilities that survey designers will accurately measure what they intend to measure.

Figures

Figure 1
Figure 1
Visual-design “best practices” based on scientific evidence from questionnaire design research.

References

    1. American Educational Research Association (AERA), American Psychological Association (APA) & National Council on Measurement in Education (NCME) Standards for education and psychological testing. Washington, DC: American Educational Research Association; 1999.
    1. Artino AR, Gehlbach H, Durning SJ. AM last page: Avoiding five common pitfalls of survey design. Acad Med. 2011;86:1327.
    1. Artino AR, Gehlbach H. AM last page: Avoiding four visual-design pitfalls in survey development. Acad Med. 2012;87:1452.
    1. Beck CT, Gable RK. Ensuring content validity: An illustration of the process. J Nurs Meas. 2001;9:201–215.
    1. Christian LM, Parsons NL, Dillman DA. 2009. Designing scalar questions for web surveys. Sociol Method Res 37:393–425.
    1. Colliver JA, Conlee MJ, Verhulst SJ, Dorsey JK. Reports of the decline of empathy during medical education are greatly exaggerated: A reexamination of the research. Acad Med. 2010;85:588–593.
    1. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: Theory and application. Am J Med. 2006;119:166.e7–166.e16.
    1. DeVellis RF. 2003. Scale development: Theory and applications. 2nd ed. Newbury Park, CA: Sage.
    1. Dillman D, Smyth J, Christian L. Internet, mail, and mixed-mode surveys: The tailored design method. 3rd. Hoboken, NJ: Wiley; 2009.
    1. Drennan J. Cognitive interviewing: Verbal data in the design and pretesting of questionnaires. J Adv Nurs. 2003;42(1):57–63.
    1. Fabrigar LR, Wegener DT. 2012. Exploratory factor analysis. New York: Oxford University Press.
    1. Fowler FJ. 2009. Survey research methods. 4th ed. Thousand Oaks, CA: Sage.
    1. Gehlbach H, Artino AR, Durning S. AM last page: Survey development guidance for medical education researchers. Acad Med. 2010;85:925.
    1. Gehlbach H, Brinkworth ME. Measure twice, cut down error: A process for enhancing the validity of survey scales. Rev Gen Psychol. 2011;15:380–387.
    1. Kane MT. Validation in educational measurement. 4th. Westport, CT: American Council on Education/Praeger; 2006.
    1. Karabenick SA, Woolley ME, Friedel JM, Ammon BV, Blazevski J, Bonney CR, De Groot E, Gilbert MC, Musu L, Kempler TM, Kelly KL. Cognitive processing of self-report items in educational research: Do they think what we mean? Educ Psychol. 2007;42(3):139–151.
    1. Krosnick JA. 1999. Survey research. Annu Rev Psychol 50:537–567.
    1. Magee C, Byars L, Rickards G, Artino AR. Tracing the steps of survey design: A graduate medical education research example. J Grad Med Educ. 2013;5(1):1–5.
    1. McCoach DB, Gable RK, Madura JP. Instrument development in the affective domain: School and corporate applications. 3rd. New York: Springer; 2013.
    1. McIver JP, Carmines EG. Unidimensional scaling. Beverly Hills, CA: Sage; 1981.
    1. McKenzie JF, Wood ML, Kotecki JE, Clark JK, Brey RA. Establishing content validity: Using qualitative and quantitative steps. Am J Health Behav. 1999;23(4):311–318.
    1. Napoles-Springer AM, Olsson-Santoyo J, O’Brien H, Stewart AL. Using cognitive interviews to develop surveys in diverse populations. Med Care. 2006;44(11):s21–s30.
    1. Pett MA, Lackey NR, Sullivan JJ. Making sense of factor analysis: The use of factor analysis for instrument development in health care research. Thousand Oaks, CA: Sage Publications; 2003.
    1. Polit DF, Beck CT. Nursing research: Principles and methods. 7th. Philadelphia: Lippincott, Williams, & Wilkins; 2004.
    1. Polit DF, Beck CT. The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Res Nurs Health. 2006;29:489–497.
    1. Rickards G, Magee C, Artino AR. You can’t fix by analysis what you’ve spoiled by design: developing survey instruments and collecting validity evidence. J Grad Med Educ. 2012;4(4):407–410.
    1. Rubio DM, Berg-Weger M, Tebb SS, Lee ES, Rauch S. Objectifying content validity: Conducting a content validity study in social work research. Soc Work Res. 2003;27(2):94–104.
    1. Schmitt N. Uses and abuses of coefficient alpha. Psychol Assess. 1996;8:350–353.
    1. Schwarz N. 1999. Self-reports: How the questions shape the answers. Am Psychol 54:93–105.
    1. Sullivan G. A primer on the validity of assessment instruments. J Grad Med Educ. 2011;3(2):119–120.
    1. Sullivan GM, Artino AR. Analyzing and interpreting data from Likert-type scales. J Grad Med Educ. 2013;5(4):541–542.
    1. Tourangeau R, Rips LJ, Rasinski KA. The psychology of survey response. New York: Cambridge University Press; 2000.
    1. Waltz CF, Strickland OL, Lenz ER. Measurement in nursing and health research. 3rd. New York: Springer Publishing Co; 2005.
    1. Watt T, Rasmussen AK, Groenvold M, Bjorner JB, Watt SH, Bonnema SJ, Hegedus L, Feldt-Rasmussen U. Improving a newly developed patient-reported outcome for thyroid patients, using cognitive interviewing. Quality of Life Research. 2008;17:1009–1017.
    1. Weng LJ. 2004. Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educ Psychol Meas 64:956–972.
    1. Willis GB, Artino AR. What do our respondents think we’re asking? Using cognitive interviewing to improve medical education surveys. J Grad Med Educ. 2013;5(3):353–356.
    1. Willis GB. Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage Publications; 2005.

Source: PubMed

Sponsors en medewerkers

Medische omstandigheden

Geneesmiddelinterventies

3
Abonneren