Impact of question order on prioritisation of outcomes in the development of a core outcome set: a randomised controlled trial

Sara T Brookes, Katy A Chalmers, Kerry N L Avery, Karen Coulman, Jane M Blazeby, ROMIO study group, C Paul Barham, Richard Berrisford, Jenny Donovan, Jackie Elliott, Stephen Falk, Robert Goldin, George Hanna, Andrew Hollowood, Sian Noble, Grant Sanders, Tim Wheatley, Sara T Brookes, Katy A Chalmers, Kerry N L Avery, Karen Coulman, Jane M Blazeby, ROMIO study group, C Paul Barham, Richard Berrisford, Jenny Donovan, Jackie Elliott, Stephen Falk, Robert Goldin, George Hanna, Andrew Hollowood, Sian Noble, Grant Sanders, Tim Wheatley

Abstract

Background: Core outcome set (COS) developers increasingly employ Delphi surveys to elicit stakeholders' opinions of which outcomes to measure and report in trials of a particular condition or intervention. Research outside of Delphi surveys and COS development demonstrates that question order can affect response rates and lead to 'context effects', where prior questions determine an item's meaning and influence responses. This study examined the impact of question order within a Delphi survey for a COS for oesophageal cancer surgery.

Methods: A randomised controlled trial was nested within the Delphi survey. Patients and health professionals were randomised to receive a survey including clinical and patient-reported outcomes (PROs), where the PRO section appeared first or last. Participants rated (1-9) the importance of 68 items for inclusion in a COS (ratings 7-9 considered 'essential'). Analyses considered the impact of question order on: (1) survey response rates; (2) participants' responses; and (3) items retained at end of the survey.

Results: In total, 116 patients and 71 professionals returned completed surveys. Question order did not affect response rates among patients, but fewer professionals responded when clinical items appeared first (difference = 31.3%, 95% confidence interval [CI] = 13.6-48.9%, P = 0.001). Question order led to different context effects within patients and professionals. While patients rated clinical items highly, irrespective of question order, more PROs were rated essential when appearing last rather than first (difference = 23.7%, 95% CI = 10.5-40.8%). Among professionals, the greatest impact was on clinical items; a higher percentage rated essential when appearing last (difference = 11.6%, 95% CI = 0.0-23.3%). An interaction between question order and the percentage of PRO/clinical items rated essential was observed for patients (P = 0.025) but not professionals (P = 0.357). Items retained for further consideration at the end of the survey were dependent on question order, with discordant items (retained by one question order group only) observed in patients (18/68 [26%]) and professionals (20/68 [29%]).

Conclusions: In the development of a COS, participants' ratings of potential outcomes within a Delphi survey depend on the context (order) in which the outcomes are asked, consequently impacting on the final COS. Initial piloting is recommended with consideration of the randomisation of items in the survey to reduce potential bias.

Trial registration: The randomised controlled trial reported within this paper was nested within the development of a core outcome set to investigate processes in core outcome set development. Outcomes were not health-related and trial registration was not therefore applicable.

Keywords: Context effects; Core outcome set; Delphi; Question order.

Conflict of interest statement

Ethics approval and consent to participate

Ethical approval for this study was granted by the South-West – Frenchay Research Ethics Committee (12/SW/0161). All patients were informed about the study and invited to participate. Only those returning a completed consent form were then posted a round 1 questionnaire. For professionals, a round 1 questionnaire was provided with initial study information and invitation to participate; in this instance the return of a completed questionnaire was deemed consent to participate, as agreed by the local ethics committee.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures

Fig. 1
Fig. 1
Flow diagram
Fig. 2
Fig. 2
Patients: percentage of items rated essential within the non-comparative and comparative context (a consistency effect)
Fig. 3
Fig. 3
Health professionals: percentage of items rated essential within the non-comparative and comparative context (a contrast effect)

References

    1. Williamson PR, Gamble C, Altman DG, Hutton JL. Outcome selection bias in meta-analysis. Stat Methods Med Res. 2005;14(5):515–24. doi: 10.1191/0962280205sm415oa.
    1. Clarke M. Standardising outcomes for clinical trials and systematic reviews. Trials. 2007;8(1):39. doi: 10.1186/1745-6215-8-39.
    1. Kirkham JJ, Gargon E, Clarke M, Williamson PR. Can a core outcome set improve the quality of systematic reviews?–a survey of the Co-ordinating Editors of Cochrane Review Groups. Trials. 2013;14(1):21. doi: 10.1186/1745-6215-14-21.
    1. Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13(1):132. doi: 10.1186/1745-6215-13-132.
    1. Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med. 2011;8(1):e1000393. doi: 10.1371/journal.pmed.1000393.
    1. Gorst SL, Gargon E, Clarke M, Blazeby JM, Altman DG, Williamson PR. Choosing important health outcomes for comparative effectiveness research: an updated review and user survey. PLoS One. 2016;11(1):e0146444. doi: 10.1371/journal.pone.0146444.
    1. COMET (Core Outcome Measures in Effectiveness Trials) Initiative. . Accessed 07 Feb 2017.
    1. Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Manag Sci. 1963;9(3):458–67. doi: 10.1287/mnsc.9.3.458.
    1. McNair AG, Whistance RN, Forsythe RO, Macefield R, Rees J, Pullyblank AM, et al. Core outcomes for colorectal cancer surgery: a consensus study. PLoS Med. 2016;13(8):e1002071. doi: 10.1371/journal.pmed.1002071.
    1. Coulman KD, Hopkins J, Brookes ST, Chalmers K, Main B, Owen-Smith A, et al. A core outcome set for the benefits and adverse events of bariatric and metabolic surgery: the BARIACT project. PLoS Med. 2016;13(11):e1002187. doi: 10.1371/journal.pmed.1002187.
    1. Harman NL, Bruce IA, Kirkham JJ, Tierney S, Callery P, O’Brien K, et al. The importance of integration of stakeholder views in core outcome set development: otitis media with effusion in children with cleft palate. PLoS One. 2015;10(6):e0129514. doi: 10.1371/journal.pone.0129514.
    1. Brookes ST, Macefield RC, Williamson PR, McNair AG, Potter S, Blencowe NS, et al. Three nested randomized controlled trials of peer-only or multiple stakeholder group feedback within Delphi surveys during core outcome and information set development. Trials. 2016;17(1):409. doi: 10.1186/s13063-016-1479-x.
    1. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.
    1. Keeney S, Hasson F, McKenna HP. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud. 2001;38(2):195–200. doi: 10.1016/S0020-7489(00)00044-4.
    1. Hsu CC, Sandford BA. The Delphi technique: making sense of consensus. Pract Assess Res Eval. 2007;12(10):1–8.
    1. Hallowell MR, Gambatese JA. Qualitative research: application of the Delphi method to CEM research. J Constr Eng Manag. 2009;136(1):99–107. doi: 10.1061/(ASCE)CO.1943-7862.0000137.
    1. Landeta J. Current validity of the Delphi method in social sciences. Technol Forecast Soc Change. 2006;73(5):467–82. doi: 10.1016/j.techfore.2005.09.002.
    1. Sackman H. Delphi Critique. Lexington, MA: Lexington Books; 1975.
    1. Krosnick JA, Presser S. Question and questionnaire design. In: Marsden PV, Wright JD, editors. Handbook of survey research. 2. Bingley, UK: Emerald; 2010.
    1. McColl E, Jacoby A, Thomas L, Soutter J, Bamford C, Steen N, et al. Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health Technol Assess. 2001;5(31):1–256. doi: 10.3310/hta5310.
    1. Sudman S, Bradburn NM. Asking questions: a practical guide to questionnaire design. San Francisco, CA: Jossey-Bass; 1982.
    1. Jones WH, Lang JR. Sample composition bias and response bias in a mail survey: a comparison of inducement methods. J Market Res. 1980;26(1):69–76. doi: 10.2307/3151119.
    1. Roberson MT, Sundstrom E. Questionnaire design, return rates, and response favorableness in an employee attitude questionnaire. J Appl Psychol. 1990;75(3):354–7. doi: 10.1037/0021-9010.75.3.354.
    1. Nakash RA, Hutton JL, Jørstad-Stein EC, Gates S, Lamb SE. Maximising response to postal questionnaires–a systematic review of randomised trials in health research. BMC Med Res Methodol. 2006;6(1):5. doi: 10.1186/1471-2288-6-5.
    1. Moore DW. Measuring new types of question-order effects: additive and subtractive. Public Opin Q. 2002;66(1):80–91. doi: 10.1086/338631.
    1. Schuman H, Presser S. Questions and answers in attitude surveys: experiments on question form, wording, and context. Thousand Oaks, CA: Sage; 1996.
    1. Lasorsa DL. Question-order effects in surveys: the case of political interest, news attention, and knowledge. J Mass Commun Q. 2003;80(3):499–512.
    1. DeMoranville CW, Bienstock CC. Question order effects in measuring service quality. Int J Res Mark. 2003;20(3):217–31. doi: 10.1016/S0167-8116(03)00034-X.
    1. McFarland SG. Effects of question order on survey responses. Public Opin Q. 1981;45(2):208–15. doi: 10.1086/268651.
    1. Schuman H, Presser S, Ludwig J. Context effects on survey responses to questions about abortion. Public Opin Q. 1981;45(2):216–23. doi: 10.1086/268652.
    1. Tenvergert E, Gillespie MW, Kingma J, Klasen H. Abortion attitudes, 1984-1987-1988: effects of item order and dimensionality. Percept Mot Skills. 1992;74(2):627–42. doi: 10.2466/pms.1992.74.2.627.
    1. Avery KNL, Chalmers KA, Brookes ST, Blencowe NS, Coulman K, Whale K, et al. Development of a core outcome set for clinical effectiveness trials in esophageal cancer resection surgery. Ann Surg. 2017;15:10.1097/SLA.0000000000002204.
    1. Blencowe NS, Strong S, McNair AG, Brookes ST, Crosby T, Griffin SM, et al. Reporting of short-term clinical outcomes after esophagectomy: a systematic review. Ann Surg. 2012;255:658–66. doi: 10.1097/SLA.0b013e3182480a6a.
    1. Macefield RC, Jacobs M, Korfage IJ, Nicklin J, Whistance RN, Brookes ST, et al. Developing core outcome sets: methods for identifying and including patient-reported outcomes (PROs) Trials. 2014;15:49. doi: 10.1186/1745-6215-15-49.
    1. The Royal College of Surgeons of England. National Oesophago-Gastric Audit 2013 [NHS website]. Available at: . Accessed 7 Sept 2016.
    1. McNair AG, MacKichan F, Donovan JL, Brookes ST, Avery KNL, Griffin SM, et al. What surgeons tell patients and what patients want to know before major cancer surgery: a qualitative study. BMC Cancer. 2016;16:258–65. doi: 10.1186/s12885-016-2292-3.
    1. StataCorp . Stata Statistical Software: Release 14. College Station, TX: StataCorp LP; 2015.
    1. Altman DG. Practical statistics for medical research. London: Chapman and Hall; 1991.
    1. Tourangeau R, Rasinski KA, Bradburn N, D’Andrade R. Belief accessibility and context effects in attitude measurement. J Exp Soc Psychol. 1989;25(5):401–21. doi: 10.1016/0022-1031(89)90030-9.
    1. Bickart BA. Carryover and backfire effects in marketing research. J Market Res. 1993;30(1):52. doi: 10.2307/3172513.
    1. Coulman KD, Howes N, Hopkins J, Whale K, Chalmers K, Brookes S, et al. A comparison of health professionals’ and patients’ views of the importance of outcomes of bariatric surgery. Obes Surg. 2016;26(11):2738–46. doi: 10.1007/s11695-016-2186-0.
    1. Potter S, Brookes ST, Holcombe C, Ward JA, Blazeby JM. Exploring methods for the selection and integration of stakeholder views in the development of core outcome sets: a case study in reconstructive breast surgery. Trials. 2016;17:463. doi: 10.1186/s13063-016-1591-y.
    1. Wills GB. Cognitive interviewing: a tool for improving questionnaire design. Thousand Oaks, CA: Sage; 2005.
    1. Schwarz N, Schuman H. Political knowledge, attribution, and inferred interest in politics: the operation of buffer items. Int J Public Opinion Res. 1997;9(2):191–5. doi: 10.1093/ijpor/9.2.191.
    1. Perreault WD. Controlling order-effect bias. Public Opin Q. 1975;39(4):544–51. doi: 10.1086/268251.

Source: PubMed

Sponsor e collaboratori

Condizioni mediche

Interventi farmacologici

3
Sottoscrivi