Delphi methodology in healthcare research: How to decide its appropriateness

Prashant Nasa, Ravi Jain, Deven Juneja, Prashant Nasa, Ravi Jain, Deven Juneja

Abstract

The Delphi technique is a systematic process of forecasting using the collective opinion of panel members. The structured method of developing consensus among panel members using Delphi methodology has gained acceptance in diverse fields of medicine. The Delphi methods assumed a pivotal role in the last few decades to develop best practice guidance using collective intelligence where research is limited, ethically/logistically difficult or evidence is conflicting. However, the attempts to assess the quality standard of Delphi studies have reported significant variance, and details of the process followed are usually unclear. We recommend systematic quality tools for evaluation of Delphi methodology; identification of problem area of research, selection of panel, anonymity of panelists, controlled feedback, iterative Delphi rounds, consensus criteria, analysis of consensus, closing criteria, and stability of the results. Based on these nine qualitative evaluation points, we assessed the quality of Delphi studies in the medical field related to coronavirus disease 2019. There was inconsistency in reporting vital elements of Delphi methods such as identification of panel members, defining consensus, closing criteria for rounds, and presenting the results. We propose our evaluation points for researchers, medical journal editorial boards, and reviewers to evaluate the quality of the Delphi methods in healthcare research.

Keywords: Consensus; Coronavirus disease 2019; Delphi studies; Delphi technique; Expert panel; Quality tools for methodology; Research methods; SARS-CoV-2.

Conflict of interest statement

Conflict-of-interest statement: Prashant Nasa declared to be on the advisory board of Edwards life sciences. Other authors do not declare any conflict of interest.

©The Author(s) 2021. Published by Baishideng Publishing Group Inc. All rights reserved.

Figures

Figure 1
Figure 1
Stepwise quality assessment of Delphi studies.
Figure 2
Figure 2
Stability assessment for Delphi rounds.

References

    1. Linstone HA, Turoff M. The Delphi Method: Techniques and Applications, 1975. Available from: .
    1. Dalkey N, Helmer O. An Experimental Application of the DELPHI Method to the Use of Experts. Manag Sci. 1963;9:458–467.
    1. Goodman CM. The Delphi technique: a critique. J Adv Nurs. 1987;12:729–734.
    1. McGrath BA, Brenner MJ, Warrillow SJ, Pandian V, Arora A, Cameron TS, Añon JM, Hernández Martínez G, Truog RD, Block SD, Lui GCY, McDonald C, Rassekh CH, Atkins J, Qiang L, Vergez S, Dulguerov P, Zenk J, Antonelli M, Pelosi P, Walsh BK, Ward E, Shang Y, Gasparini S, Donati A, Singer M, Openshaw PJM, Tolley N, Markel H, Feller-Kopman DJ. Tracheostomy in the COVID-19 era: global and multidisciplinary guidance. Lancet Respir Med. 2020;8:717–725.
    1. Jünger S, Payne S, Brearley S, Ploenes V, Radbruch L. Consensus building in palliative care: a Europe-wide delphi study on common understandings and conceptual differences. J Pain Symptom Manage. 2012;44:192–205.
    1. Vogel C, Zwolinsky S, Griffiths C, Hobbs M, Henderson E, Wilkins E. A Delphi study to build consensus on the definition and use of big data in obesity research. Int J Obes (Lond) 2019;43:2573–2586.
    1. Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, Bellomo R, Bernard GR, Chiche JD, Coopersmith CM, Hotchkiss RS, Levy MM, Marshall JC, Martin GS, Opal SM, Rubenfeld GD, van der Poll T, Vincent JL, Angus DC. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) JAMA. 2016;315:801–810.
    1. Han H, Ahn DH, Song J, Hwang TY, Roh S. Development of mental health indicators in Korea. Psychiatry Investig. 2012;9:311–318.
    1. van Hasselt FM, Oud MJ, Loonen AJ. Practical recommendations for improvement of the physical health care of patients with severe mental illness. Acta Psychiatr Scand. 2015;131:387–396.
    1. Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, Wales PW. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67:401–409.
    1. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6:e20476.
    1. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017;31:684–706.
    1. Green B, Jones M, Hughes D, Williams A. Applying the Delphi technique in a study of GPs' information requirements. Health Soc Care Community. 1999;7:198–205.
    1. Robba C, Poole D, Citerio G, Taccone FS, Rasulo FA Consensus on brain ultrasonography in critical care group. Brain Ultrasonography Consensus on Skill Recommendations and Competence Levels Within the Critical Care Setting. Neurocrit Care. 2020;32:502–511.
    1. Niederberger M, Spranger J. Delphi Technique in Health Sciences: A Map. Front Public Health. 2020;8:457.
    1. Santaguida P, Dolovich L, Oliver D, Lamarche L, Gilsing A, Griffith LE, Richardson J, Mangin D, Kastner M, Raina P. Protocol for a Delphi consensus exercise to identify a core set of criteria for selecting health related outcome measures (HROM) to be used in primary health care. BMC Fam Pract. 2018;19:152.
    1. Dunn WN. Public policy analysis: an introduction. Upper Saddle River: Pearson Prentice Hall, 2004.
    1. Mitchell VW. The Delphi technique: an exposition and application. Technol Anal Strateg Manag. 1991;3:333–358.
    1. Trevelyan E, Robinson N. Delphi methodology in health research: How to do it? Eur J Integr Med . 2015;7:423–428.
    1. Grant S, Booth M, Khodyakov D. Lack of preregistered analysis plans allows unacceptable data mining for and selective reporting of consensus in Delphi studies. J Clin Epidemiol. 2018;99:96–105.
    1. Dajani JS, Sincoff MZ, Talley WK. Stability and agreement criteria for the termination of Delphi studies. Technol Forecast Soc Change. 1979;13:83–90.
    1. Holey EA, Feeley JL, Dixon J, Whittaker VJ. An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Med Res Methodol. 2007;7:52.
    1. Chaffin WW, Talley WK. Individual stability in Delphi studies. Technol Forecast Soc Change. 1980;16:67–73.
    1. Vitacca M, Lazzeri M, Guffanti E, Frigerio P, D'Abrosca F, Gianola S, Carone M, Paneroni M, Ceriana P, Pasqua F, Banfi P, Gigliotti F, Simonelli C, Cirio S, Rossi V, Beccaluva CG, Retucci M, Santambrogio M, Lanza A, Gallo F, Fumagalli A, Mantero M, Castellini G, Calabrese M, Castellana G, Volpato E, Ciriello M, Garofano M, Clini E, Ambrosino N, Arir Associazione Riabilitatori dell'Insufficienza Respiratoria Sip Società Italiana di Pneumologia Aifi Associazione Italiana Fisioterapisti And Sifir Società Italiana di Fisioterapia E Riabilitazione OBOAAIPO. Italian suggestions for pulmonary rehabilitation in COVID-19 patients recovering from acute respiratory failure: results of a Delphi process. Monaldi Arch Chest Dis. 2020;90
    1. Mikuls TR, Johnson SR, Fraenkel L, Arasaratnam RJ, Baden LR, Bermas BL, Chatham W, Cohen S, Costenbader K, Gravallese EM, Kalil AC, Weinblatt ME, Winthrop K, Mudano AS, Turner A, Saag KG. American College of Rheumatology Guidance for the Management of Rheumatic Disease in Adult Patients During the COVID-19 Pandemic: Version 1. Arthritis Rheumatol. 2020;72:1241–1251.
    1. Greenhalgh T, Thompson P, Weiringa S, Neves AL, Husain L, Dunlop M, Rushforth A, Nunan D, de Lusignan S, Delaney B. What items should be included in an early warning score for remote assessment of suspected COVID-19? BMJ Open. 2020;10:e042626.
    1. Lamb CR, Desai NR, Angel L, Chaddha U, Sachdeva A, Sethi S, Bencheqroun H, Mehta H, Akulian J, Argento AC, Diaz-Mendoza J, Musani A, Murgu S. Use of Tracheostomy During the COVID-19 Pandemic: American College of Chest Physicians/American Association for Bronchology and Interventional Pulmonology/Association of Interventional Pulmonology Program Directors Expert Panel Report. Chest. 2020;158:1499–1514.
    1. Welsh Surgical Research Initiative (WSRI) Collaborative. Surgery during the COVID-19 pandemic: operating room suggestions from an international Delphi process. Br J Surg. 2020;107:1450–1458.
    1. Eibensteiner F, Ritschl V, Ariceta G, Jankauskiene A, Klaus G, Paglialonga F, Edefonti A, Ranchin B, Schmitt CP, Shroff R, Stefanidis CJ, Walle JV, Verrina E, Vondrak K, Zurowska A, Stamm T, Aufricht C European Pediatric Dialysis Working Group. Rapid response in the COVID-19 pandemic: a Delphi study from the European Pediatric Dialysis Working Group. Pediatr Nephrol. 2020;35:1669–1678.
    1. Bhandari P, Subramaniam S, Bourke MJ, Alkandari A, Chiu PWY, Brown JF, Keswani RN, Bisschops R, Hassan C, Raju GS, Muthusamy VR, Sethi A, May GR, Albéniz E, Bruno M, Kaminski MF, Alkhatry M, Almadi M, Ibrahim M, Emura F, Moura E, Navarrete C, Wulfson A, Khor C, Ponnudurai R, Inoue H, Saito Y, Yahagi N, Kashin S, Nikonov E, Yu H, Maydeo AP, Reddy DN, Wallace MB, Pochapin MB, Rösch T, Sharma P, Repici A. Recovery of endoscopy services in the era of COVID-19: recommendations from an international Delphi consensus. Gut. 2020;69:1915–1924.
    1. Guckenberger M, Belka C, Bezjak A, Bradley J, Daly ME, DeRuysscher D, Dziadziuszko R, Faivre-Finn C, Flentje M, Gore E, Higgins KA, Iyengar P, Kavanagh BD, Kumar S, Le Pechoux C, Lievens Y, Lindberg K, McDonald F, Ramella S, Rengan R, Ricardi U, Rimner A, Rodrigues GB, Schild SE, Senan S, Simone CB 2nd, Slotman BJ, Stuschke M, Videtic G, Widder J, Yom SS, Palma D. Practice recommendations for lung cancer radiotherapy during the COVID-19 pandemic: An ESTRO-ASTRO consensus statement. Radiother Oncol. 2020;146:223–229.
    1. Aj B, C B, T A, Harper E R, Rl H, Rj E Welsh Surgical Research Initiative (WSRI) Collaborative. International surgical guidance for COVID-19: Validation using an international Delphi process - Cross-sectional study. Int J Surg. 2020;79:309–316.
    1. Gelfand JM, Armstrong AW, Bell S, Anesi GL, Blauvelt A, Calabrese C, Dommasch ED, Feldman SR, Gladman D, Kircik L, Lebwohl M, Lo Re V 3rd, Martin G, Merola JF, Scher JU, Schwartzman S, Treat JR, Van Voorhees AS, Ellebrecht CT, Fenner J, Ocon A, Syed MN, Weinstein EJ, Smith J, Gondo G, Heydon S, Koons S, Ritchlin CT. National Psoriasis Foundation COVID-19 Task Force Guidance for Management of Psoriatic Disease During the Pandemic: Version 1. J Am Acad Dermatol. 2020;83:1704–1716.
    1. Allan M, Mahawar K, Blackwell S, Catena F, Chand M, Dames N, Goel R, Graham YN, Kothari SN, Laidlaw L, Mayol J, Moug S, Petersen RP, Pryor AD, Smart NJ, Taylor M, Toogood GJ, Wexner SD, Zevin B, Wilson MS PRODUCE study. COVID-19 research priorities in surgery (PRODUCE study): A modified Delphi process. Br J Surg. 2020;107:e538–e540.
    1. Shanbehzadeh M, Kazemi-Arpanahi H, Mazhab-Jafari K, Haghiri H. Coronavirus disease 2019 (COVID-19) surveillance system: Development of COVID-19 minimum data set and interoperable reporting framework. J Educ Health Promot. 2020;9:203.
    1. Bergman C, Stall NM, Haimowitz D, Aronson L, Lynn J, Steinberg K, Wasserman M. Recommendations for Welcoming Back Nursing Home Visitors During the COVID-19 Pandemic: Results of a Delphi Panel. J Am Med Dir Assoc. 2020;21:1759–1766.
    1. Daigle P, Leung V, Yin V, Kalin-Hajdu E, Nijhawan N. Personal protective equipment (PPE) during the COVID-19 pandemic for oculofacial plastic and orbital surgery. Orbit. 2020:1–6.
    1. Sorbello M, Rosenblatt W, Hofmeyr R, Greif R, Urdaneta F. Aerosol boxes and barrier enclosures for airway management in COVID-19 patients: a scoping review and narrative synthesis. Br J Anaesth. 2020;125:880–894.
    1. Jheon S, Ahmed AD, Fang VW, Jung W, Khan AZ, Lee JM, Sihoe AD, Thongcharoen P, Tsuboi M, Turna A, Nakajima J. Thoracic cancer surgery during the COVID-19 pandemic: a consensus statement from the Thoracic Domain of the Asian Society for Cardiovascular and Thoracic Surgery. Asian Cardiovasc Thorac Ann. 2020;28:322–329.
    1. Olmos-Gómez MDC. Sex and Careers of University Students in Educational Practices as Factors of Individual Differences in Learning Environment and Psychological Factors during COVID-19. Int J Environ Res Public Health. 2020;17
    1. Sawhney MS, Bilal M, Pohl H, Kushnir VM, Khashab MA, Schulman AR, Berzin TM, Chahal P, Muthusamy VR, Varadarajulu S, Banerjee S, Ginsberg GG, Raju GS, Feuerstein JD. Triaging advanced GI endoscopy procedures during the COVID-19 pandemic: consensus recommendations using the Delphi method. Gastrointest Endosc. 2020;92:535–542.
    1. Sciubba DM, Ehresman J, Pennington Z, Lubelski D, Feghali J, Bydon A, Chou D, Elder BD, Elsamadicy AA, Goodwin CR, Goodwin ML, Harrop J, Klineberg EO, Laufer I, Lo SL, Neuman BJ, Passias PG, Protopsaltis T, Shin JH, Theodore N, Witham TF, Benzel EC. Scoring System to Triage Patients for Spine Surgery in the Setting of Limited Resources: Application to the Coronavirus Disease 2019 (COVID-19) Pandemic and Beyond. World Neurosurg. 2020;140:e373–e380.
    1. Errett NA, Howarth M, Shoaf K, Couture M, Ramsey S, Rosselli R, Webb S, Bennett A, Miller A. Developing an Environmental Health Sciences COVID-19 Research Agenda: Results from the NIEHS Disaster Research Response (DR2) Work Group's Modified Delphi Method. Int J Environ Res Public Health. 2020;17
    1. Arezzo A, Francis N, Mintz Y, Adamina M, Antoniou SA, Bouvy N, Copaescu C, de Manzini N, Di Lorenzo N, Morales-Conde S, Müller-Stich BP, Nickel F, Popa D, Tait D, Thomas C, Nimmo S, Paraskevis D, Pietrabissa A EAES Group of Experts for Recovery Amid COVID-19 Pandemic. EAES Recommendations for Recovery Plan in Minimally Invasive Surgery Amid COVID-19 Pandemic. Surg Endosc. 2021;35:1–17.
    1. Dashash M, Almasri B, Takaleh E, Halawah AA, Sahyouni A. Educational perspective for the identification of essential competencies required for approaching patients with COVID-19. East Mediterr Health J. 2020;26:1011–1017.
    1. Ramalho R, Adiukwu F, Gashi Bytyçi D, El Hayek S, Gonzalez-Diaz JM, Larnaout A, Grandinetti P, Nofal M, Pereira-Sanchez V, Pinto da Costa M, Ransing R, Teixeira ALS, Shalbafan M, Soler-Vidal J, Syarif Z, Orsolini L. Telepsychiatry During the COVID-19 Pandemic: Development of a Protocol for Telemental Health Care. Front Psychiatry. 2020;11:552450.
    1. Saldarriaga Rivera LM, Fernández Ávila D, Bautista Molano W, Jaramillo Arroyave D, Bautista Ramírez AJ, Díaz Maldonado A, Hernán Izquierdo J, Jáuregui E, Latorre Muñoz MC, Restrepo JP, Segura Charry JS. Recommendations on the management of adult patients with rheumatic diseases in the context of SARS-CoV-2/COVID-19 infection. Colombian Association of Rheumatology. Reumatol Clin. 2020;16:437–446.
    1. Tchouaket Nguemeleu E, Boivin S, Robins S, Sia D, Kilpatrick K, Brousseau S, Dubreuil B, Larouche C, Parisien N. Development and validation of a time and motion guide to assess the costs of prevention and control interventions for nosocomial infections: A Delphi method among experts. PLoS One. 2020;15:e0242212.
    1. Santana RF, Silva MBD, Marcos DADSR, Rosa CDS, Wetzel Junior W, Delvalle R. Nursing recommendations for facing dissemination of COVID-19 in Brazilian Nursing Homes. Rev Bras Enferm. 2020;73:e20200260.
    1. Tang Z, Sun B, Xu B. A quick evaluation method of nosocomial infection risk for cancer hospitals during the COVID-19 pandemic. J Cancer Res Clin Oncol. 2020;146:1891–1892.
    1. Jiménez-Rodríguez D, Ruiz-Salvador D, Rodríguez Salvador MDM, Pérez-Heredia M, Muñoz Ronda FJ, Arrogante O. Consensus on Criteria for Good Practices in Video Consultation: A Delphi Study. Int J Environ Res Public Health. 2020;17
    1. Reina Ortiz M, Grijalva MJ, Turell MJ, Waters WF, Montalvo AC, Mathias D, Sharma V, Renoy CF, Suits P, Thomas SJ, Leon R. Biosafety at Home: How to Translate Biomedical Laboratory Safety Precautions for Everyday Use in the Context of COVID-19. Am J Trop Med Hyg. 2020;103:838–840.
    1. Douillet D, Mahieu R, Boiveau V, Vandamme YM, Armand A, Morin F, Savary D, Dubée V, Annweiler C, Roy PM HOME-CoV expert group. Outpatient management or hospitalization of patients with proven or suspected SARS-CoV-2 infection: the HOME-CoV rule. Intern Emerg Med. 2020;15:1525–1531.
    1. Richez C, Flipo RM, Berenbaum F, Cantagrel A, Claudepierre P, Debiais F, Dieudé P, Goupille P, Roux C, Schaeverbeke T, Wendling D, Pham T, Thomas T. Managing patients with rheumatic diseases during the COVID-19 pandemic: The French Society of Rheumatology answers to most frequently asked questions up to May 2020. Joint Bone Spine. 2020;87:431–437.
    1. Yılmaz Yalçınkaya E, Karadağ Saygı E, Özyemişci Taşkıran Ö, Çapan N, Kutlay Ş, Sonel Tur B, El Ö, Ünlü Akyüz E, Tekin S, Ofluoğlu D, Zİnnuroğlu M, Akpınar P, Özekli Mısırlıoğlu T, Hüner B, Nur H, Çağlar S, Sezgin M, Tıkız C, Öneş K, İçağasıoğlu A, Aydın R. Consensus recommendations for botulinum toxin injections in the spasticity management of children with cerebral palsy during COVID-19 outbreak. Turk J Med Sci. 2021;51:385–392.
    1. Tanasijevic AM, Revette A, Klepin HD, Zeidan A, Townsley D, DiNardo CD, Sebert M, DeZern AE, Stone RM, Magnavita ES, Chen R, Sekeres MA, Abel GA. Consensus minimum hemoglobin level above which patients with myelodysplastic syndromes can safely forgo transfusions. Leuk Lymphoma. 2020;61:2900–2904.
    1. Alarcón MA, Sanz-Sánchez I, Shibli JA, Treviño Santos A, Caram S, Lanis A, Jiménez P, Dueñas R, Torres R, Alvarado J, Avendaño A, Galindo R, Umanzor V, Shedden M, Invernizzi C, Yibrin C, Collins J, León R, Contreras L, Bueno L, López-Pacheco A, Málaga-Figueroa L, Sanz M. Delphi Project on the trends in Implant Dentistry in the COVID-19 era: Perspectives from Latin America. Clin Oral Implants Res. 2021;32:521–537.

Source: PubMed

Sponsorit ja yhteistyökumppanit

Sairaudet

Huumeiden interventiot

3
Tilaa