WHO Standards-based questionnaire to measure health workers' perspective on the quality of care around the time of childbirth in the WHO European region: development and mixed-methods validation in six countries

Emanuelle Pessa Valente, Benedetta Covi, Ilaria Mariani, Sandra Morano, Marina Otalea, Ioana Nanu, Micaela Iuliana Nanu, Helen Elden, Karolina Linden, Mehreen Zaigham, Eline Skirnisdottir Vik, Sigrun Kongslien, Ingvild Nedberg, Raquel Costa, Carina Rodrigues, Heloísa Dias, Daniela Drandić, Magdalena Kurbanović, Emma Sacks, Moise Muzigaba, Ornella Lincetto, Marzia Lazzerini, IMAgiNE EURO Study Group, Amira Ćerimagic, Rozée Virginie, Elise deLa Rochebrochard, Kristina Löfgren, Céline Miani, Stephanie Batram-Zantvoort, Lisa Wandschneider, Giuseppa Verardi, Beatrice Zanin, Ilana Chertok, Rada Artzi-Medvedik, Elizabete Pumpure, Dace Rezeberga, Agnija Vaska, Dārta Jakovicka, Paula Rudzīte, Elīna Ērmane, Katrīna Paula Vilcāne, Maryse Arendt, Barbara Tasch, Barbara Baranowska, Urszula Tataj-Puzyna, Maria Węgrzynowska, Catarina Barata, Teresa Santos, Ekaterina Yarotskaya, Jelena Radetić, Jovana Ružičić, Zalka Drglin, Barbara Mihevc Ponikvar, Anja Bohinec, Serena Brigidi, Lara Martín Castañeda, Ana Canales Viver, Verena Sengpiel, Claire De Labrusse, Alessia Abderhalden, Anouck Pfund, Harriet Thorn, Emanuelle Pessa Valente, Benedetta Covi, Ilaria Mariani, Sandra Morano, Marina Otalea, Ioana Nanu, Micaela Iuliana Nanu, Helen Elden, Karolina Linden, Mehreen Zaigham, Eline Skirnisdottir Vik, Sigrun Kongslien, Ingvild Nedberg, Raquel Costa, Carina Rodrigues, Heloísa Dias, Daniela Drandić, Magdalena Kurbanović, Emma Sacks, Moise Muzigaba, Ornella Lincetto, Marzia Lazzerini, IMAgiNE EURO Study Group, Amira Ćerimagic, Rozée Virginie, Elise deLa Rochebrochard, Kristina Löfgren, Céline Miani, Stephanie Batram-Zantvoort, Lisa Wandschneider, Giuseppa Verardi, Beatrice Zanin, Ilana Chertok, Rada Artzi-Medvedik, Elizabete Pumpure, Dace Rezeberga, Agnija Vaska, Dārta Jakovicka, Paula Rudzīte, Elīna Ērmane, Katrīna Paula Vilcāne, Maryse Arendt, Barbara Tasch, Barbara Baranowska, Urszula Tataj-Puzyna, Maria Węgrzynowska, Catarina Barata, Teresa Santos, Ekaterina Yarotskaya, Jelena Radetić, Jovana Ružičić, Zalka Drglin, Barbara Mihevc Ponikvar, Anja Bohinec, Serena Brigidi, Lara Martín Castañeda, Ana Canales Viver, Verena Sengpiel, Claire De Labrusse, Alessia Abderhalden, Anouck Pfund, Harriet Thorn

Abstract

Objectives: Develop and validate a WHO Standards-based online questionnaire to measure the quality of maternal and newborn care (QMNC) around the time of childbirth from the health workers' perspective.

Design: Mixed-methods study.

Setting: Six countries of the WHO European Region.

Participants and methods: The questionnaire is based on lessons learnt in previous studies, and was developed in three sequential phases: (1) WHO Quality Measures were prioritised and content, construct and face validity were assessed through a Delphi involving a multidisciplinary board of experts from 11 countries of the WHO European Region; (2) translation/back translation of the English version was conducted following The Professional Society for Health Economics and Outcomes Research guidelines; (3) internal consistency, intrarater reliability and acceptability were assessed among 600 health workers in six countries.

Results: The questionnaire included 40 items based on WHO Standards Quality Measures, equally divided into four domains: provision of care, experience of care, availability of human and physical resources, organisational changes due to COVID-19; and its organised in six sections. It was translated/back translated in 12 languages: Bosnian, Croatian, French, German, Italian, Norwegian, Portuguese, Romanian, Russian, Slovenian, Spanish and Swedish. The Cronbach's alpha values were ≥0.70 for each questionnaire section where questions were hypothesised to be interrelated, indicating good internal consistence. Cohen K or Gwet's AC1 values were ≥0.60, suggesting good intrarater reliability, except for one question. Acceptability was good with only 1.70% of health workers requesting minimal changes in question wording.

Conclusions: Findings suggest that the questionnaire has good content, construct, face validity, internal consistency, intrarater reliability and acceptability in six countries of the WHO European Region. Future studies may further explore the questionnaire's use in other countries, and how to translate evidence generated by this tool into policies to improve the QMNC.

Trail registration number: NCT04847336.

Keywords: NEONATOLOGY; OBSTETRICS; Quality in health care.

Conflict of interest statement

Competing interests: None declared.

© Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Figures

Figure 1
Figure 1
Phases of questionnaire development and multicountry validation.

References

    1. World Health Organization . Every woman every child. Global strategy for women’s, children’s and adolescents health 2016–2030, 2016. Available: [Accessed 14 Apr 2021].
    1. United Nations . The Global Strategy for Women’s, Children’s and Adolescents’ Health (2016-2030). Every Woman Every Child, 2015. Available: [Accessed 14 Apr 2021].
    1. The White Ribbon Alliance . Respectful maternity care charter: universal rights of women and newborns, 2021. Available: [Accessed 14 Apr 2021].
    1. Graham WJ, Varghese B. Quality, quality, quality: gaps in the continuum of care. Lancet 2012;379:e5–6. 10.1016/S0140-6736(10)62267-2
    1. Bohren MA, Vogel JP, Hunter EC, et al. . The mistreatment of women during childbirth in health facilities globally: a mixed-methods systematic review. PLoS Med 2015;12:e1001847. 10.1371/journal.pmed.1001847
    1. Euro-Peristat Project . European perinatal health report. core indicators of the health and care of pregnant women and babies in Europe in 2015, 2015. Available: [Accessed 14 Apr 2021].
    1. Prochaska E. Human rights in maternity care. Midwifery 2015;31:1015–6. 10.1016/j.midw.2015.09.006
    1. Shaw D, Guise J-M, Shah N, et al. . Drivers of maternity care in high-income countries: can health systems support woman-centred care? Lancet 2016;388:2282–95. 10.1016/S0140-6736(16)31527-6
    1. Koblinsky M, Moyer CA, Calvert C, et al. . Quality maternity care for every woman, everywhere: a call to action. Lancet 2016;388:2307–20. 10.1016/S0140-6736(16)31333-2
    1. World Health Organization . Health inequity and the effects of COVID-19: assessing, responding to and mitigating the socioeconomic impact on health to build a better future, 2020. Available:
    1. OECD/European Union . Health at a glance: Europe 2020: state of health in the EU cycle, 2020. Available: 10.1787/82129230-en [Accessed 27 Apr 2021].
    1. World Health Organisation . Operational guidance for maintaining essential health services during an outbreak, 2020. Available: [Accessed 27 Apr 2021].
    1. Semaan A, Audet C, Huysmans E, et al. . Voices from the frontline: findings from a thematic analysis of a rapid online global survey of maternal and newborn health professionals facing the COVID-19 pandemic. BMJ Glob Health 2020;5:e002967. 10.1136/bmjgh-2020-002967
    1. Wallace JE, Lemaire JB, Ghali WA. Physician wellness: a missing quality indicator. Lancet 2009;374:21. 10.1016/S0140-6736(09)61424-0
    1. Mira JJ, Carrillo I, Guilabert M, et al. . Acute stress of the healthcare workforce during the COVID-19 pandemic evolution: a cross-sectional study in Spain. BMJ Open 2020;10:e042555. 10.1136/bmjopen-2020-042555
    1. Chmielewska B, Barratt I, Townsend R, et al. . Effects of the COVID-19 pandemic on maternal and perinatal outcomes: a systematic review and meta-analysis. Lancet Glob Health 2021;9:e759–72. 10.1016/S2214-109X(21)00079-6
    1. Gajbhiye RK, Sawant MS, Kuppusamy P, et al. . Differential impact of COVID-19 in pregnant women from high-income countries and low- to middle-income countries: a systematic review and meta-analysis. Int J Gynaecol Obstet 2021;155:48-56. 10.1002/ijgo.13793
    1. Villar J, Ariff S, Gunier RB, et al. . Maternal and neonatal morbidity and mortality among pregnant women with and without COVID-19 infection: the INTERCOVID multinational cohort study. JAMA Pediatr 2021;175:817–26. 10.1001/jamapediatrics.2021.1050
    1. Bozorgmehr K, Saint V, Kaasch A, et al. . COVID and the convergence of three crises in Europe. Lancet Public Health 2020;5:e247–8. 10.1016/S2468-2667(20)30078-5
    1. Anderson M, Mckee M, Mossialos E. Covid-19 exposes weaknesses in European response to outbreaks. BMJ 2020;368:m1075. 10.1136/bmj.m1075
    1. Nanda M, Sharma R, Aashima SR. COVID-19: a comprehensive review of epidemiology and public health system response in Nordic region. Int J Health Serv 2021;51:287–99. 10.1177/0020731421994840
    1. Tunçalp Ӧ, Were WM, MacLennan C, et al. . Quality of care for pregnant women and newborns-the who vision. BJOG 2015;122:1045–9. 10.1111/1471-0528.13451
    1. World Health Organization . Standards for improving quality of maternal and newborn care in health facilities, 2016. Available: [Accessed 14 Apr 2021].
    1. Word Health Organization . Improving health worker performance: in search of promising practices. Evidence and information for policy, department of human resources for health, 2006. Available: [Accessed 12 Jul 2021].
    1. Word Health Organization . Health 2020. A European policy framework and strategy for the 21st century, 2013. Available: [Accessed 27 Apr 2021].
    1. World Health Organization . Global strategy on human resources for health: workforce 2030, 2016:. Available: [Accessed 27 Apr 2021].
    1. Kruk ME, Myers M, Varpilah ST, et al. . What is a resilient health system? lessons from Ebola. The Lancet 2015;385:1910–2. 10.1016/S0140-6736(15)60755-3
    1. Mannava P, Durrant K, Fisher J, et al. . Attitudes and behaviours of maternal health care providers in interactions with clients: a systematic review. Global Health 2015;11:36. 10.1186/s12992-015-0117-9
    1. Lazzerini M, Valente EP, Covi B, et al. . Use of who standards to improve quality of maternal and newborn hospital care: a study collecting both mothers' and staff perspective in a tertiary care hospital in Italy. BMJ Open Qual 2019;8:e000525. 10.1136/bmjoq-2018-000525
    1. Lazzerini M, Argentini G, Mariani I, et al. . WHO standards-based tool to measure women's views on the quality of care around the time of childbirth at facility level in the WHO European region: development and validation in Italy. BMJ Open 2022;12:e048195. 10.1136/bmjopen-2020-048195
    1. Lazzerini M, Mariani I, de MeloELima TR. WHO standards-based tools to measure service providers’ and service users’ views on the quality of hospital child care: development and validation in Italy. BMJ Open 2021:e052115. 10.1136/bmjopen-2021-052115
    1. Lazzerini M, Mariani I, Semenzato C, et al. . Association between maternal satisfaction and other indicators of quality of care at childbirth: a cross-sectional study based on the who standards. BMJ Open 2020;10:e037063. 10.1136/bmjopen-2020-037063
    1. Lazzerini M, Semenzato C, Kaur J, et al. . Women's suggestions on how to improve the quality of maternal and newborn hospital care: a qualitative study in Italy using the who standards as framework for the analysis. BMC Pregnancy Childbirth 2020;20:200. 10.1186/s12884-020-02893-0
    1. Lazzerini M, Covi B, Mariani I, et al. . Quality of facility-based maternal and newborn care around the time of childbirth during the COVID-19 pandemic: online survey investigating maternal perspectives in 12 countries of the who European region. Lancet Reg Health Eur 2022;13:100268. 10.1016/j.lanepe.2021.100268
    1. Streiner DL, Norman GR, Cairney J. Health measurement scales: a practical guide to their development and use. 5 edn. Oxford University Press, 2014.
    1. Terwee CB, Bot SDM, de Boer MR, et al. . Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007;60:34–42. 10.1016/j.jclinepi.2006.03.012
    1. Tsang S, Royse CF, Terkawi AS. Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine. Saudi J Anaesth 2017;11:S80–9. 10.4103/sja.SJA_203_17
    1. Taherdoost H. Validity and reliability of the research instrument; how to test the validation of a Questionnaire/Survey in a research. SSRN Electronic Journal 2016;19:28–36. 10.2139/ssrn.3205040
    1. Sawyer A, Ayers S, Abbott J, et al. . Measures of satisfaction with care during labour and birth: a comparative review. BMC Pregnancy Childbirth 2013;13:108. 10.1186/1471-2393-13-108
    1. Wild D, Grove A, Martin M, et al. . Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (pro) measures: report of the ISPOR Task force for translation and cultural adaptation. Value Health 2005;8:94–‐104. 10.1111/j.1524-4733.2005.04054.x
    1. Braithwaite D, Emery J, De Lusignan S, et al. . Using the Internet to conduct surveys of health professionals: a valid alternative? Fam Pract 2003;20:5. 10.1093/fampra/cmg509
    1. Lusk C, Delclos GL, Burau K, et al. . Mail versus Internet surveys: determinants of method of response preferences among health professionals. Eval Health Prof 2007;30:186–201. 10.1177/0163278707300634
    1. Audibert C, Glass D, Johnson TP. Method and transparency of online physician surveys: an overview. survey methods: insights from the field, 2020.
    1. World Health organization . Considerations for public health and social measures in the workplace in the context of COVID-19, 2020. Available:
    1. WHO . Clinical management of severe acute respiratory infection (SARI) when COVID-19 disease is suspected. interim guidance, 2020. Available:
    1. World Health organization . Addressing human rights as key to the COVID-19 response, 2020. Available: [Accessed 28 Apr 2021].
    1. World Health organization . Gender and COVID-19: advocacy brief, 14 may 2020, 2020. Available:
    1. World Health organization . Pregnancy, childbirth, breastfeeding and COVID-19, 2020. Available: [Accessed 16 Apr 2020].
    1. World Health organization . Q&A on COVID-19, pregnancy, childbirth and breastfeeding. Available: [Accessed 16 Apr 2020].
    1. World Health organization . Frequently asked questions: breastfeeding and COVID-19For health care workers. Available: [Accessed 29 Apr 2020].
    1. World Health organization . Maintaining essential health services: operational guidance for the COVID-19 contextInterim guidance, 2020. Available:
    1. International Confederation of midwives protecting midwives to sustain care for women, newborns, and their families in the COVID-19 pandemic. joint statement, 2020. Available:
    1. International Confederation of Midwives . The Hague; 2020. Women’s rights in childbirth must be upheld during the coronavirus epidemic. Available:
    1. International Confederation of Midwives . Protecting midwives to sustain care for women, newborns and their families in the COVID-19 pandemic. global call to action, 2020. Available:
    1. NSW Health . COVID-19: information for women accessing maternity services. Available: [Accessed 16 Apr 2020].
    1. Istituto Superiore di Sanità. Epicentro . COVID-19 in gravidanza, parto e allattamento. Available: [Accessed 28 Apr 2020].
    1. SIN. ALLATTAMENTO e INFEZIONE da SARS-CoV-2 (Coronavirus Disease 2019 - COVID-19). Indicazioni ad interim della Societ Italiana di Neonatologia (SIN) Versione 2. 22 marzo, 2020. Available: [Accessed 03 Apr 2020].
    1. Giusti A, Zambri F, Marchetti F, et al. . Indicazioni ad interim per gravidanza, parto, allattamento e cura dei piccolissimi di 0-2 anni in risposta all’emergenza COVID-19. Versione 31 maggio 2020. Roma: Istituto Superiore di Sanit, 2020 (Rapporto ISS COVID-19 n. 45/2020).
    1. Poon LC, Yang H, Kapur A, et al. . Global interim guidance on coronavirus disease 2019 (COVID-19) during pregnancy and puerperium from FIGO and allied partners: information for healthcare professionals. Int J Gynaecol Obstet 2020;149:273-286. 10.1002/ijgo.13156
    1. Royal College of Obstetricians and Gynaecologists, The Royal College of Midwives UK, Royal College of Paediatrics and Child Health, Royal College of Anaesthetists & Obstetric Anaesthetists’ Association . Coronavirus (COVID-19) infection in pregnancy information for healthcare professionals version 9, 2020. Available:
    1. Birthrights . Human rights charity calls for protection of UK women in childbirth during national emergency. statement, 2020. Available:
    1. Vivilaki VG, Asimaki E. Respectful midwifery care during the COVID-19 pandemic. Eur J Midwifery 2020;4:8. 10.18332/ejm/120070
    1. Lazzerini M, Barbi E, Apicella A, et al. . Delayed access or provision of care in Italy resulting from fear of COVID-19. Lancet Child Adolesc Health 2020;4:e10–11. 10.1016/S2352-4642(20)30108-5
    1. Legido-Quigley H, Mateos-García JT, Campos VR, et al. . The resilience of the Spanish health system against the COVID-19 pandemic. Lancet Public Health 2020;5:e251–2. 10.1016/S2468-2667(20)30060-8
    1. Afulani PA, Phillips B, Aborigo RA, et al. . Person-Centred maternity care in low-income and middle-income countries: analysis of data from Kenya, Ghana, and India. Lancet Glob Health 2019;7:e96–109. 10.1016/S2214-109X(18)30403-0
    1. Vedam S, Stoll K, Rubashkin N, et al. . The mothers on respect (MOR) index: measuring quality, safety, and human rights in childbirth. SSM Popul Health 2017;3:201–10. 10.1016/j.ssmph.2017.01.005
    1. Vedam S, Stoll K, Martin K, et al. . The mother's autonomy in decision making (MADM) scale: Patient-led development and psychometric testing of a new instrument to evaluate experience of maternity care. PLoS One 2017;12:e0171804. 10.1371/journal.pone.0171804
    1. Tavakol M, Dennick R. Making sense of Cronbach's alpha. Int J Med Educ 2011;2:53–5. 10.5116/ijme.4dfb.8dfd
    1. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37–46. 10.1177/001316446002000104
    1. Zec S, Soriani N, Comoretto R, et al. . High agreement and high prevalence: the paradox of Cohen's kappa. Open Nurs J 2017;11:211–8. 10.2174/1874434601711010211
    1. Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 1990;43:543–9. 10.1016/0895-4356(90)90158-l
    1. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74.
    1. Kyriazos TA. Applied Psychometrics: sample size and sample power considerations in factor analysis (EFA, CFA) and SEM in general. Psychology 2018;09:2207–30. 10.4236/psych.2018.98126
    1. Sousa VD, Rojjanasrirat W. Translation, adaptation and validation of instruments or scales for use in cross-cultural health care research: a clear and user-friendly guideline. J Eval Clin Pract 2011;17:268–74. 10.1111/j.1365-2753.2010.01434.x
    1. World Health organization . Hospital care for mothers and newborn babies quality assessment and improvement tool, 2014. Available: [Accessed 15 Jul 2021].
    1. Abdelrahman H, Atteya S, Ihab M, et al. . Dental practice closure during the first wave of COVID-19 and associated professional, practice and structural determinants: a multi-country survey. BMC Oral Health 2021;21:243. 10.1186/s12903-021-01601-4
    1. Delgado D, Wyss Quintana F, Perez G, et al. . Personal safety during the COVID-19 pandemic: realities and perspectives of healthcare workers in Latin America. Int J Environ Res Public Health 2020;17:2798. 10.3390/ijerph17082798
    1. Akachi Y, Kruk ME. Quality of care: measuring a neglected driver of improved health. Bull World Health Organ 2017;95:465–72. 10.2471/BLT.16.180190
    1. Bombard Y, Baker GR, Orlando E, et al. . Engaging patients to improve quality of care: a systematic review. Implementation Sci 2018;13:98. 10.1186/s13012-018-0784-z

Source: PubMed

Спонсоры и соавторы

Заболевания

Медикаментозные вмешательства

Подписаться