Evaluating health worker performance in Benin using the simulated client method with real children

Alexander K Rowe, Faustin Onikpo, Marcel Lama, Michael S Deming, Alexander K Rowe, Faustin Onikpo, Marcel Lama, Michael S Deming

Abstract

Background: The simulated client (SC) method for evaluating health worker performance utilizes surveyors who pose as patients to make surreptitious observations during consultations. Compared to conspicuous observation (CO) by surveyors, which is commonly done in developing countries, SC data better reflect usual health worker practices. This information is important because CO can cause performance to be better than usual. Despite this advantage of SCs, the method's full potential has not been realized for evaluating performance for pediatric illnesses because real children have not been utilized as SCs. Previous SC studies used scenarios of ill children that were not actually brought to health workers. During a trial that evaluated a quality improvement intervention in Benin (the Integrated Management of Childhood Illness [IMCI] strategy), we conducted an SC survey with adult caretakers as surveyors and real children to evaluate the feasibility of this approach and used the results to assess the validity of CO.

Methods: We conducted an SC survey and a CO survey (one right after the other) of health workers in the same 55 health facilities. A detailed description of the SC survey process was produced. Results of the two surveys were compared for 27 performance indicators using logistic regression modeling.

Results: SC and CO surveyors observed 54 and 185 consultations, respectively. No serious problems occurred during the SC survey. Performance levels measured by CO were moderately higher than those measured by SCs (median CO - SC difference = 16.4 percentage-points). Survey differences were sometimes much greater for IMCI-trained health workers (median difference = 29.7 percentage-points) than for workers without IMCI training (median difference = 3.1 percentage-points).

Conclusion: SC surveys can be done safely with real children if appropriate precautions are taken. CO can introduce moderately large positive biases, and these biases might be greater for health workers exposed to quality improvement interventions.

Trial number: https://ichgcp.net/clinical-trials-registry/NCT00510679" title="See in ClinicalTrials.gov">NCT00510679.

Figures

Figure 1
Figure 1
Validity of the conspicuous observation survey method for 25 performance indicators in Benin: simple comparison of the conspicuous observation and simulated client surveys (i.e., conspicuous observation survey result minus simulated client survey result; see Table3, column 7). NB. Shading of the vertical bars indicates statistically significant differences.
Figure 2
Figure 2
Validity of the conspicuous observation survey method for 24 performance indicators in Benin: differential bias according to the IMCI training status of the health worker who performed the consultation (see Table3, columns 8–10). CO = Conspicuous observation; IMCI = Integrated Management of Childhood Illness; SC = simulated client. NB. Shading of the vertical bars indicates statistically significant differences.

References

    1. Madden JM, Quick JD, Ross-Degnan D, Kafle KK. Undercover careseekers: simulated clients in the study of health provider behavior in developing countries. Soc Sci Med. 1997;45:1465–1482. doi: 10.1016/S0277-9536(97)00076-2.
    1. Watson MC, Norris P, Granas AG. A systematic review of the use of simulated patients and pharmacy practice research. Int J Pharm Practice. 2006;14:83–93. doi: 10.1211/ijpp.14.2.0002.
    1. Franco LM, Daly CC, Chilongozi D, Dallabetta G. Quality of case management of sexually transmitted diseases: comparison of the methods for assessing the performance of providers. Bull World Health Organ. 1997;75:523–532.
    1. Leonard K, Masatu MC. Outpatient process quality evaluation and the Hawthorne effect. Soc Sci Med. 2006;63:2330–2340. doi: 10.1016/j.socscimed.2006.06.003.
    1. Roethlisberger FJ, Dickson WJ. Management and the worker: an account of a research program conducted by Western Electric Company, Hawthorne Works, Chicago. Cambridge: Harvard University Press; 1939.
    1. Rowe AK, Lama M, Onikpo F, Deming MS. Health worker perceptions of how being observed influences their practices during consultations with ill children. Trop Doc. 2002;32:166–167.
    1. Rowe SY, Olewe MA, Kleinbaum DG, McGowan JE, McFarland DA, Rochat R. et al.The influence of observation and setting on community health workers’ practices. Int J Qual Health Care. 2006;18:299–305. doi: 10.1093/intqhc/mzl009.
    1. Peabody JW, Luck J, Glassman P, Dresselhaus TR, Lee M. Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality. JAMA. 2000;283:1715–1722. doi: 10.1001/jama.283.13.1715.
    1. Gove S for the WHO Working Group on Guidelines for Integrated Management of the Sick Child. Integrated management of childhood illness by outpatient health workers: technical basis and overview. Bull World Health Organ. 1997;75(Suppl 1):7–24.
    1. Rowe AK, Onikpo F, Lama M, Osterholt DM, Rowe SY, Deming MS. A multi-faceted intervention to improve health worker adherence to integrated management of childhood illness guidelines in Benin. Am J Public Health. 2009;99:837–846. doi: 10.2105/AJPH.2008.134411.
    1. Benin Ministry of Public Health. Prise en charge intégrée des maladies de l’enfant. Livret des tableaux. Porto-Novo, Benin; 2002.
    1. Rowe AK, Onikpo F, Lama M, Cokou F, Deming MS. Management of childhood illness at health facilities in Benin: problems and their causes. Am J Public Health. 2001;91:1625–1635. doi: 10.2105/AJPH.91.10.1625.
    1. SAS Institute. SAS/STAT(R) 9.2 User’s Guide, Second Edition. 2012. Available on-line at , accessed on May 21. 2012.
    1. Glassman PA, Luck J, O’Gara EM, Peabody JW. Using standardized patients to measure quality: evidence from the literature and a prospective study. Jt Comm J Qual Improv. 2000;26:644–653.
    1. Ali J, Cherry RA, Williams JI. Trauma base knowledge and the effect of the trauma evaluation and management program among senior medical students in seven countries. J Trauma. 2005;59:43–48. doi: 10.1097/01.TA.0000171461.12129.10.
    1. Brooks AJ, Phipson M, Potgieter A, Koertzen H, Boffard KD. Education of the trauma team: video evaluation of the compliance with universal barrier precautions in resuscitation. Eur J Surg. 1999;165:1125–1128. doi: 10.1080/110241599750007621.
    1. Scherer LA, Chang MC, Meredith JW, Battistella FD. Videotape review leads to rapid and sustained learning. Am J Surg. 2003;185:516–520. doi: 10.1016/S0002-9610(03)00062-X.
    1. Birnbach DJ, Santos AC, Bourlier RA, Meadows WE, Datta S, Stein DJ. et al.The effectiveness of video technology as an adjunct to teach and evaluate epidural anesthesia performance skills. Anesthesiology. 2002;96:5–9. doi: 10.1097/00000542-200201000-00007.
    1. Weston CFM, Richmond P, McCabe MJ, Evans R, Evans R. Video recording of cardiac arrest management: an aid to training and audit. Resuscitation. 1992;24:13–15. doi: 10.1016/0300-9572(92)90169-D.
    1. McCormick DP, Rassin GM, Stroup-Benham CA, Baldwin CD, Levine HG, Persaud DI. et al.Use of videotaping to evaluate pediatric resident performance of health supervision examinations of infants. Pediatrics. 1993;92:116–120.
    1. Arborelius E, Bremberg S. Prevention in practice. How do general practitioners discuss life-style issues with their patients? Patient Educ Couns. 1994;23:23–31.
    1. Fossum B, Arborelius E. Patient-centred communication: videotaped consultations. Patient Educ Couns. 2004;54:163–169. doi: 10.1016/S0738-3991(03)00208-8.
    1. Harting J, van Assema P, van der Molen HT, Ambergen T, de Vries NK. Quality assessment of health counseling: performance of health advisors in cardiovascular prevention. Patient Educ Couns. 2004;54:107–118. doi: 10.1016/S0738-3991(03)00194-0.
    1. Holmström IM, Rosenqvist U. Misunderstandings about illness and treatment among patients with type 2 diabetes. J Adv Nurs. 2005;49:146–154. doi: 10.1111/j.1365-2648.2004.03274.x.
    1. Adomat R, Hicks C. Measuring nursing workload in intensive care: an observational study using closed circuit video cameras. J Adv Nurs. 2003;42:402–412. doi: 10.1046/j.1365-2648.2003.02632.x.
    1. Arborelius E, Timpka T. In what way may videotapes be used to get significant information about the patient–physician relationship? Med Teacher. 1990;12:197–208. doi: 10.3109/01421599009006697.
    1. Rogers SC, Dudley NC, McDonnell W, Scaife E, Morris S, Nelson D. Lights, camera, action… spotlight on trauma video review: an underutilized means of quality improvement and education. Pediatr Emer Care. 2010;26:803–807. doi: 10.1097/PEC.0b013e3181fa874a.
    1. Abdulhadi N, Al-Shafaee MA, Östenson C-G, Vernby Å, Wahlström R. Quality of interaction between primary health-care providers and patients with type 2 diabetes in Muscat. Oman: an observational study. BMC Family Practice. 2006;7:72.
    1. Centers for Disease Control and Prevention. Health worker performance after training in integrated management of childhood illness — Western Province, Kenya, 1996–1997. Morb Mort Weekly Report. 1998;47:998–1001.

Source: PubMed

Sponsors et collaborateurs

Les conditions médicales

Interventions en matière de drogue

3
S'abonner