Impact of a Machine Learning-Based Decision Support System for Urinary Tract Infections: Prospective Observational Study in 36 Primary Care Practices

Willem Ernst Herter, Janine Khuc, Giovanni Cinà, Bart J Knottnerus, Mattijs E Numans, Maryse A Wiewel, Tobias N Bonten, Daan P de Bruin, Thamar van Esch, Niels H Chavannes, Robert A Verheij, Willem Ernst Herter, Janine Khuc, Giovanni Cinà, Bart J Knottnerus, Mattijs E Numans, Maryse A Wiewel, Tobias N Bonten, Daan P de Bruin, Thamar van Esch, Niels H Chavannes, Robert A Verheij

Abstract

Background: There is increasing attention on machine learning (ML)-based clinical decision support systems (CDSS), but their added value and pitfalls are very rarely evaluated in clinical practice. We implemented a CDSS to aid general practitioners (GPs) in treating patients with urinary tract infections (UTIs), which are a significant health burden worldwide.

Objective: This study aims to prospectively assess the impact of this CDSS on treatment success and change in antibiotic prescription behavior of the physician. In doing so, we hope to identify drivers and obstacles that positively impact the quality of health care practice with ML.

Methods: The CDSS was developed by Pacmed, Nivel, and Leiden University Medical Center (LUMC). The CDSS presents the expected outcomes of treatments, using interpretable decision trees as ML classifiers. Treatment success was defined as a subsequent period of 28 days during which no new antibiotic treatment for UTI was needed. In this prospective observational study, 36 primary care practices used the software for 4 months. Furthermore, 29 control practices were identified using propensity score-matching. All analyses were performed using electronic health records from the Nivel Primary Care Database. Patients for whom the software was used were identified in the Nivel database by sequential matching using CDSS use data. We compared the proportion of successful treatments before and during the study within the treatment arm. The same analysis was performed for the control practices and the patient subgroup the software was definitely used for. All analyses, including that of physicians' prescription behavior, were statistically tested using 2-sided z tests with an α level of .05.

Results: In the treatment practices, 4998 observations were included before and 3422 observations (of 2423 unique patients) were included during the implementation period. In the control practices, 5044 observations were included before and 3360 observations were included during the implementation period. The proportion of successful treatments increased significantly from 75% to 80% in treatment practices (z=5.47, P<.001). No significant difference was detected in control practices (76% before and 76% during the pilot, z=0.02; P=.98). Of the 2423 patients, we identified 734 (30.29%) in the CDSS use database in the Nivel database. For these patients, the proportion of successful treatments during the study was 83%-a statistically significant difference, with 75% of successful treatments before the study in the treatment practices (z=4.95; P<.001).

Conclusions: The introduction of the CDSS as an intervention in the 36 treatment practices was associated with a statistically significant improvement in treatment success. We excluded temporal effects and validated the results with the subgroup analysis in patients for whom we were certain that the software was used. This study shows important strengths and points of attention for the development and implementation of an ML-based CDSS in clinical practice.

Trial registration: ClinicalTrials.gov NCT04408976; https://ichgcp.net/clinical-trials-registry/NCT04408976.

Keywords: ML; artificial intelligence; clinical decision support system; implementation study; information technology; machine learning; urinary tract infections.

Conflict of interest statement

Conflicts of Interest: WEH is a PhD candidate at Leiden University Medical Center (LUMC) as well as a director at Pacmed. JK, MAW, DPdB, and GC worked at Pacmed during their contributions to the research and paper.

©Willem Ernst Herter, Janine Khuc, Giovanni Cinà, Bart J Knottnerus, Mattijs E Numans, Maryse A Wiewel, Tobias N Bonten, Daan P de Bruin, Thamar van Esch, Niels H Chavannes, Robert A Verheij. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 04.05.2022.

Figures

Figure 1
Figure 1
Decision support software: interface to enter patient characteristics (top); presentation of expected outcomes and NHG (Dutch College of General Practitioners) guidelines (bottom).
Figure 2
Figure 2
The number of matches found through the sequential matching procedure.

References

    1. De Moor G, Sundgren M, Kalra D, Schmidt A, Dugas M, Claerhout B, Karakoyun T, Ohmann C, Lastic PY, Ammour N, Kush R, Dupont D, Cuggia M, Daniel C, Thienpont G, Coorevits P. Using electronic health records for clinical research: the case of the EHR4CR project. J Biomed Inform. 2015;53:162–73. doi: 10.1016/j.jbi.2014.10.006. S1532-0464(14)00226-3
    1. Jensen PB, Jensen LJ, Brunak S. Mining electronic health records: towards better research applications and clinical care. Nat Rev Genet. 2012;13(6):395–405. doi: 10.1038/nrg3208.nrg3208
    1. Krittanawong C. The rise of artificial intelligence and the uncertain future for physicians. Eur J Intern Med. 2018;48:e13–4. doi: 10.1016/j.ejim.2017.06.017.S0953-6205(17)30261-3
    1. Durso SC. Using clinical guidelines designed for older adults with diabetes mellitus and complex health status. JAMA. 2006;295(16):1935–40. doi: 10.1001/jama.295.16.1935.295/16/1935
    1. Roche N, Anzueto A, Bosnic Anticevich S, Kaplan A, Miravitlles M, Ryan D, Soriano JB, Usmani O, Papadopoulos NG, Canonica GW, Respiratory Effectiveness Group Collaborators The importance of real-life research in respiratory medicine: manifesto of the Respiratory Effectiveness Group: endorsed by the International Primary Care Respiratory Group and the World Allergy Organization. Eur Respir J. 2019;54(3):1901511. doi: 10.1183/13993003.01511-2019. 54/3/1901511
    1. Shaneyfelt TM, Centor RM. Reassessment of clinical practice guidelines: go gently into that good night. JAMA. 2009;301(8):868–9. doi: 10.1001/jama.2009.225.301/8/868
    1. Tinetti ME, Bogardus Jr ST, Agostini JV. Potential pitfalls of disease-specific guidelines for patients with multiple conditions. N Engl J Med. 2004;351(27):2870–4. doi: 10.1056/NEJMsb042458.351/27/2870
    1. Boyd CM, Darer J, Boult C, Fried LP, Boult L, Wu AW. Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance. JAMA. 2005;294(6):716–24. doi: 10.1001/jama.294.6.716.294/6/716
    1. Schoen C, Osborn R, Huynh PT, Doty M, Peugh J, Zapert K. On the front lines of care: primary care doctors' office systems, experiences, and views in seven countries. Health Aff (Millwood) 2006;25(6):w555–71. doi: 10.1377/hlthaff.25.w555.hlthaff.25.w555
    1. Panch T, Mattie H, Celi LA. The "inconvenient truth" about AI in healthcare. NPJ Digit Med. 2019;2:77. doi: 10.1038/s41746-019-0155-4. doi: 10.1038/s41746-019-0155-4.155
    1. Peiffer-Smadja N, Rawson TM, Ahmad R, Buchard A, Georgiou P, Lescure FX, Birgand G, Holmes AH. Machine learning for clinical decision support in infectious diseases: a narrative review of current applications. Clin Microbiol Infect. 2020;26(5):584–95. doi: 10.1016/j.cmi.2019.09.009. S1198-743X(19)30494-X
    1. Roshanov PS, Fernandes N, Wilczynski JM, Hemens BJ, You JJ, Handler SM, Nieuwlaat R, Souza NM, Beyene J, Van Spall HG, Garg AX, Haynes RB. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ. 2013;346:f657. doi: 10.1136/bmj.f657.
    1. Rawson TM, Moore LS, Hernandez B, Charani E, Castro-Sanchez E, Herrero P, Hayhoe B, Hope W, Georgiou P, Holmes AH. A systematic review of clinical decision support systems for antimicrobial management: are we failing to investigate these interventions appropriately? Clin Microbiol Infect. 2017;23(8):524–32. doi: 10.1016/j.cmi.2017.02.028. S1198-743X(17)30125-8
    1. Sennesael AL, Krug B, Sneyers B, Spinewine A. Do computerized clinical decision support systems improve the prescribing of oral anticoagulants? A systematic review. Thromb Res. 2020;187:79–87. doi: 10.1016/j.thromres.2019.12.023.S0049-3848(19)30556-0
    1. Keyworth C, Hart J, Armitage CJ, Tully MP. What maximizes the effectiveness and implementation of technology-based interventions to support healthcare professional practice? A systematic literature review. BMC Med Inform Decis Mak. 2018;18(1):93. doi: 10.1186/s12911-018-0661-3. 10.1186/s12911-018-0661-3
    1. Naqa IE, Kosorok MR, Jin J, Mierzwa M, Ten Haken RK. Prospects and challenges for clinical decision support in the era of big data. JCO Clin Cancer Inform. 2018;2:1–12. doi: 10.1200/CCI.18.00002.
    1. Foxman B. The epidemiology of urinary tract infection. Nat Rev Urol. 2010;7(12):653–60. doi: 10.1038/nrurol.2010.190.nrurol.2010.190
    1. Flores-Mireles AL, Walker JN, Caparon M, Hultgren SJ. Urinary tract infections: epidemiology, mechanisms of infection and treatment options. Nat Rev Microbiol. 2015;13(5):269–84. doi: 10.1038/nrmicro3432. nrmicro3432
    1. Jaarcijfers aandoeningen - Huisartsenregistraties. Nivel. [2020-12-24]. .
    1. Gupta K, Hooton TM, Naber KG, Wullt B, Colgan R, Miller LG, Moran GJ, Nicolle LE, Raz R, Schaeffer AJ, Soper DE, Infectious Diseases Society of America. European Society for Microbiology and Infectious Diseases International clinical practice guidelines for the treatment of acute uncomplicated cystitis and pyelonephritis in women: a 2010 update by the Infectious Diseases Society of America and the European Society for Microbiology and Infectious Diseases. Clin Infect Dis. 2011;52(5):e103–20. doi: 10.1093/cid/ciq257.ciq257
    1. Schneeberger C, Stolk RP, Devries JH, Schneeberger PM, Herings RM, Geerlings SE. Differences in the pattern of antibiotic prescription profile and recurrence rate for possible urinary tract infections in women with and without diabetes. Diabetes Care. 2008;31(7):1380–5. doi: 10.2337/dc07-2188. dc07-2188
    1. van Pinxteren B, Knottnerus B, Geerlings S, Visser I, Klinkhamer S. NHG-Standaard Urineweginfecties (derde herziening) Nederlands Huisartsen Genootschap. 2013. [2022-03-24]. .
    1. Lugtenberg M, Zegers-van Schaick JM, Westert GP, Burgers JS. Why don't physicians adhere to guideline recommendations in practice? An analysis of barriers among Dutch general practitioners. Implement Sci. 2009;4:54. doi: 10.1186/1748-5908-4-54. 1748-5908-4-54
    1. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay É. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12(85):2825–30.
    1. Hayes RJ, Moulton LH. Cluster randomised trials. Boca Raton, FL: Chapman and Hall/CRC; 2009.
    1. Lin J, Gamalo-Siebers M, Tiwari R. Propensity score matched augmented controls in randomized clinical trials: a case study. Pharm Stat. 2018;17(5):629–47. doi: 10.1002/pst.1879.
    1. Governance-document: Nivel Zorgregistraties Eerste Lijn. Nivel. [2022-03-24]. .
    1. Cohen J. Statistical power analysis for the behavioral sciences. Cambridge, MA: Academic Press; 1969.
    1. Ranganathan P, Aggarwal R, Pramesh CS. Common pitfalls in statistical analysis: odds versus risk. Perspect Clin Res. 2015;6(4):222–4. doi: 10.4103/2229-3485.167092. PCR-6-222
    1. Nielen MM, Spronk I, Davids R, Korevaar JC, Poos R, Hoeymans N, Opstelten W, van der Sande MA, Biermans MC, Schellevis FG, Verheij RA. Estimating morbidity rates based on routine electronic health records in primary care: observational study. JMIR Med Inform. 2019;7(3):e11929. doi: 10.2196/11929. v7i3e11929
    1. Beauchemin M, Murray MT, Sung L, Hershman DL, Weng C, Schnall R. Clinical decision support for therapeutic decision-making in cancer: a systematic review. Int J Med Inform. 2019;130:103940. doi: 10.1016/j.ijmedinf.2019.07.019. S1386-5056(19)30374-0
    1. Laka M, Milazzo A, Merlin T. Can evidence-based decision support tools transform antibiotic management? A systematic review and meta-analyses. J Antimicrob Chemother. 2020;75(5):1099–111. doi: 10.1093/jac/dkz543.5710709
    1. Watson J, Hutyra CA, Clancy SM, Chandiramani A, Bedoya A, Ilangovan K, Nderitu N, Poon EG. Overcoming barriers to the adoption and implementation of predictive modeling and machine learning in clinical care: what can we learn from US academic medical centers? JAMIA Open. 2020;3(2):167–72. doi: 10.1093/jamiaopen/ooz046. ooz046
    1. Riley P. Three pitfalls to avoid in machine learning. Nature. 2019;572(7767):27–9. doi: 10.1038/d41586-019-02307-y.10.1038/d41586-019-02307-y
    1. Greenes RA, Bates DW, Kawamoto K, Middleton B, Osheroff J, Shahar Y. Clinical decision support models and frameworks: seeking to address research issues underlying implementation successes and failures. J Biomed Inform. 2018;78:134–43. doi: 10.1016/j.jbi.2017.12.005. S1532-0464(17)30275-7

Source: PubMed

Sponsorer og samarbejdspartnere

Medicinske tilstande

Narkotikainterventioner

3
Abonner