Implementation of machine learning in the clinic: challenges and lessons in prospective deployment from the System for High Intensity EvaLuation During Radiation Therapy (SHIELD-RT) randomized controlled study

Julian C Hong, Neville C W Eclov, Sarah J Stephens, Yvonne M Mowery, Manisha Palta, Julian C Hong, Neville C W Eclov, Sarah J Stephens, Yvonne M Mowery, Manisha Palta

Abstract

Background: Artificial intelligence (AI) and machine learning (ML) have resulted in significant enthusiasm for their promise in healthcare. Despite this, prospective randomized controlled trials and successful clinical implementation remain limited. One clinical application of ML is mitigation of the increased risk for acute care during outpatient cancer therapy. We previously reported the results of the System for High Intensity EvaLuation During Radiation Therapy (SHIELD-RT) study (NCT04277650), which was a prospective, randomized quality improvement study demonstrating that ML based on electronic health record (EHR) data can direct supplemental clinical evaluations and reduce the rate of acute care during cancer radiotherapy with and without chemotherapy. The objective of this study is to report the workflow and operational challenges encountered during ML implementation on the SHIELD-RT study.

Results: Data extraction and manual review steps in the workflow represented significant time commitments for implementation of clinical ML on a prospective, randomized study. Barriers include limited data availability through the standard clinical workflow and commercial products, the need to aggregate data from multiple sources, and logistical challenges from altering the standard clinical workflow to deliver adaptive care.

Conclusions: The SHIELD-RT study was an early randomized controlled study which enabled assessment of barriers to clinical ML implementation, specifically those which leverage the EHR. These challenges build on a growing body of literature and may provide lessons for future healthcare ML adoption.

Trial registration: NCT04277650. Registered 20 February 2020. Retrospectively registered quality improvement study.

Keywords: Artificial intelligence; Chemoradiation; Implementation; Machine learning; Quality improvement; Radiation therapy.

Conflict of interest statement

JCH and MP, are coinventors on a pending patent, “Systems and methods for predicting acute care visits during outpatient cancer therapy,” studied in this manuscript.

© 2022. The Author(s).

Figures

Fig. 1
Fig. 1
Patient identification workflow. New treatment courses were labeled as one of three potential options that required subsequent manual review

References

    1. Coiera E. The last mile: Where artificial intelligence meets reality. J Med Internet Res. 2019;21:e16323. doi: 10.2196/16323.
    1. Beede E, Baylor E, Hersch F, Iurchenko A, Wilcox L, Ruamviboonsuk P, et al. A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In: Proceedings of the 2020 CHI conference on human factors in computing systems. Honolulu, HI, USA: Association for Computing Machinery; 2020. p. 1–12. doi:10.1145/3313831.3376718.
    1. Nimri R, Battelino T, Laffel LM, Slover RH, Schatz D, Weinzimer SA, et al. Insulin dose optimization using an automated artificial intelligence-based decision support system in youths with type 1 diabetes. Nat Med. 2020;26:1380–1384. doi: 10.1038/s41591-020-1045-7.
    1. Hong JC, Eclov NCW, Dalal NH, Thomas SM, Stephens SJ, Malicki M, et al. System for High-Intensity Evaluation during Radiation Therapy (SHIELD-RT): a prospective randomized study of machine learning-directed clinical evaluations during radiation and chemoradiation. JCO. 2020;38:3652–3661. doi: 10.1200/JCO.20.01688.
    1. Wijnberge M, Geerts BF, Hol L, Lemmers N, Mulder MP, Berge P, et al. Effect of a machine learning-derived early warning system for intraoperative hypotension vs standard care on depth and duration of intraoperative hypotension during elective noncardiac surgery: the HYPE randomized clinical trial. JAMA. 2020 doi: 10.1001/jama.2020.0592.
    1. Safavi K, Mathews SC, Bates DW, Dorsey ER, Cohen AB. Top-funded digital health companies and their impact on high-burden. High-Cost Conditions Health Affairs. 2019;38:115–123. doi: 10.1377/hlthaff.2018.05081.
    1. Jairam V, Lee V, Park HS, Thomas CR, Melnick ER, Gross CP, et al. Treatment-related complications of systemic therapy and radiotherapy. JAMA Oncol. 2019;5:1028–1035. doi: 10.1001/jamaoncol.2019.0086.
    1. Phillips CM, Deal K, Powis M, Singh S, Dharmakulaseelan L, Naik H, et al. Evaluating patients’ perception of the risk of acute care visits during systemic therapy for cancer. JCO Oncol Pract. 2020;16:e622–e629. doi: 10.1200/JOP.19.00551.
    1. Admissions and Emergency Department (ED) Visits for Patients Receiving Outpatient Chemotherapy. . Accessed 19 Dec 2019.
    1. Morse KE, Bagley SC, Shah NH. Estimate the hidden deployment cost of predictive models to improve patient care. Nat Med. 2020;26:18–19. doi: 10.1038/s41591-019-0651-8.
    1. Hong JC, Niedzwiecki D, Palta M, Tenenbaum JD. Predicting Emergency visits and hospital admissions during radiation and chemoradiation: an internally validated pretreatment machine learning algorithm. JCO Clinical Cancer Informatics. 2018;2:1–11. doi: 10.1200/CCI.18.00037.
    1. Horvath MM, Winfield S, Evans S, Slopek S, Shang H, Ferranti J. The DEDUCE Guided Query tool: providing simplified access to clinical data for research and quality improvement. J Biomed Inform. 2011;44:266–276. doi: 10.1016/j.jbi.2010.11.008.
    1. Murphy SN, Mendis M, Hackett K, Kuttan R, Pan W, Phillips LC, et al. Architecture of the open-source clinical research chart from Informatics for Integrating Biology and the Bedside. AMIA Annu Symp Proc. 2022;2007:548–52.
    1. Murphy SN, Chueh HC. A security architecture for query tools used to access large biomedical databases. Proc AMIA Symp. 2022;2002:552–6.
    1. Lowe HJ, Ferris TA, Hernandez PM, Weber SC. STRIDE–An integrated standards-based translational research informatics platform. AMIA Annu Symp Proc. 2009;2009:391–395.
    1. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research Electronic Data Capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–381. doi: 10.1016/j.jbi.2008.08.010.
    1. Hong JC, Fairchild AT, Tanksley JP, Palta M, Tenenbaum JD. Natural language processing for abstraction of cancer treatment toxicities: accuracy versus human experts. JAMIA Open. 2020;3:513–517. doi: 10.1093/jamiaopen/ooaa064.
    1. Lyell D, Coiera E. Automation bias and verification complexity: a systematic review. J Am Med Inform Assoc. 2017;24:423–431. doi: 10.1093/jamia/ocw105.

Source: PubMed

スポンサーと協力者

医学的状態

薬物療法

3
購読する